Review to submit: a solution to the peer review crisis

The peer review system in science seems overwhelmed: everyone gets too many review requests while waiting too long for their own papers to get reviewed. What can be done about this?

Luis Pedro Coelho🌻 (in Lisbon. July 21-31) @luispedrocoelho

The solution is "review to submit" - papers from authors who don't review other papers are simply administratively rejected

Nathan Whitmore @brain_juices

@drkellyallen @AcademicChatter In all seriousness it seems like the solution to this is competitive pay for reviewers. Like I get that reviewing has traditionally been considered an academic service thing. But if that's not working well anyone shouldn't we think about switching to a different model?

Here is the system, in broad strokes:

1. Every scientist, as identified by their ORCID, has a “review account” with a certain number of review tokens.

2. When you review a paper, you get (at least) 2 review tokens for a first review, 1 for a re-review. Journals may choose to increase this (in a coordinated fashion) if reviews become hard to obtain.

3. Submitting a paper costs 1 token for the initial submission and an extra 4 tokens if it goes out for review. You do not get them back if the paper is rejected. This is for the paper as a whole, so only one author needs to spend their tokens. In fields like life science, the expectation is that it’s the senior author that is “paying the review,” especially if the first-author is just starting out.

I think it has the right incentives: it encourages authors to both (1) review more and (2) submit less to venues where a manuscript has very little chance of acceptance. It avoids the most egregious abuses (authors who simply never review—there are many of them), and should generally speed things up.¹

I discussed this on Twitter and some objections immediately showed up.

What about junior researchers who are not getting review invitations because they have not published yet (or have only one or two papers, so unknown to editors)?

First of all, in fields where a senior author is common, this is not an issue: your first papers would be sponsored by a senior author and by the time you are a senior author yourself, you’d hopefully have been invited to review many times already. I realize this would not cover all fields, but at least large chunks of the scientific space would be covered.

It also seems easy to have everyone start with a small number of tokens when they create their ORCID number.

An interesting point is this family of objections boils down “what if people don’t have enough opportunities to review?” which is the opposite problem of what exists now, which is that people are not reviewing! This would be a good problem to have and it’s easy to think of a few pieces of infrastructure that would help (tagging yourself as interested in reviewing a preprint would be an excellent help for editors that are later receiving that preprint for peer-review).

Also, as an editor, one of the main reasons why I don’t invite junior people more often for review is because I cannot easily find their email address! If there were more incentives to review, I would also hope that people would make themselves easier to find.²

How do you get there from here?

One good thing about the concentration in academic publishing is that we only really need a few publishers to adopt the system and that would be most journals.

However, this system makes sense even if implemented only across a small group of journals (for example, all of the ASM journals). This would also solves some issues of trust (see below).

Wouldn’t people just submit low-quality reviews?

I would add to editors the power to reject reviews and not attribute any tokens.³

What about malicious journals that just give out tokens without any quality control? In this case, the editors would be in cahoots with the authors (like predatory journals now).

This is a serious objection, but some forms of quality control already exist and it would not be an issue if it was done within a smaller group of journals (ASM, as mentioned above, or PLoS).

This is also why journals should not be able to unilaterally assign more tokens for reviews. This would be coordinated or subject to rules (e.g., if 5 people have refused invitations to review, then the journal can increase the number of tokens it assigns; or a publication group can only give a fraction more tokens than the ones it receives as submissions).

A scientist reviews the work of another scientist while she is also being reviewed, in the style of Paula Rego, by MidJourney

Why not just pay people in cash? Cash is the ultimate token.

This is the alternative model: just pay people in euros or dollars. There are a few issues, in my mind, with this proposal:

First of all, a significant fraction of scientists would not be able to participate because they cannot get income outside their main salary (either for contractual reasons with their employer or because of visa rules). If some people are getting paid, would I want to keep doing it for free? Would salaries start to be adjusted down to account for the fact that some people can get easy outside income just by reviewing?

Secondly, any amount would be inappropriate in some part of the world (either too high or too low). 450 USD is a number that makes sense in the US, but is very high for other countries (even for some European countries, this would mean that two reviews in a month already represent a significant increase in income). Maybe this is a feature (transferring wealth to poorer countries is a good thing!), but I am not sure this would be a stable equilibrium: most likely the price would get bid down to a value too low for high-income countries and then scientists in those countries would stop doing it. Review tokens are fundamentally fairer: they are worth the same to any scientist.

Thirdly, this would fundamentally increase the costs of publishing, which would get passed on to authors in the form of fees (the cry to pay reviewers is nothing more than an informal conspiracy to turn grant money into personal income).⁴

Monetary economics is hard and setting prices is harder. How do you avoid getting into dysfunctional equilibria?

This objection did not show up, but this system is reminiscent of the Capitol Hill Babysitting Co-Op and similar problems can show up here. This is why I think the price of a review should be allowed to change.

The problem is eased compared to other situations because people will still want to publish (spend tokens) because of external pressures. The issue only arises if people are not reviewing. By allowing the cost of a review to rise, this should be enough to ensure that the system does not collapse for macro-economic reasons.

Acknowledgements. I thank Calvin McCarter, Anna Cuscó, and Svetlana U. Perović for comments on an earlier version of the post.

1. The biggest delays are in finding reviewers that accept to review (or even click No on the invitation to review); it’s rarely in the reviewers being late with reviews (which is not uncommon, but rarely more than a few days).

2. Adding your email to your ORCID record or linking your Google Scholar profile to your webpage would go a long way to getting me to invite you more.

3. The current system also leads itself to abuse, though.

4. There is an oft-repeated claim that Elsevier makes such large profits that these payments could come from simply cutting into profit margins. I think this is wrong as every time I tried to track down the claim, I get to the same documents which present an EBITDA profit and not a full cost accounting. EMBO published an accounting of its costs and it does not seem that there is room to spend a few extra thousand euros per paper paying for reviews without increasing fees to authors.