Poll: should all reviewers be satisfied before a paper is accepted for publication?

I was very surprised by the results of Meg’s recent poll on what reviewers mean when they say that, yes, they’d be willing to review a revised version of an ms. 34% mean not merely that they’re willing to review a revised version, but that they want to see a revised version to make sure the authors have addressed their concerns. Like Meg, I had no idea that reviewers who feel that way are such a large minority!

Which got me thinking about the roles of reviewers and editors, and if my own view on their roles isn’t as universal as I had (naively?) assumed. So below is a one-question poll. Do you see reviewers as advisers to the editor? Or do you think editors should ordinarily defer to reviewers, so that all reviewers should be satisfied before a paper is accepted for publication?

The poll gives you four options. First, I briefly elaborate on each; then comes the poll where you choose the option that best matches your own views. I’ll give my own view at the end, but no peeking until you do the poll! (Note that for purposes of this poll I’m glossing over the roles of the EiC vs. the editorial board. At many journals ultimate decision-making authority rests with the EiC, but don’t worry about that for purposes of this poll.)

  1. Reviewers advise, the editor decides. Editors will take the advice of reviewers seriously, but sometimes will overrule some or even all of the reviewers when they disagree with the reviewers or when the reviewers disagree with one another.
  2. Editors defer to reviewers. Review ordinarily will continue until all reviewers are satisfied, with the ms being rejected otherwise. Only in rare circumstances, such as when a reviewer is sloppy or biased, will an editor accept an ms over a reviewer’s objections.
  3. It depends if it’s a technical or subjective issue. Editors ordinarily will defer to reviewers on all technical issues, and not accept an ms unless all reviewers are satisfied on all technical issues. But “reviewers advise, the editor decides” when it comes to more subjective issues (well, issues that are widely seen to be more subjective) like whether the ms is a good “fit” for the journal and whether the ms is sufficiently “interesting” for the journal.
  4. I have some other view as to the roles of editors and reviewers, very different from the other options.

My own view (which I hold as both a reviewer and a former editor) is that reviewers advise, the editor decides. That’s because a final decision has to be made somehow, because reviewers sometimes disagree with one another, and because editors generally are better-placed than reviewers to make that decision. Editors typically have a broader view of the field than do reviewers, for instance. Editors also know more about how the ms compares to others recently submitted to the journal. Also, as a practical matter, it can be hard to get good reviews and as an editor you don’t want to burden your colleagues asking them for reviews unless you really need them. Until I saw Meg’s poll, I had assumed that my view was nearly universal–we’ll find out if that assumption was correct!

p.s. Before anyone comments to point this out: yes, I know the poll options define points on a continuum. For instance, as an editor at Oikos and Axios I was less deferential to reviewers when handling mss that were “in my wheelhouse”, more deferential when handling mss far from my area of expertise. But I couldn’t figure out how to write a one-question poll that would capture the range of people’s views, so I just went with the poll above in the hopes that it will be a good conversation starter.

22 thoughts on “Poll: should all reviewers be satisfied before a paper is accepted for publication?

  1. It would also be interesting to know if there are differences between what people think should ideally be the case and what they think is currently the case.

    • I have experienced the editor deferring to the majority opinion of the reviewers more often than I (and others, according to the poll results at this point) think they should. It would be nice if the editor would at least read the manuscript carefully and allow their own judgement have the weight of at least one reviewer.

      • “It would be nice if the editor would at least read the manuscript carefully and allow their own judgement have the weight of at least one reviewer.”

        I think it’s hard to argue with that.

  2. Good thing I decided at the last minute to include the “depends if it’s a technical or subjective issue” option! That’s the second most popular choice so far (40-ish votes so far). It’s getting about 1/3 of the votes, about the same fraction as “I want to see a revised ms so I can check whether the authors addressed my concerns” got in Meg’s poll.

    I’m tempted to infer that it’s the *same* 1/3 of people in both polls. That’s probably not totally correct, but I bet there’s a fair bit of overlap.

  3. While I agree that final decisions rest with the Editor, I lean more in the opposite direction: I prefer to see Editors relying more heavily on advice from referees than suggested by choices 1 or 3. Editors still retain tremendous power to determine what gets published under choice 2, so that was my selection.

  4. Part of my reason for wanting to re-review the MS, that I haven’t seen mentioned, is to learn. I’m an ECR and I don’t know all of the norms. Seeing what comments of mine get push back and which are supported by the editor, help both my reviewing and writing.

    But also, I expect I spend much more time with a paper than the editor so I don’t know why an editor wouldn’t defer to my comments. Sure, general ‘fit’ and broad field view but I’m the one picking up on the overselling, sloppy definitions, and misleading references.

    • “I expect I spend much more time with a paper than the editor ”

      Depends on the editor. As an editor at Oikos, I mostly read papers as carefully as I do when I’m a reviewer. I believe most Am Nat editors do the same.

      • I can back this up. As an Am Nat editor, I think I read as carefully as the reviewers do, and I think that’s typical for the journal. But of course, other journals don’t have the same culture.

  5. ‘Reviewers advise, the editor decides’ is what I would normally say. But given your elaboration of that option, ie

    ‘Editors will take the advice of reviewers seriously, but sometimes will overrule some or even all of the reviewers when they disagree with the reviewers or when the reviewers disagree with one another’,

    I couldn’t choose it as I had a problem with editors overruling all of the reviewers. Deciding not to go with some parts of the reviews/recommendations of reviewers is fine, an important editorial responsibility, and should be handled transparently. But I doubt good/experienced editors, who’ve identified the most appropriate reviewers and hold fairness uppermost, would find themselves in the situation of overriding all of the reviewers (unless there were some very unusual circumstances). ‘Peer review’ then becomes solely the editor’s opinion.

    So I chose ‘It depends if it’s a technical or subjective issue’ (despite having a bit of an issue with the ‘all technical issues’ and ‘all reviewers’ bits).

    • “But I doubt good/experienced editors, who’ve identified the most appropriate reviewers and hold fairness uppermost, would find themselves in the situation of overriding all of the reviewers (unless there were some very unusual circumstances).”

      I agree that that would be unusual. But I can imagine it happening. I think it’s most likely to happen in two somewhat contrasting cases: very unconventional papers, and very conventional papers. For instance, I’ve heard that Ellstrand (1983) bounced around for a while before finding an editor willing to take it. It’s a satire, so highly unconventional even for an opinion/perspectives-type paper. Conversely, I can imagine a very conventional paper getting rejected by an editor even if all the reviewers like it. On grounds of being insufficiently novel, of insufficiently broad interest, or maybe because it makes some common mistake that’s not widely recognized as a mistake (so doesn’t get caught by any of the reviewers).

      • Jeremy, in your “conversely” cases, I think these should normally be handled by a reject-without-review decision. These are papers that do not merit the investment of time and effort by reviewers (or investment of time waiting for decision by authors), at least in their current incarnation or at their current journal. So this is not a matter of overruling reviewers, but rather of opting not to use them in the first place!

      • Yes, agree with you, those sorts of cases are ones where an editor can override reviewers’ feedback or opinions (would first couple fall into ‘subjective’ bit in choice #3?). One of the ‘very unusual circumstances’ I had in mind would be if the editor became aware of an ethical issue that meant the ms couldn’t be accepted however positive the reviewers were.

        Had forgotten once I was in the post that its title is ‘should all reviewers be satisfied before a paper is accepted for publication?’ 😊 – answer to that would be ‘no’. Editors have to act as editors and make decisions on publication and, importantly, on what revisions authors need/need not do to get their manuscripts accepted. Doesn’t always happen, eg ‘End the wasteful tyranny of reviewer experiments’ http://www.nature.com/news/2011/110427/full/472391a.html

  6. Is there such a thing as a purely technical issue on which people disagree? There are technical flaws on which all (well, a vast majority) the scientists of a field agree. But if there are disagreements, this issue becomes more subjective… Off the top of my head, statistical issues, e.g. how seriously the assumptions of a test have to be violated to make the conclusions invalid.

    • I actually agree with you that there’s no hard and fast line between “objective” and “subjective” issues. But I suspect that’s a minority viewpoint and so I phrased the question as I thought someone with more typical views would phrase it.

  7. This poll and the previous one point out to something important: academic publishing has been changing in the past decades, and now there are multiple views among the people involved in the system. The signs have been there for a while as, for instance, we stopped calling the ad hoc help “referees” (editors must defer to them) and now the most used term is “reviewers” (editors listen to them, but must not defer to them). Independently of which view is predominant, maybe journals should state roles (editor-in-chief, associate editor, handling editor, and reviewer) more clearly on their websites.

      • It’s only a personal impression, as I haven’t seen the term “referee” being used for some years. The only problem is how to collect the data and test this hypothesis.

    • I agree with you and use of ‘referee’ is one of my big bugbears! 10 years ago I wrote the following in my peer review book (pp16-17), and have been trying to persuade people in favour of ‘reviewer’ since then: ‘A variety of names are used for the people to whom a manuscript is sent for assessment or review, the most common being reviewer, referee, assessor or advisor. It is largely a matter of personal preference as to which is used, but as in all things, it is better for the term to reflect the role as much as possible. Hence, throughout this book I will use the term ‘reviewer’ (to reflect peer review, i.e. review by peers). The term referee, which is very commonly used, implies that the person evaluating the manuscript is an umpire, someone deciding on fair play between opposing sides and enforcing rules and settling disputes. But it is editors who have the decision-making role and who weigh up all the arguments and evidence and decide how to settle disputes, and so any names that might cause confusion should be avoided. Reviewers advise and make recommendations; it is editors who make the decisions.’

      • Thank you for the reply, you wrote beautifully about the issue. The change you asked for seems to have happened, at least in part.

  8. This is a really tough question. I just recently had an ms rejected by the editor, but the reviewers comments were overall very positive and easy to accommodate. It was very clear that the editor was unfamiliar with the approach that we used for analyzing the data, yet the reviewers were. This was a huge disappointment, and the editor was not aware of the analysis that we used and did not both to do a little bit of research into the technique. Not only that, but they did not rely on the reviewers comments at all. However, I could certainly see this going the other way.

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s