Journal referees often disagree. Referee disagreements can be challenging for editors to handle. How should editors deal with them?
One common approach, especially among editors at selective journals, is to just reject the paper. That is, anything other than unanimous approval or near-approval of the referees is fatal. This is the path of least resistance for editors. It’s usually justified on the grounds that there are lots of good, or potentially-good, papers to choose from and so decisions have to be made somehow. Plus, editors often aren’t specialists on the topics of the papers that they handle, so when faced with a disagreement between specialists they may feel it’s safest to reject the ms, figuring that if that’s the wrong call the ms can just be published somewhere else.
Personally, while I appreciate the motivations for that approach, it’s one I tried to avoid taking during my days as an editor at Oikos. I felt like my job as an editor went beyond simply choosing the referees. I took the view that my job was to make decisions, informed but not dictated by the reviews. I also felt that it was my job to give authors clear direction as to how the ms could be improved. Just saying “address the concerns of the referees” isn’t a clear direction if the referees disagree with one another. So if the referees disagreed on an important point, or on their overall view of the ms, I saw it as my job to decide who was right, and then explain my reasoning to the author.
These two approaches are really two ends of a continuum. For instance, probably few editors literally just compile the “votes” of the referees and reject any paper that doesn’t get a unanimous “yes”. But I do think editors vary a fair bit in how willing they are to dig into the details of explicit or implicit disagreements among referees*, and how willing they are to take sides when referees disagree (particularly when doing so would involve overruling a negative review).
Of course, another approach is for editors to hedge their bets by rejecting the paper if referees disagree, but with the possibility of resubmission as a “new” ms. We’ve talked about that approach before.
Another approach, if two referees disagree, is for the editor to get a third “tiebreaker” review. I don’t know how common this practice is, although anecdotally I think it’s rare in ecology. There are various reasons why an editor might want a “tiebreaker” review. An editor might be genuinely unsure how to resolve a substantive disagreement between two referees, and so want additional input before deciding who’s right. Personally, I think that’s the best reason to want a “tiebreaker” review. At the other end of the continuum, the editor might simply want a tiebreaking vote on whether to accept the ms or not. Personally, I think that’s an abdication of editorial responsibility; I hope nobody ever does that.
But while I certainly have my own preferences and my reasons for them, there are principled and pragmatic arguments for various approaches. So as an author, what approach do you prefer editors to take when referees disagree? What about when you’re one of the referees? What about about when you’re the editor? What about when you’re a reader of the journal? Why? Looking forward to your comments.
*An implicit disagreement being a case where referee A sees some problem with the ms that isn’t mentioned by referee B. This is common. Sometimes this occurs because referees have different interests and so read the same ms with different eyes (e.g., a specialist in system X might care a lot about how the results relate to previous work in system X, while another referee might not care). Sometimes this occurs because one referee spotted something another missed. Neither of those are really disagreements. But sometimes it occurs because one referee thinks something is a problem but another referee doesn’t and so doesn’t mention it. It’s for this reason that, when I’m reviewing a paper, I try to be explicit about where I think there aren’t problems, as well as where I think there are problems. Indeed, if there’s some aspect of an ms that I think is fine, but that I anticipate that other referees might object to, I sometimes go out of my way to explain why I disagree with those objections, so as to give the author ammunition against any reviewers who might raise them.
Dynamic Ecology 
As an editor, I never “count votes” or look for unanimity. Reviewer comments are much more useful than their summary score (accept, reject, etc.), because the reviewers usually don’t have the broad perspective to see how the paper stacks up against those typical for the journal. One person’s “reject” is another’s “minor revision”, and it should depend on the journal (as long as the science is defensible).
As an author, I say just accept it OK?!
Hi Chris,
You’re operating the way a good editor should, I think–but how common do you think that way of operating is? I’d like to think it’s the norm, but I honestly don’t know.
Think for instance of the way Ecology Letters writes their decision letters. The decision letter from the EiC is a form letter which includes the following:
“Although I realize that you could probably address many of the comments in a revision, the overall nature of the reviews is such that a revised study would not compete for the limited space in Ecology Letters…Despite the fact that most manuscripts sent for external review receive positive assessments, we can only publish those few attracting the strongest support.”
It’s right there in black and white: EcoLetts policy is to only take those mss that attract uniformly positive reviews (and if that’s not the policy, why write the decision letter that way? Why not instead write the decision letter so as to focus on what the *editor* thinks (and why), not on what the *reviewers* think? That may seem like a subtle tweak, but to me as an author it makes a world of difference. Personally, “Hey, we’re just the messengers here” decision letters really bug me.)
Now, in practice I know that that’s not actually the way all EcoLetts editors operate. To their credit, their handling editors do sometimes explain the reasoning for their decisions (in more substantive terms than just “the reviewers had some criticisms, so I decided to reject the ms”). But they often don’t. Plus, it looks like recently EcoLetts has gone to a policy of making handling editors anonymous to authors (?) Maybe there’s information or context I don’t have, but this all has bad optics to me.
EcoLetts isn’t the norm of course. As far as I know, they’re the only ecology journal at which handling editors are anonymous to authors (someone correct me if I’m wrong on EcoLetts policy here). And it’s certainly not rare for handling editors (and EiCs) of other leading ecology & evolution journals to take the approach you and I favor, and to explain themselves and take responsibility in their decision letters. But I am curious exactly how common it is to follow that model, vs. following something like the EcoLetts model.
Very interesting post and discussions. I’m learning a lot here. About editors being anonymous and thus not taking their responsabilities, I have actually experienced it in another ecology journal than Ecology Letters. Surprisingly (I’m being ironic here), the editor’s decision was negative and looked more like a voting decision than someone who actually read the ms… I remember that my first thought, when receiving this anonymous editorial decision, was: what sort of editor is this? I was so shocked and surprised by this anonymous editorial decision that I could not even contain my anger, especially because my experience with this journal was always signed editorial decision letters so far. Therefore, I wrote a rebuttal letter and learned from the EiC that anonymous editorial decision letter are very rare but that it happens from time to time. I really felt that the editor was hiding behind his decision without having the gut to sign it. Would it be possible that the editor had some conflicts of interest with our work? I guess not, but this way of doing really looked suspicious to me…
This is very interesting, especially as a grad student. I am a little disappointed that one bad review is enough for a rejection, but I always suspected this was the case. Twice I have had papers rejected under this scenario. Those papers were eventually accepted, but in one case it took two years! In both cases the negative reviews were not overly harsh or fatal, and were even encouraging. Thus, it was frustrating that the editors rejected with the resubmit as a new submission option. It seems to me that the “accepted with major revisions” option is being less used today. In one of these cases it was apparent that the reviewer did not read, or skimmed my methods, and their negative review focused on improper field design and analysis. When I compared the negative reviewers comments to the my methods, I had done what the reviewer suggested I should have done. Yet, I had to resubmit as a new submission and begin the process again.
As a referee, author, and reader I’d like to see a tie-breaker or allow the author the opportunity to rebut a negative review before the decision. Ideally this would remove a bias or flaws from a possible negative review. There can be many reasons beyond the scope of the MS that can elicit a negative review such as competition, personal biases, or a misunderstanding of the methods used to name a few. Even experts cannot agree on proper methods, and any two humans are likely to disagree on almost anything, from the best pitcher in baseball to when to use a certain statistical test. I would like to better understand why negative reviews have more sway than a positive one considering the vagaries of opinions. Being in the early stages of publishing and trying to build a good publication record it can be demoralizing to start the submission process from scratch based off of a slightly negative review. This take months in some cases and in some cases could be avoided if allowed the opportunity to address a negative or have a third party provide a tie-breaker.
I also have the feeling that “reject with option to resubmit” becomes more and more common in the journal policy. Mostly because it keeps the time from “submission” to “acceptance” low if they reset the clock after the first round. I had this also in submissions where the reviews basically were just “minor revision” but it safes the journal at least a few weeks when they can reset the ticking clock for that metric which allow them to promote themselves as a fast publishing journal…
For you as an author it not bad per se as you normally mention the previous manuscript submission number in the submission form for your resubmission. I guess that in most cases the manuscripts return to the same editors and (hopefully) to the same reviewers.
I think the most frustrating thing about reviewer disagreements for me as an author is when one reviewer says ‘great, here are some minor comments’ and the other writes 2 pages single spaced with criticism, much of which could be easily addressed — but the editor rejects.
In my limited experience, that has always been accompanied by editor comments that basically just say ‘reviewer 2 didn’t like it so out it goes’ without a feeling that the editor read the paper.
It’s much harder to write a lengthy, good review of anything (same goes for course evaluations), and so the negative evaluations will tend to look more considered, which speaks to the point you make in the footnote. If things are good about a paper, say so! A short review is not necessarily tacit approval, but it’s also not necessarily a sign of a lax reviewer, either.
That’s a good point. It’s important as a reviewer that you write your review in such a way that it won’t be misread or misunderstood. I myself sometimes write lengthy reviews that include detailed critiques. But when I do so, I make sure to preface my critiques with a clear, up-front statement of the strengths of the ms and my overall view of it. I think once I even wrote something like “Don’t let the length of my comments give the impression that I didn’t like the ms. I liked it very much, but my minor concerns are sufficiently technical that it was necessary to write a lengthy review”. Although I admit I’m not sure this completely solves the problem you identify. You’re right that it really is rare for someone to write a lengthy review comprised entirely of positive comments, and that for that reason lengthy negative reviews do tend to look more careful and thoughtful than brief positive ones.
I totally agree. Reviewers have a lot of responsibity to write a thorough review. This includes highlighting what is good about the paper, what is weak and where it could be improved, and (if applicable), what sections you feel under qualified to comment on (hopefully the latter doesn’t happen too often if reiviewers choose (and are chosen) carefully). To me, nothing is worse than a short, lukewarm and vague review. It makes me feel the reviewer hardly spent any time on the ms.
I agree. Reviewers should say what is good, what is weak,,where it could be improved and (if policy lie) what the reviewer feels under qualified to evaluate.
Some of these flaws could be avoided if reviews get the opportunity to get reviewed by the same peers that wrote the other review. In “Peerage of Science” they built upon that phylosophy and it really works well. You can write if the opponent reviewer took up some good points you forget to raise (or haven’t think about it) or you can state that you hardly disagree with some points of your opponent. And you have to score the three mandatory sections of a PoS reviews: Merits, Critique and Discussion. This way every reviewer is motivated to explicitly write something justifying in either section.
This way of “peer review by peer evaluation” gives editors another fundation for their decision. Sure its a bit of more work as you (in the worst case) have to write two or three reviews. But mostly the peer review reviews are more like a commentary.
I think that I’ve been mostly blessed with editors that make decisions with nuance based on the content of the reviews. So, I’ve had two mostly negative reviews result in a non-rejection at least once, and in my experience having two reviews disagree with one another has worked out for me as often as it hasn’t.
When I’m handling manuscripts, I typically seek reviews from reviewers that have different sub-specializations to be able to get a more in-depth evaluation of specific aspects of the paper. I need and value the overall perspective of reviewers, but I also am expecting different things from reviewers, based on their experience and talents. The recommendation for the manuscript is predicated not just on whether the reviews are positive or negative, but what those positive and negative reviews have to say. A positive review that doesn’t indicate a careful evaluation of the manuscript has little weight, just as there are a number of indicators of negative reviews that suggest that some remarks from reviewers are more central or valuable than others. Ultimately, the editor has to make an individual call based on total evidence, and any attempt at a formula for how to balance the evidence is weak sauce. That is why editors are supposed to both experts and experienced, because the journal is prevailing on the judgment of the editors. I end up on the bad side of that judgement as often as (if not more often than) others, but I still appreciate the fact that the editor isn’t taking votes but making an individual decision.
Having just finished my PhD, and barely been out of undergrad for 5 years, I have already had most of the experiences you mention in your post (split reviews, multiple tiebreaks, contradictory reviews, rejections with invitation to resubmit following split and easily addressable reviews, negative reviews misunderstanding methods, etc.), and agree we need to do something to make the process less stochastic and more streamlined, so that the field isn’t slowed down. As an interdisciplinary researcher (I do research with ecological components, economic components, theory, and data), the odds of getting split reviews, or reviews whose suggestions contradict one another, is much higher because you tend to get reviewers from completely different backgrounds with completely different sets of expectations. As an early career researcher, this is frustrating because unpredictable and often longer review processes can have significant career impacts, at a stage where everyone is counting your publications. Thus, while we increasingly preach the importance of being interdisciplinary to our students, we may be creating strong incentives against early career interdisciplinarity, and filters that weed out interdisciplinary researchers, in our peer review system.
However, I see hope in the fact that most researchers I talk to share my (and your) frustrations, and are looking for ways to do things better. The more we engage in creating processes that work better at the journals we edit and review for, the more people will want to submit to these journals, and the more influential they will become. Similarly, the more we start using preprint forums, like the arXiv, which is used by most mathematical, physical, and chemical scientists, as well as some theoretical biologists (myself included), the less long review processes will slow fields down (because breaking results will be available and citable even as review processes drag on), and the more journals will have incentives to streamline so as to not be publishing old ideas. This is exactly what has happened in physics, for example.
As a quick aside, I also want to mention that I think the ‘Reject with an Invitation to Resubmit’ route is even more problematic for interdisciplinary researchers, because it effectively doubles the number of reviewers you have to please (they make you address the previous review comments in your resubmission but give you brand new reviewers usually), and thus doubles the number of possible conflicts you can have. For example, I recently had a paper get rejected with an invitation to resubmit on the grounds that the ecologist reviewers thought it was too dense, and then simplified and streamlined it, and then had a major revision because one of the new reviewers was a mathematician who wanted more mathematical details (some of which I had taken out to address the previous reviews). Editors should be more decisive, and in my opinion, should only use reject with an invitation to resubmit in unusual cases where the paper has real hope but needs to change so fundamentally that it would be a different paper. For example, the first time I submitted the same paper I mentioned above, I had 3 reviews where one wanted more details on the theory, one wanted more details on the case study I had included at the time, and the third thought it was too long. Here, I got a reject-resubmit, which I think made sense because the only way to satisfy them all was to split it up into separate papers, which would really be (and were) completely different from the original. Outside of cases like this, most uses I see of that option seem like ways to make the journals’ turnaround time look faster, which doesn’t fool anyone because we all know which journals have given us multiple reject-resubmits, and often think twice about submitting there again.
“agree we need to do something to make the process less stochastic and more streamlined, so that the field isn’t slowed down”
Just to clarify, I actually don’t have a big problem with referees disagreeing, and I don’t think cutting down “stochasticity” in the review process is a feasible or even necessarily desirable goal. See this old post:
Thanks for sharing this old post. As you say in the first sentence: “In finance and economics, the answer is: pretty random”. My experience, and that of many of my colleagues, has been that ecological economics is similar. Perhaps this is because it is a type of economics, but I think the interdisciplinary nature of the field has something to do with it also.
To be clear, I don’t have a problem with reviewers disagreeing either per se, but I think (and I thought you were arguing this in today’s post also) that reviewers disagreeing makes the editor’s job more important. The point I was trying to make in my reply was that interdisciplinary fields tend to have more common reviewer disagreements because reviewers from different disciplines expect different styles and conventions. This isn’t necessarily a problem on its own, but can become one if the editor abdicates responsibility, because conflicting interdisciplinary reviews very often have mutually exclusive suggestions (e.g. more details, vs. fewer, use economic conventions vs. ecological ones, etc.). If editors keep abdicating responsibility, and keep giving you new reviewers at the same time, it can lead to a cycle of repeated review that I think sometimes slows the field down and doesn’t improve the papers much (they can become a hodgepodge of conventions and styles that satisfies the specific concerns of the various different reviewers at the expense of the overall flow, cohesiveness and readability of the paper). Again, this is just my personal observation, and I would love to hear what others who do interdisciplinary research think/have experienced.
Moreover, while I agree with your argument that we won’t be able to do away with stochasticity in what reviewers find interesting or important, I don’t think this implies that cutting down on stochasticity is completely infeasible and undesirable. The stochasticity in what reviewers find important or interesting will be inevitably higher in interdisciplinary research, and we may indeed just have to deal with that. But the source of stochasticity in interdisciplinary reviews where reviewers are disagreeing on technical presentation styles based on differing conventions is something I think we can and should try to minimize. In fact, the guidelines you recommend in your post for editors would go a long way towards reducing that type of stochasticity, in my opinion.
Thanks for the further comments Matt, I see where you’re coming from now, and I agree with everything you say.
Thanks Jeremy, and thanks for starting this conversation with your post! I wonder if it would be worth starting a website or something where people anonymously enter different information about their reviews relevant to this (e.g. their review times, number of rounds, number of reviewers, whether the editor added their own insights or not, etc.), and then use the resulting database to generate summary statistics for different journals that could help people decide where to/not to submit. Has anyone done something like this? I feel like making these trials transparent would help put the pressure back on journals to clean up their acts. My sense is that the reject-resubmit route has allowed journals to make long review times less transparent, and in the process probably increased them.
Being asked to meet conflicting statistical preferences from both economists and ecologists in interdisciplinary papers was such a recurrent issue for one of my postdoc mentors, that he decided to just write a paper outlining the ways that their world views differ:
http://onlinelibrary.wiley.com/doi/10.1111/j.1365-2664.2009.01628.x/full
Regrettably, its existence hasn’t spared us the same phenomenon Matt observes above. I’ve seen papers rejected over this issue, as well as those that have suffered considerably in clarity and brevity by trying to reconcile conflicting disciplinary view points without direct guidance from an editor. I’m far less experienced in this than Matt is, but it is definitely aggravating to be pinged back and forth between (often changing) reviewer opinions in the absence of an assertive editor.
Giving authors guidance on how to reconcile conflicting disciplinary viewpoints is a great example of why good editors are needed. The same issue comes up within disciplines too. For instance, there’s a leading ecology journal that I stopped submitting to for a long time because twice in a row they sent my microcosm papers to microbial ecologists, who (predictably) didn’t understand them at all and raised lots of irrelevant objections.
Thanks for your comment Eric, and for pointing me to this paper! Fortunately, I’ve only experienced the econometrics vs. ecology statistics review battle once, but I have a few colleagues and friends that would get a kick out of this paper (and Paul Armsworth is a great ecological economist).
Happy to see that the “interdisciplinary struggle to fit” gets some interest. I was surprised many years ago to realize that the topic of publication (and thus, reviews and editorial decisions) was hardly ever considered in discussions about interdisciplinary science. It is something I encourage my peers and students to consider when developing an interdisciplinary curriculum – we have to remember that, sadly, most of the evaluation frameworks in academia are field-oriented. It also applies to reviews within a discipline, of course – I have suffered from conflicts between empiricists and theorists for some of my papers.
So to complement what has been said, I would add that reviewers need to not only balance their negative comments with some positive ones (if they exist), but that they should also state clearly whether they could judge some aspects of the paper well or not. Most people are happy to say that about complicated stats, but what about saying “I am a bit skeptical of the use of theoretical models to advance science, so what I have to say probably reflect that view”? (I’d like a pony, too)
I recently had the experience of publishing outside ecology in an InformationSystems journal with two IS colleagues. Some differences from Eco journals 1) the review was double-blind and 2) the AE was VERY involved, providing detailed comments of his (we relieved the identity upon acceptance) views of the reviewer comments and ideas/suggestions for revision. The AE comments were not boilerplate, and hugely helped to improve the manuscript. The downside: the prices from initial submission to acceptance involved four revisions (including addition of a new experiment and a second set of statistical analysis) was over two years!!! I was aghast, but my IS colleagues said this was normal for this prestigious (in the IS field) journal… The journal takes the position that it wants all papers to end up being “classics” in the field. Obviously not all are, but they are all probably darn high quality (I wouldn’t be able to evaluate).
The first time I reviewed a MS, I was surprised I had to make a recommendation to accept, reject, etc. I figured that was the editor’s decision, as I didn’t have the experience to make judgement calls about fit at the journal or priority of MSs at the journal. I figured my responsibility was to carefully read the paper for accuracy and for making a contribution to the field. I had to email the editor and ask what constitutes the difference between, e.g. major vs minor revision. I ended up suggesting accept with revision based on the paper’s soundness, but it ended up being rejected based on “fit” with the journal — it wouldn’t appeal enough to the journal’s audience.
Perhaps reviewers shouldn’t be asked to make a call about accept, reject, etc, and should just give written feedback to editors. I agree that it really should be the editor who makes the call, and this would discourage lazy editing a bit…
Agreed. A good editor won’t really care about the reviewers’ accept/reject/revise recommendations. Good editors know much more about what sort of mss the journal is receiving, and also have a much broader view of the field than a typical reviewer. So good editors are much better placed than reviewers to make the decisions.
I partly agree. I think you’re absolutely right that reviewers (particularly early career ones) don’t always have the best sense for where the goal post is, but I think the recommendations can help give context to the sentiment of the reviews that sometimes doesn’t come through clearly. This is particularly likely to happen when there are cultural differences between reviewers and the editor. For example, on one occasion I saw a review that was really detailed and looked grumpy (using words like ‘concerns’ a lot), but then the recommendation was ‘Accept with minor revisions’, so they obviously liked the paper, but just wanted to give as much helpful feedback as possible. If I hadn’t seen the vote, I would have thought they hated the paper. So I think having the votes can be helpful, but agree that the editors need to not treat them literally as votes and use their judgement.
But wouldn’t it be better, as a reviewer to say something like, “this paper is well written, technically sound, and advances the field of X. Following are my minor suggestions for improving the MS,” rather than relying on the selection of “accept”? Shouldn’t reviewers be explaining WHY the MS should be accepted? I might argue that not having reviewers select from accept, reject, etc. might reduce lazy reviewing, too…
Perhaps. I agree that what you suggest reviewers do is better than having them vote, but I think your suggestion might be an optimistic communication standard for some reviewers, particularly when there’s a language barrier, and language/cultural differences is what I think are usually to blame for reviews that sound grumpier than they mean to. It’s also true that the voting could make reviewers lazier though. Hard to know to what extent this would happen, but agree it’s worth considering.
If the paper is rejected,then there is no worth resubmitting it as then too the same will happen