I love that journals let reviewers see the other reviews

Can I just say that I love that many journals let reviewers see the other reviews of the ms after the decision is made? I learn so much from comparing my own evaluation of a ms with those of others. Did other reviewers pick up on the things I picked up on? Did other reviewers pick up on things I missed? Do I seriously disagree with anything the other reviewers said? Did all the reviewers pick up on the same things but disagree on how to weight them or what to do about them? Etc.

I also learn a lot from seeing the editor’s decision letter, in those cases where it explains the editor’s thinking (as it should; decision letters shouldn’t be form letters). Particularly when the reviewers disagree.

I confess I’m proud that my reviews rarely are way out of line with the other reviews, and that when they are the editor generally broadly agrees with my review. I take this as reassuring evidence that I am the thoughtful, careful reviewer I try to be.

As Hannah Gay astutely points out in The Silwood Circle, two things that separate the best scientists from others are (i) heightened willingness to pass judgment (including negative judgment) on the quality, interest, and importance of the work of others, and (ii) heightened yet selective attention to what other scientists are thinking and doing. One of the best ways to acquire and maintain both those traits is to serve as a reviewer for selective journals, and then to read the reviews of others who evaluated the same papers. It hones your judgment.

It’s for this reason that I wouldn’t want to live in a world in which everything was published in non-selective journals that evaluated mss only on technical soundness.* In that hypothetical world (which I don’t think will ever come about, but which some people are calling for), I’d feel cut off from the evaluative judgments of others and so would worry about my own judgment atrophying. And before you ask, no, I wouldn’t consider social media or “post-publication review” a substitute. Social media mostly only exposes you to the judgments of your friends rather than the much broader group of people comprising your field.** Social media also mostly exposes you only to people’s positive judgments of other people’s papers, and mostly doesn’t expose you to the reasons behind people’s judgments. Nobody retweets or Facebook shares papers they don’t like, and people rarely spell out their thinking at length on Twitter. As for post-publication review, it doesn’t exist for most papers. And when it does it’s mostly checks for image manipulation and other misconduct, non-substantive comments, comments that aren’t actually about the paper in question, or abuse,*** so mostly doesn’t let you compare your evaluative judgments to those of others.

*Don’t misunderstand, journals like Plos One have their place. I’ve published in Plos One.

**That’s also why things like lab groups and journal clubs are only an imperfect substitute for exposure to the evaluative judgments of other peer reviewers.

***This statement describes the 20 most recent comments on PubPeer as I’m typing this.

28 thoughts on “I love that journals let reviewers see the other reviews

  1. Agreed, seeing others’ review is useful and an important part of the ongoing learning curve of how to be an effective reviewer. However I wonder about Gay’s/your comment that the “best” scientists show a “heightened willingness to pass judgment (including negative judgment) on the quality, interest, and importance of the work of others”. Some of that “willingness to pass judgment” can be ill-founded in my experience, and result from jealousy, competitiveness, insecurity, and refusal to see other perspectives, amongst other factors. It’s uncommon, but it does happen, and I’m sure most of us can relate anecdotes about it.

  2. Greetings, Jeremy,
    I only recently began to review papers (basically last year, when I finished my PhD and began my post-doc), and I completely agree with this post. Seeing other reviewers’ opinions, as well as the editors’ decisions, permits me to assess how sound or sensible are my own ideas and assessments. In some cases this made me think that my opinions may not be so sound after all and more thought and study may be needed!
    The one think I’d like to add is about technical quality of a paper: reviewers often have different expertises, so whereas one reviewer may pay more attention to e.g. the statistical analyses, another may give more thought to the underlying theory or the field methods. Although ideally all reviewers should look at all these aspects, it’s sort of impossible to have such a deep knowledge of the field, and so reading other people’s reviews is a great way to improve one’s own understanding.

    • Can you elaborate? Do you mean that all reviewers should be non-anonymous? That all reviews should be published alongside the paper? Something else?

      Here’s a review of the research literature on anonymity and openness in peer review:


      Short version: the available data are mostly hard to interpret because many studies have small sample sizes and big design flaws. But it’s common for reforms to anonymity and openness in peer review to have no detectable effect, or even have effects opposite to those that were intended.

      • “Open” in the sense of “you can read all the reviews of the original manuscript”.

        For instance, PeerJ gives reviewers the choice of whether they wish to have their name associated with their review. If they choose “yes”, then it is divulged to the authors. The authors are given the option to publish all the reviews alongside the final paper. If the authors say “yes”, then all reviews are made public. Reviewers who chose to remain anonymous will also be anonymous on the publicly available reviews. Reviewers who chose to reveal their identities to the authors will have their name publicly associated with the review.

        Or at F1000, where–better still–all revisions and reviews of the paper are public, and reviewers’ names are associated with the review.

      • Ok, thx.

        I think it’s great for journals to experiment with different policies on this. Reviewers and authors can vote with their feet if they don’t like journal X’s policy.

  3. Yes to this! In one of my first reviews, the other reviewer signed their review and is a well-respected senior person in my subfield. It was super reassuring to see that my thoughts on the paper lined up with theirs.

    • Your experience here is a reminder that, if you sign your review, it’s not just the authors who will know who you are–it’s also the other reviewers. Which doesn’t affect my own decision-making as to whether to sign my reviews–if I’m fine with the authors knowing it was me, I’m fine with the other reviewers knowing too. But worth keeping in mind.

      • Related to this, I’ve had the experience of knowing who the other reviewers were in the online reviewing system even when they didn’t sign their reviews. Don’t know how common that is but it certainly happens.

  4. Agreed, but as a reviewer, you should be aware that if your review is signed, then other reviewers will see your identity. Worth considering when deciding whether or not to reveal your identity.

  5. I love this, too. For early career folks, being able to read many more reviews as a reviewer than you have yet seen on your own manuscripts *really* helps when writing manuscripts number 1, 2, 3, 4… (and perhaps n, n+1, n+2,… based on your post).

    One really concrete place it has helped me is in determining one’s “overall recommendation” of ‘accept’, ‘minor revisions’, ‘major revisions’, ‘reject’, etc. In my first reviews I felt very unqualified to make that judgement — I was happy to point out good points and bad points in the papers, but I felt uncomfortable about making a judgement as to what those individual comments became as an “overall recommendation,” especially with regard to “fit.” I still don’t think I have a good feel for it, but as I see more reviews, I’m becoming more comfortable making an overall judgement.

  6. Hello,
    I completely agree with this post and I always ask for the other reviews/editorial decision if they don’t automatically send me. Only one journal denied my request, something to do about their policies.

    I recently reviewed a manuscript for Limnology & Oceanography Letters and they have an interesting system. Whenever all the reviews are submitted they send them to all reviewers before an editorial decision is made, and allow you to add/remove/edit your review based on other people’s review. For instance if something you read on another review made you think about another aspect that you didn’t thought before. Or if you have some disagreement with other reviews you are allowed to comment/argue. Anyway, I thought it was interesting but I am not sure how positive/negative it can be for the process.

      • Science has a similar system they call cross review. They send you this message once all reviews are in:

        “We invite you to cross-review this manuscript. During cross review, we invite all reviewers to read the other reviews and make additional comments within 2 business days. Cross-review is encouraged, but not required. If we do not receive comments we will proceed based on the reviews in hand.”

        I thought it was a pretty interesting system.

      • @Alex Etz:

        Interesting, I didn’t know that. That must be fairly new. Last time I reviewed for Science, a couple of years ago, I wasn’t invited to do that.

  7. Also agreed. Reading other reviews has really helped me to figure out what makes a really valuable review and thus what I want to contribute in my own reviews and editorial decisions. It is really one of the most edifying scientific experiences.

    And on the subject of editor anonymity, serving as an editor precipitated a personal shift for me towards always signing my reviews as well. It has actually been very freeing, and in the process I have also found that it can sometimes generate new professional relationships. Real, honest constructive critique is one of the most valuable things we give to one another. Signing my reviews also forces me to make sure that 1) I aim to truly be constructive, and 2) that I can rationally support all of my comments.

    Nice post on a great topic, Jeremy. Thanks!

    • Interesting to hear that serving as an editor has moved you toward signing your reviews as well, and that you’ve found that freeing. As I noted in that old post on editor anonymity, it’s my anecdotal and possibly-incorrect impression that things are moving slowly in the other direction, at least at Ecology Letters. People seem to increasingly feel that the stakes are high, and so are increasingly worried about other people getting upset with them or holding grudges.

  8. interesting post and even at my advanced ‘scientific’ age I too like to be reassured by seeing that the other reviewer(s) agree with what I have said🙂 As an Editor I am happy that the authors can see who has handled their paper and made the correct or totally incomprehensible decision – you get both responses from authors. Sometimes you do actually get a thank you for rejecting a paper where the author (sincerely I think) feels that the comments have made a real difference and will help publication elsewhere.

  9. I would be in favour of open review with reviewers identities being made public if I was not aware of how aggressively and unpleasantly some people in ecology have behaved when faced with criticism. All of the really eye-watering stories in the UK feature people who I imagine appear a lot in “The Silwood Circle”, of course.

  10. Is there a list of the ecology journals that let their reviewers see comments from other reviewers. I find it incredibly frustrating when journals (1) don’t let you see comments from other reviewers and even worse (2) don’t let you see the decision letter [or even inform you of the decision]. The latter is pretty much incomprehensible to me. On the opposite side of the spectrum, I once reviewed a paper for a math bio journal which allowed you to see other reviewers’ reviews before you wrote your own. I found that strange, but interestingly I think that seeing that review before submitting mine made me catch even more errors, because it got me to read the paper again, to verify the other reviewer’s corrections [although I am nervous about potential bias this practice may create].

    • Off the top of my head, all ecology journals I can think of let reviewers see the other reviews. But I only mostly only submit to and review for a relatively small number of mostly-leading journals (EcoLetts, Ecology, Am Nat…) Perhaps other commenters can help answer this?

  11. Occasionally, I can’t tell if I was reviewer 1 or 2 when reading through the decision letter. It’s pretty interesting to me when the two reviews are that consistent/similar!

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s