Can I just say that I love that many journals let reviewers see the other reviews of the ms after the decision is made? I learn so much from comparing my own evaluation of a ms with those of others. Did other reviewers pick up on the things I picked up on? Did other reviewers pick up on things I missed? Do I seriously disagree with anything the other reviewers said? Did all the reviewers pick up on the same things but disagree on how to weight them or what to do about them? Etc.
I also learn a lot from seeing the editor’s decision letter, in those cases where it explains the editor’s thinking (as it should; decision letters shouldn’t be form letters). Particularly when the reviewers disagree.
I confess I’m proud that my reviews rarely are way out of line with the other reviews, and that when they are the editor generally broadly agrees with my review. I take this as reassuring evidence that I am the thoughtful, careful reviewer I try to be.
As Hannah Gay astutely points out in The Silwood Circle, two things that separate the best scientists from others are (i) heightened willingness to pass judgment (including negative judgment) on the quality, interest, and importance of the work of others, and (ii) heightened yet selective attention to what other scientists are thinking and doing. One of the best ways to acquire and maintain both those traits is to serve as a reviewer for selective journals, and then to read the reviews of others who evaluated the same papers. It hones your judgment.
It’s for this reason that I wouldn’t want to live in a world in which everything was published in non-selective journals that evaluated mss only on technical soundness.* In that hypothetical world (which I don’t think will ever come about, but which some people are calling for), I’d feel cut off from the evaluative judgments of others and so would worry about my own judgment atrophying. And before you ask, no, I wouldn’t consider social media or “post-publication review” a substitute. Social media mostly only exposes you to the judgments of your friends rather than the much broader group of people comprising your field.** Social media also mostly exposes you only to people’s positive judgments of other people’s papers, and mostly doesn’t expose you to the reasons behind people’s judgments. Nobody retweets or Facebook shares papers they don’t like, and people rarely spell out their thinking at length on Twitter. As for post-publication review, it doesn’t exist for most papers. And when it does it’s mostly checks for image manipulation and other misconduct, non-substantive comments, comments that aren’t actually about the paper in question, or abuse,*** so mostly doesn’t let you compare your evaluative judgments to those of others.
*Don’t misunderstand, journals like Plos One have their place. I’ve published in Plos One.
**That’s also why things like lab groups and journal clubs are only an imperfect substitute for exposure to the evaluative judgments of other peer reviewers.
***This statement describes the 20 most recent comments on PubPeer as I’m typing this.