In our recent compilation of ‘advice’ posts, I solicited requests for new topics on which readers want advice. Commenter Colin asked for advice on writing peer reviews. Ask and ye shall receive!
A bit of googling will turn up many good sources of advice on how to write a good peer review. Here are a few:
The Ecological Society of America’s official instructions to reviewers for its selective journals. Goes into detail on what the ESA’s journals want from reviewers.
How to be “professionally judgmental” from The Research Whisperer.
Tips for reviewers from the Australian Academy of Science.
I’ll try not to duplicate these and other sources of advice more than necessary. You should read those other sources in addition to my advice.
Advice for before you start writing your review
Remember, the purpose of your review is to provide rigorous but fair feedback, which will help the author improve the ms, and help the editor make a decision on the ms.
Read the ms carefully and critically. Not that you should set out with the goal of finding flaws. But it’s the author’s job to convince you that the work reported in the ms is technically sound, interesting, and important. Read the ms in a skeptical-but-open-to-being-convinced frame of mind. And do read carefully–remember, you and the other reviewers may well be the only people who ever read and pay attention to every line of the ms (seriously).
Remember that you are free to question any aspect of the ms. Don’t take anything in the ms for granted, including things like the framing of the question, the accuracy and completeness of the background review in the Introduction, the relevance to larger issues claimed in the Discussion, etc. Don’t read the Methods and Results carefully while skimming the rest of the ms.
Don’t worry about who the author is. Don’t presume that anything from Dr. Famous’ lab must be right (or wrong!), or that anything you’re reviewing for Nature must be brilliant (or oversold rubbish!)
Check and see if the journal has any instructions for reviewers. If they do, follow them.
If you’re not familiar with the journal, have a glance at a recent issue so you have some sense of what sort of ms they’re looking for. One thing you’ll have to comment on is whether the ms is a good “fit” for the journal.
The three parts of a review
For most journals, there are three parts to a review: an online form you have to complete, comments that only go to the editor but not the authors, and comments that go to the authors (which the editor will read as well).
The online form is the least important. What matters are your comments to the editor and author. So don’t agonize over exactly what score to give the ms on the “originality” scale in the online form or whatever.
Comments to the author
I write my comments to the authors first, because comments to the editor need to reflect and summarize comments to the author.
Begin with a one-paragraph summary of the ms–what was done, the main results, and the main take-home message. This helps demonstrate to the author (and editor) that you read and understood the ms.
Next, summarize the biggest strengths of the ms. Find something good to say even if the ms is terrible. Conversely, if you thought the ms was really great, it’s fine to say so to the authors. In general, the overall impression your review gives to the authors should be consistent with your overall impression of the ms.
Then summarize your main criticisms and suggestions for improvement.
Then provide a detailed list of comments. Number them so the authors can refer to them easily in the event they’re invited to submit a revision (in which case they’ll need to describe to the editor their response to every comment you made). Refer to page and line numbers in your comments wherever possible. I like to organize my comments into “major comments” and “minor comments”. The more detailed your comments are, the better. Detailed comments show that you read the ms carefully and encourage the editor and authors to take your comments seriously. Non-specific comments like “the ms isn’t very interesting” or “the ms should be 20% shorter” are useless without elaboration. Why isn’t it interesting? What material should be cut? Etc. If you have a suggestion as to how to address a problem you’ve identified, provide it.
If some of your comments are more along the lines of suggestions–just food for thought, something the authors might want to consider–say so. (UPDATE: on reflection, I should’ve emphasized how important this is. A very common mistake in peer review is to criticize an author for not doing something the way you personally would have done it, or for not doing something in the “best” way, even though the way the author did it is perfectly adequate. There’s often more than one way to skin a cat when you’re doing science, and as a reviewer you need to recognize that.)
If you think the authors have made a mistake so basic or serious that you can hardly believe anyone could’ve made it, it’s fine to hedge your criticism by saying something like “Apologies if I’ve totally misunderstood, but the claim on line XXX makes no sense…” Then, after you explain the mistake, tell the authors to either correct the mistake or clarify the writing. This is a useful way of writing even if you’re quite confident that you have understood perfectly well, and that the authors really have made a totally boneheaded mistake.
If your comments are really negative, be polite and professional about it (not always easy because it’s really annoying to have to waste your time reviewing a very poor ms). But don’t pull your punches–you need to be clear, honest, and forthright about what the problems are. Yes, even if the lead author is a student. In deciding to submit a ms to peer review, the authors have indicated that, in their view, their ms is ready to be evaluated (and indeed, to be published!), and so are prepared for any evaluation the ms might receive. And if they’re not prepared for any evaluation the ms might receive, that is their fault, not yours, and there’s nothing you can do about it. You should write as a professional addressing a fellow professional. If the author you’re addressing can’t take that, tough. It’s your job to review the ms, not to help the author feel good. You do no one any favors by trying to make your criticisms sound less negative than they actually are. If you feel really badly about being negative, include some positive suggestions for how the authors could build on, improve, or redo the work reported in the ms, even if the ms itself isn’t salvageable.
There is no particular length you should aim for. Your review should be as long or as short as necessary to say everything you need to say. I’ve written reviews that ran to several single-spaced pages, and I have colleagues who’ve done the same.
When recommending revisions, take care not to recommend changes that, individually or collectively, would be unreasonable or impossible to make. For instance, if the ms is already about as long as the journal permits, do not recommend adding a bunch of material without also specifying what material you’d like to see cut. Otherwise the authors will (quite rightly) respond “We appreciate the suggestion to add discussion of issues X, Y, and Z, but as the ms was already lengthy we were not able to do so.”
If you’re not sure about a comment–say, you think there might be a problem with the stats, but you’re not sure–say so, and encourage the authors to clarify.
While there’s such a thing as being too nitpicky–rare is the ms that reports perfect science!–when in doubt err on the side of being more nitpicky rather than less.
It’s good to get in the habit of checking at least the key references, if not all of them. Make sure they say what the authors claim. Miscitations are common, sadly, especially in the Introduction.
Point out any grammatical errors or typos you find, unless there are a lot of them (e.g., because the author is not a native English speaker), in which case just recommend that the ms would benefit from careful editing. If the ms is by a non-native English speaker, note in your comments to the author and editor whether there were any places in which you found the ms difficult to understand, or whether the grammatical problems were merely cosmetic.
Most journals leave it up to you whether to sign your review, with a default to anonymity (I think Nature insists that its reviewers remain anonymous, if I recall correctly) Some people always sign their reviews, some never do, some do only sometimes, usually when they know the author or when the review is positive. It’s really up to you. I don’t want to debate whether or not it should be that way, I’m just telling you how it is.
Comments to the editor
Remember, the authors won’t see your comments to the editor, so don’t mince words. Don’t be rude about the authors themselves, of course. But if you think the ms is horrendous, then the editor needs to know that. And if you think the ms is the greatest thing since sliced bread, the editor needs to know that!
If you’re not sure whether or not to recommend acceptance, minor revision, major revision, rejection, or whatever, say so in your comments to the editor, and explain why you’re uncertain. That’s no problem. A good editor won’t really care much what decision you recommend anyway, they’ll mostly care about the comments on which your recommendation is based.
It’s often useful to tell the editor about the perspective from which you read the ms. It’s always good to give the editor context to help him or her fully understand your comments. For instance, Science once asked me to review an experimental evolution ms on the evolution of evolvability. I have a passing interest in this topic, but I’m very far from an expert. Nor am I an expert on the experimental system used. I assume Science just wanted the opinion of a broadminded non-specialist (which after all is a large part of the audience for any Science paper), and so I wrote my review from that perspective and said so. Or for instance, if you’re aware that you have a minority or idiosyncratic view on some issue relevant to the ms, or you take one side of a controversy on which the ms takes the other side, it’s good to tell the editor where you’re coming from. In my experience, editors always welcome this sort of context.
Do not say anywhere in your comments to the author what decision you recommended to the editor, no matter what decision you recommended and no matter how sure you are that the editor will follow your recommendation. It’s the editor’s job to make the decision, not yours. Sharing your decision recommendation with the author risks undermining the editor. It really annoys editors when you do this!
Do not stop reading the ms when you come to the first mistake so serious and unfixable as to render the ms unpublishable, unless your goal is to develop a reputation as a lazy reviewer so that people will stop asking you to review stuff. You agreed to review the whole ms–do so. Further, you could be wrong about whether the mistake is a mistake, about how serious it is, and/or about whether it’s fixable in a revision. In which case, you’re going to look pretty silly for stopping where you did. (I’ve seen this happen, by the way. Just as authors really do sometimes make really bad, how-could-they-do-that mistakes, reviewers really do sometimes mistakenly think that authors have much such mistakes!)
Don’t just check for technical mistakes, unless that’s all the journal asks you to check for. You’re free to question anything about the ms. For instance, just because the author says the ms is about topic X doesn’t mean it is! If you think the ms would work much better if reframed as an ms about topic Y, say so.
UPDATE: I should’ve remembered to say this in the original post: provide citations as needed in your comments to the author, to back up your claims. For instance, if the authors claim that previous work shows X, and you think previous work shows nothing of the sort, provide citations to back your claim.
Don’t be deferential, and don’t worry unduly about making a mistake. Even if you’re a grad student, or it’s your first review, and no matter how famous the author. If you really don’t think you know enough to do a review, don’t agree to do it. But once you’ve agreed, just do it, to the best of your ability. Nobody expects you to be infallible. Remember, you’re just providing advice, the fate of the ms is in the editor’s hands, not yours. So while you don’t want to write your review in an arrogant way, there’s no need to hedge every criticism with “I could be wrong, but…” or “This is just my opinion, but…”. And even if your review ends up disagreeing substantially with the other reviews, that does not mean your review was incorrect or bad or unhelpful. It’s rare for all reviews to agree in every detail, and fairly common for them to disagree substantially, even when they’re all written by experienced reviewers. You were asked for your professional opinion, and you gave it. The fact that others have different professional opinions is neither here nor there.
It is not ok to share an ms you’ve agreed to review with anyone, not even your labmates or your supervisor. No, you can’t even summarize it for them, or talk about it in general terms, or talk about it without saying who wrote it, or etc.
If you feel like you need to consult a colleague in order to complete the review (e.g., you need to consult on some highly technical issue), you first need to email the editor and get permission. Alternatively, if there is some technical aspect of the ms you don’t feel qualified to review, you can just indicate as much in your comments to the author and editor, and just review the remainder of the ms. That’s not all that unusual. In all likelihood, the editor and/or another reviewer has the expertise to evaluate whatever aspect of the ms you can’t. It’s not unusual for editors to choose reviewers with different, complementary expertise so that collectively (but not individually) they can evaluate the whole ms.
Turn in your review on time. Ideally early. It’s going to take you the same amount of time to do no matter when you actually do it, and it’s going to take time away from some other activity no matter when you actually do it. So unless you really do have some other more pressing deadline looming over you, you may as well do the editors and your colleagues a favor and do the review as early as you can. And if you have so many pressing deadlines that you find yourself unable to complete the review on time, why did you agree to do it in the first place?
When you agree to review an ms, you are also agreeing to review any invited revisions of the ms (that’s my view, anyway). So when you get asked to review an invited revision, agree! An ms that was rejected, but that gets resubmitted as a new ms (and thus has a different tracking number), is a different story. It’s a new ms, so you’re under no more (or less) obligation to agree to review it than you would be for any new ms.