In our recent compilation of ‘advice’ posts, I solicited requests for new topics on which readers want advice. Commenter Colin asked for advice on writing peer reviews. Ask and ye shall receive!
A bit of googling will turn up many good sources of advice on how to write a good peer review. Here are a few:
The Ecological Society of America’s official instructions to reviewers for its selective journals. Goes into detail on what the ESA’s journals want from reviewers.
How to be “professionally judgmental” from The Research Whisperer.
Tips for reviewers from the Australian Academy of Science.
I’ll try not to duplicate these and other sources of advice more than necessary. You should read those other sources in addition to my advice.
Advice for before you start writing your review
Remember, the purpose of your review is to provide rigorous but fair feedback, which will help the author improve the ms, and help the editor make a decision on the ms.
Read the ms carefully and critically. Not that you should set out with the goal of finding flaws. But it’s the author’s job to convince you that the work reported in the ms is technically sound, interesting, and important. Read the ms in a skeptical-but-open-to-being-convinced frame of mind. And do read carefully–remember, you and the other reviewers may well be the only people who ever read and pay attention to every line of the ms (seriously).
Remember that you are free to question any aspect of the ms. Don’t take anything in the ms for granted, including things like the framing of the question, the accuracy and completeness of the background review in the Introduction, the relevance to larger issues claimed in the Discussion, etc. Don’t read the Methods and Results carefully while skimming the rest of the ms.
Don’t worry about who the author is. Don’t presume that anything from Dr. Famous’ lab must be right (or wrong!), or that anything you’re reviewing for Nature must be brilliant (or oversold rubbish!)
Check and see if the journal has any instructions for reviewers. If they do, follow them.
If you’re not familiar with the journal, have a glance at a recent issue so you have some sense of what sort of ms they’re looking for. One thing you’ll have to comment on is whether the ms is a good “fit” for the journal.
The three parts of a review
For most journals, there are three parts to a review: an online form you have to complete, comments that only go to the editor but not the authors, and comments that go to the authors (which the editor will read as well).
Online form
The online form is the least important. What matters are your comments to the editor and author. So don’t agonize over exactly what score to give the ms on the “originality” scale in the online form or whatever.
Comments to the author
I write my comments to the authors first, because comments to the editor need to reflect and summarize comments to the author.
Begin with a one-paragraph summary of the ms–what was done, the main results, and the main take-home message. This helps demonstrate to the author (and editor) that you read and understood the ms.
Next, summarize the biggest strengths of the ms. Find something good to say even if the ms is terrible. Conversely, if you thought the ms was really great, it’s fine to say so to the authors. In general, the overall impression your review gives to the authors should be consistent with your overall impression of the ms.
Then summarize your main criticisms and suggestions for improvement.
Then provide a detailed list of comments. Number them so the authors can refer to them easily in the event they’re invited to submit a revision (in which case they’ll need to describe to the editor their response to every comment you made). Refer to page and line numbers in your comments wherever possible. I like to organize my comments into “major comments” and “minor comments”. The more detailed your comments are, the better. Detailed comments show that you read the ms carefully and encourage the editor and authors to take your comments seriously. Non-specific comments like “the ms isn’t very interesting” or “the ms should be 20% shorter” are useless without elaboration. Why isn’t it interesting? What material should be cut? Etc. If you have a suggestion as to how to address a problem you’ve identified, provide it.
If some of your comments are more along the lines of suggestions–just food for thought, something the authors might want to consider–say so. (UPDATE: on reflection, I should’ve emphasized how important this is. A very common mistake in peer review is to criticize an author for not doing something the way you personally would have done it, or for not doing something in the “best” way, even though the way the author did it is perfectly adequate. There’s often more than one way to skin a cat when you’re doing science, and as a reviewer you need to recognize that.)
If you think the authors have made a mistake so basic or serious that you can hardly believe anyone could’ve made it, it’s fine to hedge your criticism by saying something like “Apologies if I’ve totally misunderstood, but the claim on line XXX makes no sense…” Then, after you explain the mistake, tell the authors to either correct the mistake or clarify the writing. This is a useful way of writing even if you’re quite confident that you have understood perfectly well, and that the authors really have made a totally boneheaded mistake.
If your comments are really negative, be polite and professional about it (not always easy because it’s really annoying to have to waste your time reviewing a very poor ms). But don’t pull your punches–you need to be clear, honest, and forthright about what the problems are. Yes, even if the lead author is a student. In deciding to submit a ms to peer review, the authors have indicated that, in their view, their ms is ready to be evaluated (and indeed, to be published!), and so are prepared for any evaluation the ms might receive. And if they’re not prepared for any evaluation the ms might receive, that is their fault, not yours, and there’s nothing you can do about it. You should write as a professional addressing a fellow professional. If the author you’re addressing can’t take that, tough. It’s your job to review the ms, not to help the author feel good. You do no one any favors by trying to make your criticisms sound less negative than they actually are. If you feel really badly about being negative, include some positive suggestions for how the authors could build on, improve, or redo the work reported in the ms, even if the ms itself isn’t salvageable.
There is no particular length you should aim for. Your review should be as long or as short as necessary to say everything you need to say. I’ve written reviews that ran to several single-spaced pages, and I have colleagues who’ve done the same.
When recommending revisions, take care not to recommend changes that, individually or collectively, would be unreasonable or impossible to make. For instance, if the ms is already about as long as the journal permits, do not recommend adding a bunch of material without also specifying what material you’d like to see cut. Otherwise the authors will (quite rightly) respond “We appreciate the suggestion to add discussion of issues X, Y, and Z, but as the ms was already lengthy we were not able to do so.”
If you’re not sure about a comment–say, you think there might be a problem with the stats, but you’re not sure–say so, and encourage the authors to clarify.
While there’s such a thing as being too nitpicky–rare is the ms that reports perfect science!–when in doubt err on the side of being more nitpicky rather than less.
It’s good to get in the habit of checking at least the key references, if not all of them. Make sure they say what the authors claim. Miscitations are common, sadly, especially in the Introduction.
Point out any grammatical errors or typos you find, unless there are a lot of them (e.g., because the author is not a native English speaker), in which case just recommend that the ms would benefit from careful editing. If the ms is by a non-native English speaker, note in your comments to the author and editor whether there were any places in which you found the ms difficult to understand, or whether the grammatical problems were merely cosmetic.
Most journals leave it up to you whether to sign your review, with a default to anonymity (I think Nature insists that its reviewers remain anonymous, if I recall correctly) Some people always sign their reviews, some never do, some do only sometimes, usually when they know the author or when the review is positive. It’s really up to you. I don’t want to debate whether or not it should be that way, I’m just telling you how it is.
Comments to the editor
Remember, the authors won’t see your comments to the editor, so don’t mince words. Don’t be rude about the authors themselves, of course. But if you think the ms is horrendous, then the editor needs to know that. And if you think the ms is the greatest thing since sliced bread, the editor needs to know that!
If you’re not sure whether or not to recommend acceptance, minor revision, major revision, rejection, or whatever, say so in your comments to the editor, and explain why you’re uncertain. That’s no problem. A good editor won’t really care much what decision you recommend anyway, they’ll mostly care about the comments on which your recommendation is based.
It’s often useful to tell the editor about the perspective from which you read the ms. It’s always good to give the editor context to help him or her fully understand your comments. For instance, Science once asked me to review an experimental evolution ms on the evolution of evolvability. I have a passing interest in this topic, but I’m very far from an expert. Nor am I an expert on the experimental system used. I assume Science just wanted the opinion of a broadminded non-specialist (which after all is a large part of the audience for any Science paper), and so I wrote my review from that perspective and said so. Or for instance, if you’re aware that you have a minority or idiosyncratic view on some issue relevant to the ms, or you take one side of a controversy on which the ms takes the other side, it’s good to tell the editor where you’re coming from. In my experience, editors always welcome this sort of context.
Other tips
Do not say anywhere in your comments to the author what decision you recommended to the editor, no matter what decision you recommended and no matter how sure you are that the editor will follow your recommendation. It’s the editor’s job to make the decision, not yours. Sharing your decision recommendation with the author risks undermining the editor. It really annoys editors when you do this!
Do not stop reading the ms when you come to the first mistake so serious and unfixable as to render the ms unpublishable, unless your goal is to develop a reputation as a lazy reviewer so that people will stop asking you to review stuff. You agreed to review the whole ms–do so. Further, you could be wrong about whether the mistake is a mistake, about how serious it is, and/or about whether it’s fixable in a revision. In which case, you’re going to look pretty silly for stopping where you did. (I’ve seen this happen, by the way. Just as authors really do sometimes make really bad, how-could-they-do-that mistakes, reviewers really do sometimes mistakenly think that authors have much such mistakes!)
Don’t just check for technical mistakes, unless that’s all the journal asks you to check for. You’re free to question anything about the ms. For instance, just because the author says the ms is about topic X doesn’t mean it is! If you think the ms would work much better if reframed as an ms about topic Y, say so.
UPDATE: I should’ve remembered to say this in the original post: provide citations as needed in your comments to the author, to back up your claims. For instance, if the authors claim that previous work shows X, and you think previous work shows nothing of the sort, provide citations to back your claim.
Don’t be deferential, and don’t worry unduly about making a mistake. Even if you’re a grad student, or it’s your first review, and no matter how famous the author. If you really don’t think you know enough to do a review, don’t agree to do it. But once you’ve agreed, just do it, to the best of your ability. Nobody expects you to be infallible. Remember, you’re just providing advice, the fate of the ms is in the editor’s hands, not yours. So while you don’t want to write your review in an arrogant way, there’s no need to hedge every criticism with “I could be wrong, but…” or “This is just my opinion, but…”. And even if your review ends up disagreeing substantially with the other reviews, that does not mean your review was incorrect or bad or unhelpful. It’s rare for all reviews to agree in every detail, and fairly common for them to disagree substantially, even when they’re all written by experienced reviewers. You were asked for your professional opinion, and you gave it. The fact that others have different professional opinions is neither here nor there.
It is not ok to share an ms you’ve agreed to review with anyone, not even your labmates or your supervisor. No, you can’t even summarize it for them, or talk about it in general terms, or talk about it without saying who wrote it, or etc.
If you feel like you need to consult a colleague in order to complete the review (e.g., you need to consult on some highly technical issue), you first need to email the editor and get permission. Alternatively, if there is some technical aspect of the ms you don’t feel qualified to review, you can just indicate as much in your comments to the author and editor, and just review the remainder of the ms. That’s not all that unusual. In all likelihood, the editor and/or another reviewer has the expertise to evaluate whatever aspect of the ms you can’t. It’s not unusual for editors to choose reviewers with different, complementary expertise so that collectively (but not individually) they can evaluate the whole ms.
Turn in your review on time. Ideally early. It’s going to take you the same amount of time to do no matter when you actually do it, and it’s going to take time away from some other activity no matter when you actually do it. So unless you really do have some other more pressing deadline looming over you, you may as well do the editors and your colleagues a favor and do the review as early as you can. And if you have so many pressing deadlines that you find yourself unable to complete the review on time, why did you agree to do it in the first place?
When you agree to review an ms, you are also agreeing to review any invited revisions of the ms (that’s my view, anyway). So when you get asked to review an invited revision, agree! An ms that was rejected, but that gets resubmitted as a new ms (and thus has a different tracking number), is a different story. It’s a new ms, so you’re under no more (or less) obligation to agree to review it than you would be for any new ms.
Dynamic Ecology 
Happy New Year and thanks for an excellent summary, Jeremy.
I’ll add that good reviewers include citations in their review. This helps both the authors and the editor see any relevant work that may have been missed (and I’ve seen reviews that simply state ‘this article is missing important literature’), or backs up the reviewers’ opinions, making them appear more objective and constructive. This can be important for salving egos if it’s a critical review.
I might send a link to this post with every paper I submit in the future 😉 Alternatively, I’ll just put you down as a preferred reviewer. I don’t mind if a reviewer rips any of my work a new one, as long as they clearly say why it needs a new one and where that new one should be positioned.
Yes, good reviewers do indeed include citations. That’s an excellent point.
Be aware that if you put me down as a preferred reviewer in the next few months, I might say no. I am living off my accumulated “PubCred” balance for a while and only agreeing to review papers that sound really interesting (judging from the title and abstract). 😉
(Editor’s note: comment mistakenly posted to another thread, moved here)
I wish I’d seen this a few days ago. My PI gave me my first ms to review and I have a meeting with him today about it. I now want to go back and re-read the ms with all this in mind. But, alas, I am out of time! Next ms, right?
Great post!
I’d be interested in your thoughts on reviewing the same paper for multiple journals. When reviewing a revised manuscript (that is, for the same journal for which I wrote the original review), I usually read through my original review and the other reviews and the response to reviewers before rereading the manuscript. What I’m reviewing a paper that I get to review from two different journals, I always waffle on whether to use that same approach, or whether to just treat it like it’s the first time I’ve seen the paper. I usually end up reading it, skimming the old reviews, and then trying to review it more-or-less like a first time review. How do you handle these sorts of reviews?
Sometimes when doing the above, it’s necessary to admit that I’m reviewing the paper for the second (or third!) time. I always wondered if that ends up influencing the editor’s decision. I don’t think it should, but it seems like it could potentially be held against the paper.
Thanks Meg, glad you liked it!
When asked to review the same paper for a second journal, I usually let the handling editor know in my comments that I’ve reviewed the ms previously, but otherwise I proceed as usual. I actually avoid looking back at my old review (and I also tell the handling editor that I’ve avoided doing so), in an effort to read the ms with fresh eyes. Yes, in revealing that I’ve reviewed the ms before another journal, I’ve revealed that the ms has been rejected before, which I suppose could be held against the ms. But in practice I don’t think it is.
Unless of course, if the second submission is exactly the same as the first… i this case (it has happened) I will simply submit the first review with a stern warning to the editor. Such a paper is DOA – nothing antagonizes a reviewer more than willful blindness to his/her concerns!
Yeah, if asked to review something I’ve previously reviewed for another journal, I always charitably assume it’s been revised to address my concerns, even if the title and abstract are identical. And then of course, I usually read it and find out that it’s exactly the same (or worse, that the authors tried to address multiple pages of criticisms with a couple of cosmetic revisions). And then I declare “open season”: I let both the authors and editors know, in no uncertain terms, that I’ve already seen the ms once and am seriously unimpressed with the authors’ behavior.
Not infrequently, I’m re-reviewing a manuscript that was previously rejected from another journal on the basis of perceived impact, rather than any major scientific flaws. This isn’t any fault of the authors, and, having been in the situation myself a few times, I’m sympathetic to not wanting to chum the waters against them.
That said, I will check out my “old” review after writing the first draft of the new one, just to see if there were any major issues (failure to acknowledge previous work, or methodological flaw, or whatever) that are still unresolved.
Your experiences have been very different from mine! Every time I’ve been asked to review something a second time for another journal (and in one case, a third time for a third journal!), it was because of serious scientific flaws.
I’ve had a mixture of the two — cases where it was rejected for a serious flaw (and where that is not at all addressed in the revision) and cases where it was rejected for fit/sexiness/impact. I agree that it’s really frustrating to deal with the former.
Great post, Jeremy, especially the admonition to leave the recommendations to the editor! It’s very frustrating when reviewers recommend publication in their comments to the authors, I’ve had to deal with appeals based primarily on a rogue reviewer taking this liberty. Anything you can do to educate reviewers is appreciated by editors far and wide.
I would only add as an emphasis that reviewers need to be accountable for their criticisms. Reviewers some times level sweeping judgments without any basis in fact or literature, This echoes an earlier comment that good referees provide citations – indeed they do, along with solid reasons for their criticisms. All of this can be done constructively.
I would be careful about the pat “please have this reviewed by a native speaker” that seems to get thrown on to almost any paper from my team where the first author does not have an Englis name. If there is one English name on the author list (who isn’t a “big deal” person), assume that part of that person’s job is to edit the manuscript. There may be an errant comma or awkward phrase or two, but it’s in much better shape than when they got it. Plus given the racial make-up of the US, there are plenty of people with non-English names who are native speakers (and my American friends have received these comments).
I would recommend saying something like “This paper would benefit from a more thorough edit to improve some awkward phrasing and readability of the manuscript.” This comment acknowledges the fact that it has likely been edited by a native speaker while giving them more room for a deeper edit.
Well said!
I’ve been frustrated by this type of reviewer comment in the past. Although I don’t have an English-sounding name and I am based at a University in a non-English country, I am a native English speaker. So it boils my blood when I’m told that the ms “should be reviewed by a native speaker”.
I don’t expect sentence-by-sentence coaching, but simple pointers – like sentences are often too long or tenses are mixed – give me something to base my revisions on. Condescending blanket-statements, on the other hand, are not constructive at all.
As I said in the post, I think it’s important for referees to indicate if grammatical problems are merely cosmetic, or make the ms difficult to understand. If they’re merely cosmetic, my own view is that correcting them probably isn’t essential, and so editing from a native speaker isn’t essential. I certainly don’t routinely recommend editing by a native English speaker to fix minor grammatical errors (even if numerous) that don’t inhibit understanding of the ms.
But if the grammatical problems are sufficiently serious as to make the ms difficult to understand, I’m afraid that the reviewers can’t fix it (because they can’t understand the ms well enough to do so). Further, it’s not their job to fix it (they’re scientific reviewers, not copy editors). Whose job it should be is another conversation, but in such cases I don’t see what else reviewers are supposed to say besides “the ms needs editing by a native English speaker”.
Nice summary,
I am curious on how do you approach conflict of interests, for example when asked to review a paper from an author with wich you have coauthored papers recently. I usually asume the editor already knows that fact, and I accept the review and just mention the potential conflict of interests in the comments to the editor, but I always wonder if I should warn the editor beforehand.
Ignasi.
No, don’t assume the editor knows about any conflicts of interest besides employment at the same institution. Check the journal’s policies, or alert the editor to the potential conflict before agreeing to do the review.
It’s important all potential conflicts of interest are declared, and that this is done at the time a journal approaches you to review. It’s then up to the editor whether they’d still like a review from you.
Some journals set out what they consider to be conflicts. If they don’t, it can be difficult, especially for those new to reviewing, to know what needs to be declared. Generally, the following should be – close personal or financial relationships, working in the same department or institute as any of the authors (journal checks may have missed this; there have even been cases of authors being asked to review their own manuscripts) or if applying for a job or about to take up a position there, being a close collaborator or joint grant holder, being a recent mentor or mentee. What journals consider ‘recent’ varies, but up to 3-5 years is a useful guide. Any involvement with the work should also be declared. If there’s something you think might be a conflict but isn’t included in the journal’s list, it’s good to let the journal know anyway – always aim for transparency. If any time you feel you wouldn’t be able to provide a fair review – which could be for a number of reasons – let the journal know. Everyone is human and editors will appreciate your honesty and fairness. If a potential conflict becomes apparent only after you’ve received a manuscript for review, let the journal know.
At COPE (where I’m on the Council, http://publicationethics.org/) we’re currently producing Ethical Guidelines for Peer Reviewers, which will apply across disciplines. The final draft will soon be ready, and then we’ll probably put this up on the website for open commenting before finalising the guidelines. Be great to have as many people as possible give feedback.
Im finding it more and more frustrating to review articles these days (although I still do so about twice a month). While I sometimes have frustrations with the manuscripts themselves, I am very often frustrated with the short review turnarounds that seem to be expected by most journals these days. I understand it is a competitive market out there for papers these days and many journals ‘sell’ rapid turnaround heavily to prospective authors. However, since I am doing this as a courtesy basically to reciprocate for the papers I submit, about half the time something comes up in my schedule, and I am not able to meet the turnaround times requested. PLoS regularly (as a default) asks for your review within 10 days. Taking the time to thoughtfully review, critique and write a review as you point out is not quick endeavor, yet many AEs are rather pushy on this. To often I feel rushed to meet the deadlines by AEs, rather than praised for taking the time for a thorough (if somewhat late) review.
Additionally, I sometimes wonder why more journals don’t prescreen more articles. Even given the discussion above, in my opinion, papers with extreme language impediments should not be wasting my time. Science is a precise endeavor, with most fields employing terminologies and accepted norms that have been developed for decades and centuries. After the first dozen or so bad sentences or poor word choices that could lead to missinterpretations, im usually tempted to give up rather than try to glean the correct meaning of the authors. It wastes the reviewers time and makes the job much more difficult than it has to be. In my opinion, a quick read by an Editorial Assistant could even weed out many of these problem papers, send them back for further revision before peer review, and save volunteer peer reviewers a lot of headaches.
Pingback: Posterior samples « Sam Clifford
A late comment on an older post, but I had to say thanks. I’ve reviewed plenty of manuscripts and have never thought to mention my perspective in the comments to the editor. I see how that could be useful, and will do that starting today.
Following on from my message above – the draft COPE Ethical Guidelines for Peer Reviewers are now online at the Committee on Publication Ethics http://publicationethics.org/resources/guidelines
We’re inviting comments and suggestions from anyone with an interest in this area, and would like to hear from researchers and authors as well as from editors and publishers. The opportunity for feedback is open until Monday 18 February.
Background: peer reviewers play a central and critical part in the peer-review process, but too often come to the role without any guidance and unaware of their ethical obligations. COPE’s guidelines set out the basic principles and standards to which all peer reviewers should adhere during the peer-review process in research publication. The aim has been to make them generic so that they can be applied across disciplines.
Pingback: When the writing distracts from the research: Reviewing, NES, and NoNES | The Contemplative Mammoth
Pingback: Friday links: Rich Lenski is blogging (!), scientist goes rogue (!), shark-bear (?!), and more | Dynamic Ecology
Pingback: Starting my career as a peer-reviewer | Melinda Trudgen
Pingback: When the language distracts from the science: Reviewing, NES, and NoNES | Aniko's World
Pingback: Happy second birthday to us! | Dynamic Ecology
Pingback: Peer Review | Brittzinator
Pingback: A plea for reasoned reviewers – advice for writing a review | Enquist Lab - The University of Arizona
Hi Jeremy,
Thanks for sharing your thoughts on this. I’ve been asked to review my first ms (I am a student) a few months ago and did my best in providing a good review in a meticulous and objective manner. The review outcome was that the ms needed major revisions, most related to the methodology. I just received the revised manuscript and all of my concerns have been addressed, as well as those of the other reviewer; the authors did make some significant changes in the methodology. I think the paper was well-written in the first place. For my follow-up review, is it really necessary to write a long review again, or is it fine to simply write something like ‘my comments have been addressed and the ms is now fit for publication to the journal’? I mean I can’t find something else that could be fixed or improved! Thanks.
It’s totally fine to simply write “my comments have been addressed” if that’s all you have to say. Be sure to thank the authors (in your comments to them) for revising the ms to address your concerns.
Make sure you don’t say “the ms is now suitable for publication”, or words to that effect, in your comments to the authors. That undermines the editor.
I’m having the opposite problem – I recommended Reject and Resubmit, but the eventual decision was Accept with Major Revisions. The revised manuscript addressed maybe 20% of my initial concerns. Do I re-hash all my original concerns? Do I have to agree to continue reviewing revisions when I got mostly ignored the first time anyway?
If the authors didn’t address your concerns, or if they tried to address them but you don’t think they were successful, you absolutely need to say that to the editor. If the editor agrees with you, then in all likelihood the ms will be rejected and you won’t have to keep reviewing further revisions.
Yes, it’s frustrating to take the time to thoroughly review an ms only for the authors to mostly ignore your comments.
Pingback: To review or not to review, that is the question | The Molecular Ecologist
Thanks for your good explanation and suggestions.
Pingback: Prescriptive reviews are a scourge | Small Pond Science
Pingback: Formatting a CV for a faculty job application | Dynamic Ecology