My goal as a reviewer: pass the Poulin test

As a graduate student, I attended my first infectious disease-themed meeting shortly after receiving the reviews on my first thesis chapter. I was excited about the work, and had sent it to Ecology Letters, which reviewed it but rejected it. I talked about the same study at that meeting. It was a small meeting, and one of the great things about the meeting was getting to interact with senior people in the field. This included Robert Poulin, someone whose work I really admired. I was really excited to get to talk to him! During our conversation, he asked about the status of the work I’d presented at the meeting. I said that it had just been rejected by Ecology Letters and then was about to launch into a vent about the reviewers. As soon as I said (in what I’m sure was an exasperated tone), “One of the reviewers”, he stopped me and said “I was one of the reviewers.” I will be eternally grateful for that.

That moment has stood with me throughout my career. In addition to preventing me from embarrassing myself (more!) in front of him, it taught me a really important lesson about peer review. We complain about Reviewer 2 and shake our fist at that mythical beast, but there’s a decent chance that Reviewer 2 is someone who carefully reviewed the manuscript and thought something was problematic. Or maybe it’s that, with a bit of distance from the work, Reviewer 2 thought the work wasn’t as novel as I did as an author, making rejection from a journal like Ecology Letters completely reasonable.

This interaction taught me an important lesson about how easy it is to think of an anonymous reviewer as an adversary, when there’s a good chance they’re a scientist whose work I admire and whose feedback I would value.

There’s an idea that anonymity leads to animosity. I think that’s more often discussed in terms of the person making the comments – for example, as a reason for the toxic nature of the comments on websites. But it also applies in the other direction – in an anonymous interaction, it can be easy to assume the person writing the comment is unreasonable (unless they think our work is brilliant – then clearly they are totally reasonable!) I think the way the scientific community discusses reviews (including on twitter) probably doesn’t help.

Personally, when I receive reviews, I have to work to put myself in the mindset that these reviews can help my paper, even if they’re negative. There are still occasions where my first reaction is something like “How is it possible for reviewers to be so clueless?!?!” but then, after coming back to the reviews a few weeks later, I realize that the reviewers were pointing out something that we didn’t explain very well or a part of the literature we really should have discussed more or an alternate explanation we hadn’t fully considered.

As I’ve blogged about before, I don’t sign most of my reviews. But I still write them with that interaction I had with Poulin in mind. My goal is to write reviews where, if I ended up in that same situation at a meeting, I would be okay with identifying myself as the reviewer, even in cases where my review was a critical one. In other words, I want to pass what I’ve come to think of as the Poulin test.

I thought about all of this at the ESA meetings this past summer, because I may have accidentally revealed myself as a reviewer to an author of a paper that had recently been rejected. At first, my internal reaction was to cringe. I started to feel anxious. But then I reminded myself that, when writing the review, I had spent time editing it with the Poulin test in mind. I had specifically thought about that interaction from 14 years ago, and about how I might have a similar interaction with the author of the study I was reviewing at the upcoming ESA meeting. I edited it to make sure everything was worded constructively. I think it passed the Poulin test.

I realize this post is a bit rambly, so to make sure the main points are clear: 1) when receiving a critical review, it’s easy to think of an anonymous reviewer as someone who is mean or clueless (or mean AND clueless), but it’s quite possible that the reviewer is a reasonable person who, after carefully reading your manuscript, has a different take on your work than you do. And 2) when writing critical reviews, make sure you are wording them constructively. If you are a senior person and the author is early career, think about whether you would feel okay stating those same criticisms to them directly; if not, it might be a sign that you have more editing to do before hitting “submit”.

Finally, on the subject of potential encounters with reviewers at meetings, it’s worth keeping in mind that, unless the review was signed or the person directly tells you they reviewed the manuscript, your guess about who is Reviewer 2 may be wrong, even if you are absolutely, 100% sure you know:

14 thoughts on “My goal as a reviewer: pass the Poulin test

  1. That’s a very important, but complicated issue. In only a few occasions I signed reviews or revealed my secret identity to an author at a conference. Although I always try to write constructive reviews, even when they need to be critical, most of those experiences were quite negative. Many people tend to take criticism personally and hate you forever for pointing out the weak points of their perfect manuscripts. In the past 10+ years, I never signed a review anymore.

    • I’ve stopped signing my reviews. I used to sign sometimes, if the review was very positive or if I knew the authors well enough to trust they wouldn’t get mad at me over a negative review. But I stopped signing because I don’t want to encourage a norm of signing reviews, or cause anyone else to feel peer pressure to sign reviews. In the rare cases where I still want to identify myself to an author I know personally, I do so in a private email.

      But like Meghan, I still want to pass the Poulin test.

  2. I am an editor in a few journals, and I learned to appreciate an honest review. Reviewers don´t get credit for how many reviews they make, how good they are, etc., but their role is fundamental for scientific validation. As an author, I also learned to appreciate a good, constructive, detailed review, however hard. It is true that not all reviews are fair, but the vast majority are, and it is hard to swallow hard reviews after all the work we have put into the field work and the writing. In my experience, most of the reviews I got worked towards improving the quality of the science.

  3. Via Twitter:

  4. Loved the Poulin test. Robert is the best person I know (unfortunately, yet not personally) to illustrate how to merge criticism and kindness in a review.
    In my Masters` Degree, I was trying to publish my first paper. I had already tried 2 or 3 journals and was getting very disappointed by the rejections. Then I decide a very audacious move (no, there is nothing audacious in it, but looked to me at that time): to reach for the author of my favorite papers and ask for his help. I sent an email to Robert, asking him to review my manuscript, almost certain that he would never waste his time in the manuscript of a completely unknown Brazilian student. He did it.
    His review was extensive, detailed and very critical, but at the same time, extremely encouraging and kind. I did not feel like a failure, I felt challenged by his comments. That’s a skill I want to develop as a reviewer.
    Turned out that, after Robert`s help, the paper was published in a very good journal.

  5. Dear Jeremy, I also see that passing the Pulin test is a must.
    I also think, however, not all reviews are good in quality or intentions. I have experienced many times, as an author and as an associate editor, opinionated reviews that although apparently correct, they are biased and therefore undermine the progress in science. These reviewers should declare from the very beginning their conflict of interests thus the editor can properly gauge their reviews. There isn´t a small amount of researchers with interests out there, because they are too in love with their own hypotheses that cannot see alternative hypotheses with potential or because some studies reject their hypotheses. Editors, in particular, have to be very cautious about potential interests not declared properly. But there is also a big community of researchers that take the time to make good reviews because it is a noble activity and, among other things, reviewing teaches us how to improve ourselves to do better research.

    • I disagree that having a pre-existing scientific opinion about the topic of the paper is a “conflict of interest” for a reviewer. No, it’s not good when a scientist isn’t willing to give up on a pet hypothesis. And yes, it’s good for editors to recognize when a reviewer has strongly-held views on the topic of the paper. But a “conflict of interest” is when a reviewer has some personal or financial stake in the fate of the paper, not a pre-existing scientific view on the topic of the paper.

      I’m sorry to hear that your own experiences with the peer review system have so often been negative. My own experiences as both author and associate editor have been mostly positive though not always positive.

  6. Pingback: Recommended reads #138 | Small Pond Science

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.