My goal as a reviewer: pass the Poulin test

As a graduate student, I attended my first infectious disease-themed meeting shortly after receiving the reviews on my first thesis chapter. I was excited about the work, and had sent it to Ecology Letters, which reviewed it but rejected it. I talked about the same study at that meeting. It was a small meeting, and one of the great things about the meeting was getting to interact with senior people in the field. This included Robert Poulin, someone whose work I really admired. I was really excited to get to talk to him! During our conversation, he asked about the status of the work I’d presented at the meeting. I said that it had just been rejected by Ecology Letters and then was about to launch into a vent about the reviewers. As soon as I said (in what I’m sure was an exasperated tone), “One of the reviewers”, he stopped me and said “I was one of the reviewers.” I will be eternally grateful for that.

That moment has stood with me throughout my career. In addition to preventing me from embarrassing myself (more!) in front of him, it taught me a really important lesson about peer review. We complain about Reviewer 2 and shake our fist at that mythical beast, but there’s a decent chance that Reviewer 2 is someone who carefully reviewed the manuscript and thought something was problematic. Or maybe it’s that, with a bit of distance from the work, Reviewer 2 thought the work wasn’t as novel as I did as an author, making rejection from a journal like Ecology Letters completely reasonable.

This interaction taught me an important lesson about how easy it is to think of an anonymous reviewer as an adversary, when there’s a good chance they’re a scientist whose work I admire and whose feedback I would value.

There’s an idea that anonymity leads to animosity. I think that’s more often discussed in terms of the person making the comments – for example, as a reason for the toxic nature of the comments on websites. But it also applies in the other direction – in an anonymous interaction, it can be easy to assume the person writing the comment is unreasonable (unless they think our work is brilliant – then clearly they are totally reasonable!) I think the way the scientific community discusses reviews (including on twitter) probably doesn’t help.

Personally, when I receive reviews, I have to work to put myself in the mindset that these reviews can help my paper, even if they’re negative. There are still occasions where my first reaction is something like “How is it possible for reviewers to be so clueless?!?!” but then, after coming back to the reviews a few weeks later, I realize that the reviewers were pointing out something that we didn’t explain very well or a part of the literature we really should have discussed more or an alternate explanation we hadn’t fully considered.

As I’ve blogged about before, I don’t sign most of my reviews. But I still write them with that interaction I had with Poulin in mind. My goal is to write reviews where, if I ended up in that same situation at a meeting, I would be okay with identifying myself as the reviewer, even in cases where my review was a critical one. In other words, I want to pass what I’ve come to think of as the Poulin test.

I thought about all of this at the ESA meetings this past summer, because I may have accidentally revealed myself as a reviewer to an author of a paper that had recently been rejected. At first, my internal reaction was to cringe. I started to feel anxious. But then I reminded myself that, when writing the review, I had spent time editing it with the Poulin test in mind. I had specifically thought about that interaction from 14 years ago, and about how I might have a similar interaction with the author of the study I was reviewing at the upcoming ESA meeting. I edited it to make sure everything was worded constructively. I think it passed the Poulin test.

I realize this post is a bit rambly, so to make sure the main points are clear: 1) when receiving a critical review, it’s easy to think of an anonymous reviewer as someone who is mean or clueless (or mean AND clueless), but it’s quite possible that the reviewer is a reasonable person who, after carefully reading your manuscript, has a different take on your work than you do. And 2) when writing critical reviews, make sure you are wording them constructively. If you are a senior person and the author is early career, think about whether you would feel okay stating those same criticisms to them directly; if not, it might be a sign that you have more editing to do before hitting “submit”.

Finally, on the subject of potential encounters with reviewers at meetings, it’s worth keeping in mind that, unless the review was signed or the person directly tells you they reviewed the manuscript, your guess about who is Reviewer 2 may be wrong, even if you are absolutely, 100% sure you know:

I had a weird encounter with someone at a meeting – she was sure I was reviewer on her paper, and was very angry. I was not. Later at the same meeting, I was telling the story to someone else, and he started laughing. "You won't believe it", he started.

— Timothée Poisot but spookier (@tpoi) September 8, 2018