The comments on Jeremy’s post last week on manuscripts often getting the same reviewers at multiple journals got me wondering about something that is related: what’s the range on the number of reviewers a paper has in a single round of review at a single journal? And what about the number of reviewers for a paper across all rounds of review at a single journal? And, if it’s possible to figure out, what about the number of reviewers for a single journal across all rounds of review across all journals?
First, a poll for that first question (note: for purposes of this poll, don’t count the associate editor, even if she/he did a substantive review):
For me, the answer is 5. I’m guessing that what happened is that the associate editor invited 5 people, assuming 2-3 would accept, and hit some sort of reviewer jackpot where they all agreed to the review. Perhaps because of this experience, I never ask more than 3 people at a time when I’m handling a paper as an AE. (This is only relevant for me at Ecology & Evolution. Fortunately for me, AmNat’s editorial office handles all the emails related to review requests, so I don’t need to make that decision with manuscripts for them.) The downside to not asking more people at once is that it can slow down the process of finding two reviewers. So, there’s a tradeoff.
Now, the second question (again, for purposes of this poll, don’t count the associate editor, even if she/he did a substantive review):
To be clear, I am not referring to cases where a manuscript bounces to another journal. Sometimes the number of referees increases during revisions. For example, the paper may get sent to one of the original reviewers and to a new reviewer. I’ve heard one horror story of where a high-profile journal did this through many rounds of review.
The last poll question will probably be harder for some people to answer/estimate, but I’m still curious:
For this, I realize you might have to guess a bit more, but it will still be interesting to see.
I’ll be curious to see what others have experienced!
Dynamic Ecology 
“Perhaps because of this experience, I never ask more than 3 people at a time when I’m handling a paper as an AE”
At AEM we tell the system how many reviewers we want. Three is the default. What I do is invite ~6 potential reviewers and if it hits 3 then it tells the others that they aren’t needed. Usually I’ll have 2 people say that they can’t review and 1 never respond. Occasionally, I’ll invite 7 and only have 1 person agree or even respond. In those cases I end up doing the second review. I find this really speeds things up.
Your post and my strategy leads to a follow up question of how many reviews people want. It probably takes longer to get 3 reviews submitted and then forces the authors to satisfy an extra person. But in theory, the paper should become a smidge better and give the authors more constructive feedback.
Excellent question. My initial instinct is to say 2-3, but I’m not sure which of those two I prefer. As you say, 3 would give it an extra set of eyes which should make it better, but also takes longer and means that there’s an extra person who could perhaps be not constructive.
Another issue with 3 as the preference is the increased burden on the reviewer pool. I feel like I already get lots of review requests!
Seven certainly seems silly on first glance, but I totally understand how this can happen. As an AE, I feel under a fair bit of pressure to make sure I secure the minimum number of reviewers in a timely manner. In my experience, getting a large proportion of folks to agree to review is quite rare, so I too pad the number I typically invite (or, at one journal, assemble for an editorial staff member to invite). Once in a while, an AE will get exuberant with that list *and* they’ll hit the jackpot in terms of the number that agree — if those all come in around the same time, you get the situation you painted.
Some other thoughts: (i) At least on Twitter, I encounter much grousing about how long a particular paper took to be reviewed. It seems pretty obvious, but: if you (the general you; not you, Megan) want your papers to be reviewed more quickly on average, then you need to be agreeing to more review requests on average (and getting them in on time! ;). If I have to routinely send out three rounds of 2-3 invitations just to get at least two folks to finally agree to review, that is going to add to the time that manuscript is in review. (ii) Again on Twitter, I encounter much self-professed love for open access journals combined with an apparent disregard for IFs, etc. Heartwarming in theory… but, my experience as AE doesn’t really hold this up in practice. That is, I routinely struggled to get folks to agree to review for one of the gold standard OA journals. It’s one of the reasons I prematurely gave up the position. Conversely, it’s a much happier process at one of the more traditional/higher IF journals I’m now associated with. I know I could use more data here, but my experiences match those of other associate and senior editors I’ve talked with.
My experience from the other side of things reinforces yours. Like many established scientists, I get several times more requests to review than papers I submit. So I can pick and choose among review requests while still doing more than my fair share. Which in practice means that I pretty much only review for established, selective journals, just because they’re much more likely to send me mss that are either good, or bad in interesting ways. If Plos One or the various obscure specialized journals that send me review requests ever send me an ms that’s sounds like it’s in my area of expertise and interesting enough to be worth reading, I’ll happily review it. But they haven’t yet.
Wait, so you get lots of requests from “obscure specialized journals” and not one has been in your area and interesting to you????
Well, I wouldn’t say I get lots of requests from such journals. The majority of requests to review that I receive come from journals like Ecology Letters, Ecology, and Am Nat. But I get some.
I also turn down many requests to review from journals like EcoLetts, Ecology, and Am Nat. But I decline a lower percentage of review requests from those sorts of journals, and a lower percentage of my declines are because the paper is outside my area of expertise or because I don’t think it would be interesting to read.
Here’s a compilation of who’s asked me to review recently, and how I’ve responded. The overall picture wouldn’t change much if I brought the post up to date: https://dynamicecology.wordpress.com/2013/05/15/whos-asked-me-to-review-recently-and-how-ive-responded/
For «What is the largest number of reviewers you’ve had for a single manuscript across all rounds of review across all journals?» the answer is 12. It was for this paper: http://onlinelibrary.wiley.com/doi/10.1002/gbc.20046/full
When I started as an AE, the EIC stressed limiting the burden on the reviewer pool. As a consequence, I would only ask 2 people initially, and chasing down reviewers didn’t often add significantly to the time in review (when it did, starting with three wouldn’t have helped; some manuscripts are just cursed by timing, subfield, or bad luck). Similarly, when I have discovered a journal wasted my time by tricking me into doing (or at least starting) a review it didn’t really need, it goes on my list of Journals I Will Never Review For Again.
Re: the optimal # of reviewers (^^). I think that two substantive reviews plus the careful evaluation of those reviews by a thoughtful AE mean that the marginal gains of a third reviewer are usually negligible.
Pingback: What makes a great scientist? Part 5: a bit of luck | Ramblings of an ecóloga