Recruiting peer reviewers: the personal touch

I recently received an automated email asking me to review an NSF grant. And immediately after I received that email, I got a surprise: a personal email from Doug Levey, a Program Director responsible for population and community ecology at the NSF Division of Environmental Biology. Yes, a personal email–addressed only to me, clearly not following any template, explaining specifically why NSF really wanted me, Jeremy Fox, to review this grant. I’d have agreed to do the review even without the personal note (the grant really was right up my alley, as the personal note indicated). But I was surprised and intrigued to get a personal email from a Program Director. I mean, surely Program Directors have lots of better things to do with their time than send personal emails to grant referees! I had a lot of questions: Why is NSF using personal emails to solicit for grant referees? Is NSF DEB sending such emails to everyone? Or only to certain people, and if so, why those people? So I asked Doug if he could satisfy my curiosity, and he was kind enough to reply. I thought his reply would be of interest to many readers, and so with Doug’s permission I’ve paraphrased it below.

***********

Doug sends personal notes to about 40% of the people from whom he requests reviews. It’s prompted by one of two situations: lots of people have already declined to review the grant or failed to respond, or because he really wants a review from a particular person who has specialist expertise making them uniquely qualified to review the grant.

Doug reports that he typically needs to send 3-4 review requests for every review he receives. These are known as “ad hoc reviews” and are in addition to the 3 reviews that are always provided by the folks who serve on review panels (“panel reviews”). Population and Community Ecology shoots for 3 ad hoc and 3 panel reviews for each of ~150 proposals/panel. With several panels/year and that response rate, finding enough reviewers isn’t easy, given the size of the pool of potential reviewers.

Personal notes work–they increase the positive response rate by ~30%. Whether they work well enough to justify the extra time it takes to write them is an open question.

***************

I want to thank Doug for taking the time to satisfy my curiosity. I think the fact that Doug is doing this is a dramatic illustration of how much pressure the peer review system is under, and how that pressure has grown over time (NSF Program Directors didn’t used to see any need to send personal notes!) And the fact that the personal notes work is a nice illustration of how it is simply not true that people are “too busy” to review. Yes, everyone has lots of demands on their time–and we choose which demands to agree to. That personal notes can affect our choices is a reminder that we do have choices. Doug Levey is choosing to spend a lot of extra time writing people personal notes in an attempt to make sure that every NSF grant gets the full and fair evaluation it deserves. Something to keep in mind next time you’re asked to review for NSF and feel the urge to decline because you’re “too busy”.

7 thoughts on “Recruiting peer reviewers: the personal touch

  1. Good point Jeremy. Brings up the issue with regards to reviewing papers as well… someone once told me that a good metric to use is 3:1; review 3 papers for every 1 you submit (not for every one that’s accepted, one for every one that’s submitted- so that paper that took you 3 submissions to three different journals to FINALLY get published means you’re on the hook for 9 reviews). I think the proposed ratio comes from the idea that it tries to reflect acceptance rates at many journals. Any thoughts on this? I know I strive for the 3:1, but I think I’m shooting a little below at the moment.

    I also wonder, given success rates of funding at institutions line NSF (I’m Canadian, but I understand it’s pretty slim), or, for example, NSERC post-docs (which had something like a 20% success rate last year) that means that the ratio of reivews to submissions should be more like 5:1 for every grant you submit?

    • That everyone should try to review 2-3 papers for every one they submit has been called the “golden rule of reviewing”. I think Mark McPeek, the EiC of Am Nat, did an editorial using that phrase a little while back.

      Owen Petchey and I are working on an article using a bunch of anonymized data to look at whether ecologists do in fact obey this rule on average, and how much individuals vary around the average. Obliging people to obey this rule is the basis of our idea of “PubCreds”.

      How many reviews you need to do to satisfy the golden rule depends on how many each paper or grant typically receives, not it’s likelihood of being published or funded. If you submit an NSF grant, it gets 3 ad hoc reviews, so I’d say you ought to agree to do 3 NSF reviews for every NSF grant you submit. As you noted with journal articles, even rejected papers and grants get reviewed, so every time you submit something, you’re drawing on the peer review system, and so you ought to contribute to the system accordingly.

      • Looking forward to the paper. I did a very informal survey of colleagues a couple years ago, and found that the golden rule among those I spoke with was not much more than something nice to think about.

        Time to go do a review 😉

      • I like that personal touch – good for the Program Director in taking the time to do it.

        The McPeek editorial you refer to is:
        McPeek, M. A., D. L. DeAngelis, R. G. Shaw, A. J. Moore, M. D. Rausher, D. R. Strong, A. M. Ellison, L. Barrett, L. Rieseberg, M. D. Breed, J. Sullivan, C. W. Osenberg, M. Holyoak, and M. A. Elgar. 2009. The golden rule of reviewing. American Naturalist 173:E155-E158.

        Something I’ve decided to do over the past year is to stop reviewing for journals run by mega-publishers (e.g., Elsevier, Wiley), and instead review for non-profit publishers (e.g., the journal for which I volunteer as Journal Manager, The Canadian Field-Naturalist – shameless plug http://www.canadianfieldnaturalist.ca/index.php/cfn). Why should I volunteer my precious time so a company can make kazillions of dollars? Given our finite (and shrinking) amount of available time, we have to say no to some requests. What criteria do we use to decide what requests to decline? I use the for-profit vs non-profit journal criteria as one of my decision factors.

      • I think it’s becoming more common for scientists to take publisher policies into account when deciding who to review for. I don’t myself, in part because it seems like a blunt instrument to me. In boycotting Wiley, are you boycotting the society journals that Wiley is contracted to publish, like the journals of the British Ecological Society and the Society for the Study of Evolution?

  2. Drug Monkey asked on twitter yesterday: “Accepting a manuscript review assignment: X% specific interest, Y% general obligation, Z% obligation to specific journal or editor?” That was an interesting way of thinking about it to me, in part because I think I say yes if I am either specifically interested or feel obliged generally or to that specific journal/editor — or, in this case, funding agency. But I’ve had to start shifting things a bit to say no more, in part because of more review requests, and in part because there are only so many I can accept while still doing a good job. I accepted 2 of the 3 requests I got from NSF this round, by the way.

    By the way, people’s replies to Drug Monkey’s tweet were interesting (and variable). You can see them here:
    https://twitter.com/drugmonkeyblog
    but you’ll have to scroll down a bit.

    • My own decision-making process doesn’t quite fit that framework. I definitely feel general obligation to do reviews, but it doesn’t affect my decision on whether or not to agree to any given review request. I’m fortunate enough to be asked to do enough reviewing, relative to how much I submit, that I can just agree to review things that look interesting and still obey the “golden rule” (and then some). But if I weren’t asked to review so much, my general obligation would take over and I’d say yes to everything I was asked to do.

Leave a Reply to Jeremy Fox Cancel reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.