A few weeks ago I posted on per-PI success rates at NSF, focusing on success rates per 3 year moving window. That is, what proportion of PIs applying anytime in a 3 year period got at least one grant in that period? I argued that this–not the more widely-discussed per-proposal success rate–is the most relevant measure of “success” if one is concerned about the ability of PIs to establish and sustain their labs. The data were pleasantly surprising, at least relative to the dire numbers one might’ve expected: the current per-PI success rate for core research grants per 3-year period is 35%.
In response, several readers quite reasonably questioned the relevance of those numbers to ecologists and evolutionary biologists. NSF-wide data aren’t specific to DEB and IOS, the NSF divisions to which ecologists and evolutionary biologists mostly apply. In particular, given that DEB and IOS brought in a pre-proposal system a few years ago, one might expect that per-PI success rates at DEB and IOS might be lower than for NSF as a whole.
Well, ask and ye shall receive, apparently! Heck, we didn’t even have to ask! A staff member at the NSF DEB saw my post and went to the trouble to crunch the relevant numbers for the DEB specifically. Here’s the very detailed DEBrief Blog post on the results.
The headline result: the 3-year per-PI success rates on core research grants at DEB (so not DDIGs, conference support, etc.) over 2012-2014 was 15.4%, or 17.5% if you exclude co-PIs. That’s the proportion of PIs who applied at least once anytime in 2012-2014 and were successful at least once. (UPDATE: This number includes authors of pre-proposals, unless I badly misread the linked post.) As I guessed, and as the linked post notes, that’s lower than the NSF-wide success rate. The 3-year per-PI success rate at DEB is down from about 28% back in 2006-2008.
The linked post also provides a lot of other data from DEB, which aid interpretation of the headline number. The most important contextual information is that stimulus funding and the onset of the pre-proposal system seem to have attracted lots of new PIs to DEB. That’s the main reason why per-PI success rate is dropping at DEB. There’s also other contextual information, like number of PIs holding more than one DEB grant at a time*. The data are broken down by gender. And the linked post also has a lot of discussion of caveats and limitations: the effects of different definitions of “PI” on preliminary vs. full proposals, the effect of moving some special programs into DEB from other NSF units, why the analysis “only” goes back to 2006,** why it’s so hard to include pre-proposals in the calculations,*** and more. And it’s all written in a clear, up-front, open style.
I’m curious to hear what commenters think of these data. In particular, that the advent of pre-proposals seems to have attracted more PIs to DEB was a surprise to me–was it a surprise to you? I think this highlights how difficult it can be for any funding agency to aim for a particular success rate. Any policy change you make that’s intended to increase success rates–and many that aren’t–might well have the unintended consequence of attracting more applicants and applications, thereby pushing success rates down. In particular, I wonder if the popular option of cutting award size to increase success rate would be at least partially self-defeating by attracting more applicants and applications.
In conclusion, a big thanks to the folks at NSF DEB for doing this. It’s a lot of work to pull an analysis like this together, using a database that’s not designed to give you these numbers. Nobody at the NSF DEB had to do this (they’ve got lots of other things to do). They just saw that some folks in the research community were interested in these numbers, and so went ahead and chased the numbers down. And published them immediately, in a blog post, for all to see. All of which is awesome.
*Very few. Even if you look over the entire 2006-2014 period, the large majority of PIs who got at least one DEB award got just the one award. So no, success rates at DEB are not low because Dr. Famous is hogging all the money.
**”[2006 was] chosen mainly because that was the span of complete years beyond which the server was getting angry with us for requesting too many records, #overlyhonestmethods”. Yes, that’s a quote.🙂
***Briefly, because of the way NSF’s record-keeping database is designed.