If you’ve saw this post in the first few hours after it went up, there’ve since been some major updates and corrections!
The title of this post is not a joke (I’ll cop to deliberate provocation…), but it does require some explanation.
My inspiration is this old comment from Chris Klausmeier:
[W]hat I’d really like to know is the success rate per PI, not per grant. That is, are there more people fighting for a constant amount of dollars leading to an increasing “unfunded rate”, or are there roughly the same number of people dividing up the same amount of dollars but with increasing number of proposals per PI?
That’s a great question. After all, if you’re worried about things like the ability of PIs to establish and maintain their labs, isn’t the per-PI success rate the most important one to look at? So I did what you do: googled “per PI sucess rate at NSF” (without the quotes).
I came up with NSF’s annual report on its merit review process from FY 2013 (probably the most recent FY available, I’m guessing). It includes data on per-PI success rates going back to 2001, along with some relevant contextual data. Thanks Google!*
The Cliff Notes** version:
- The current per-PI success rate at NSF is 35% per 3 years. That is the percentage of PIs applying who get funded, calculated in 3-year moving windows (i.e. number of PIs who were awarded at least one grant anytime in that window, divided by the number who submitted at least one proposal anytime in that window). That percentage declined slightly from 41% in 2001-2003 to 36% in 2006-2008. It rose back to 40% in 2007-2009 and 2008-2010 thanks to stimulus funding. It then dropped back slightly to 35%, where it’s been since 2010-2012 (most recent window is 2011-2013). UPDATE: These data come from Fig. 14 in the linked report.
Now, to interpret that broad-brush answer, you need contextual information. The linked report has a bunch, but not everything you probably want to know***:
- More PIs are submitting proposals. The number of PIs who submitted at least one proposal in 2011-2013 was 41% higher than in 2001-2003. UPDATE: This is from Fig. 14 in the linked report.
- More proposals are being submitted: up 70% from 2001 to 2013 (UPDATE: Table 7 of the linked report). So the number of proposals is rising substantially faster than the number of PIs.
- Per-proposal success rate is down from 2001, when it was 27%. But that number hasn’t really budged since 2005 (stimulus bump aside) and currently sits at 19% (UPDATE: Table 7 in the linked report).
- UPDATE: Because this came up in the comments: data on mean and median size of research grants, in both nominal and real terms, are in Figs. 7 and 8. Both mean and median award sizes are increasing over time in nominal terms, with some ups and downs due to to things like stimulus funding (e.g., the median annual award size for research grants increased from ~$85,000 in 2002 to ~$130,000 in 2013). In real terms, mean and median award sizes are either roughly steady or increasing only slowly over time, with ups and downs (e.g., median annual award size in 2005 dollars was ~$90,000 in 2002 and ~$110,000 in 2013).
- All of these numbers are for full proposals. An appendix presents data on the fraction of pre-proposals for which full proposals were invited, for those NSF units that have binding pre-proposals (it was ~25% in both 2012 and 2013). So per-proposal success rates are higher than they would be if you included pre-proposals. And it’s possible that per-PI funding rates would drop if you included pre-proposals, since it’s possible that some unsuccessful PIs have never been invited to submit full proposals.
- Multi-PI proposals, where the PIs are from different institutions, are counted multiple times (see here). That certainly distorts the picture of per-proposal funding rates, since in reality a multi-PI proposal is a single proposal. But it doesn’t distort per-PI funding rates. If you’re a PI on a multi-PI proposal, and the proposal gets funded, you and the other PIs all get funded (ok, you probably don’t all get as much funding as if you’d all written successful single-PI proposals, but that’s a different question). As an aside, single-PI awards continue to outnumber multiple-PI awards, but the gap is slowly closing (UPDATE: Fig. 9 in the linked report). Single-PI proposals also have slightly higher success rates than multi-PI proposals, and those rates haven’t budged much over time (UPDATE: Fig. 11 in the linked report), so presumably the increasing proportion of multi-PI awards reflects an increasing proportion of multi-PI proposals.
- These numbers are NSF-wide, they’re not specific to the NSF divisions (DEB and IOS) to which ecologists mostly apply. Which interacts with the previous two bullets, because the divisions to which ecologists mostly apply are the ones that brought in pre-proposals for their core programs in 2012, and because I think (?) DEB and IOS tend to receive a higher proportion of multi-PI proposals than some other NSF divisions.
These numbers include all categories of proposals, which I believe means they include things like DDIGs, conference support, and REU supplements (again, see here). Many of those categories have higher success rates than core research programs. Now, most of those categories involve small numbers of proposals and PIs, so won’t affect the per-PI funding rate too much. But DDIGs are more numerous.CORRECTION: The per-PI success rate data in Fig. 14 in the linked report are per-PI success rates for research grants. “Research grants” is a critical term here. This includes “typical” panel and mail-reviewed grants as well as EAGER and RAPID awards. (Aside: footnote 23 of the report notes that EAGER and RAPID awards have high success rates, but are only 1.4% of all proposals) The per-PI success rate quoted above does not include DDIGs, REU supplements, conference support, fellowships, equipment grants, Small Grants for Exploratory Research, most things funded by the education directorate, or big-ticket items like NEON construction. Similarly, all of the contextual data given above for number of PIs applying, per-proposal success rates, mean/median award size, are for research grants only. (Aside: some other figures and tables in the report do include stuff besides research grants, under the broader category of “competitive awards/actions”) Thank you to a correspondent from NSF for correcting me on this, and apologies for the error. In retrospect, I was reading too quickly–I missed the bit on the top of p. 19 in the linked report where NSF explains all this, clear as day.
- There are no data provided on how often PIs apply, what their other funding sources are, or what type of institution they’re based at. When per-PI success rates are calculated, somebody who submits one proposal in 3 years and is unsuccessful counts the same as somebody who submits several proposals in 3 years and is unsuccessful on all of them. So you can’t tell from the data if, say, some of the growth in PIs and proposals is coming from people who don’t ordinarily seek NSF funding, or haven’t in the past, but who for whatever reason have decided to take an occasional crack at it.
Some take-home thoughts:
- I think the 35% number quoted above likely is an upper bound on the current per-PI success rate over 3 years for faculty PIs seeking grants from the core programs of DEB and IOS. But I’m not dead certain.
- I’m very surprised that the per-PI success rate hasn’t dropped much since 2001-2003, even though the number of PIs has increased 41%. Anecdotally, it had been my impression from reading social media that on a per-PI basis NSF funding had suddenly gotten much more difficult to obtain just in the past few years (not that it was easy to obtain before in an absolute sense). But if so, that doesn’t show up in these data, and I’m sure NSF’s numbers are correct. Now, you can probably tell a story about why a recent crash in the per-PI success rate wouldn’t show up in these data–but it’s not as easy as you might think. I’ve been trying and failing to come up with one (am I just being dense?)
- More proposals per PI seems like a problem to me, since it suggests a sort of tragedy of the commons or Red Queen phenomenon–all these people writing more proposals just to keep up with all these people writing more proposals. All that time spent writing and reviewing proposals presumably could be spent doing something else, like science. And there’s of course all the stress and pressure
boiled frogsPIs feel, which I suspect correlates more with per-proposal success rates (though in fairness, it’s not NSF’s job to make PIs feel happy…) All of which seems like a pretty good argument for limiting the number of proposals/PI/year, as noted by Chris in another comment. Indeed, DEB and IOS now cap pre-proposals/PI/year. Since the same rules apply to everyone, a cap on proposals/PI/year won’t affect the per-PI funding rate, and so shouldn’t reduce any PI’s chances of establishing or maintaining a research program.
You tell me, US colleagues–what do you think of these per-PI numbers? Are you surprised at how little they’ve changed over time? Do you find them encouraging or depressing? Useful or too hard to interpret? And what implications, if any, do you think the per-PI data have for issues like whether NSF should reduce average grant size or limit the number of active grants PIs can hold at once?****
p.s. The report also breaks down the data by self-reported gender, ethnicity, and disability status of PI. I encourage you to click through and read the report for details, but from my admittedly-quick skim the overall picture on this front is mostly (not entirely) a mix of good news and news that’s trending in the right direction. For instance, female PIs are if anything funded at very slightly higher rates than male PIs, and are submitting an increasing fraction of proposals. Having said that, to really interpret these numbers effectively so as to identify the root causes of any disparities, I think you’d want more contextual information than NSF provides (or could reasonably be expected to provide in this sort of report). Contextual information is really important.
*Actually, first of all thanks NSF!
**My god, that reference dates me, doesn’t it?
***Understandably. The purpose of the report is to summarize NSF’s merit review process for the National Science Board, not to allow individual PIs to estimate their own odds of obtaining NSF funding with high precision.
****Re: limiting the number of active grants PIs can hold at once in order to free up money for other PIs, the linked report has relevant data on that. In particular, the large majority of PIs with at least one grant just have the one, and very few have more than two. Now, some of the PIs with single grants are holders of things like DDIGs. But still, it’s possible that, if you ran the numbers, you might find that capping the number of active grants a PI could hold wouldn’t free up much money.