Not an April Fool’s joke: PI success rates at NSF are not dropping (much) (CORRECTED and UPDATED)

If you’ve saw this post in the first few hours after it went up, there’ve since been some major updates and corrections!


The title of this post is not a joke (I’ll cop to deliberate provocation…), but it does require some explanation.

My inspiration is this old comment from Chris Klausmeier:

[W]hat I’d really like to know is the success rate per PI, not per grant. That is, are there more people fighting for a constant amount of dollars leading to an increasing “unfunded rate”, or are there roughly the same number of people dividing up the same amount of dollars but with increasing number of proposals per PI?

That’s a great question. After all, if you’re worried about things like the ability of PIs to establish and maintain their labs, isn’t the per-PI success rate the most important one to look at? So I did what you do: googled “per PI sucess rate at NSF” (without the quotes).

I came up with NSF’s annual report on its merit review process from FY 2013 (probably the most recent FY available, I’m guessing). It includes data on per-PI success rates going back to 2001, along with some relevant contextual data. Thanks Google!*

The Cliff Notes** version:

  • The current per-PI success rate at NSF is 35% per 3 years. That is the percentage of PIs applying who get funded, calculated in 3-year moving windows (i.e. number of PIs who were awarded at least one grant anytime in that window, divided by the number who submitted at least one proposal anytime in that window). That percentage declined slightly from 41% in 2001-2003 to 36% in 2006-2008. It rose back to 40% in 2007-2009 and 2008-2010 thanks to stimulus funding. It then dropped back slightly to 35%, where it’s been since 2010-2012 (most recent window is 2011-2013). UPDATE: These data come from Fig. 14 in the linked report.

Now, to interpret that broad-brush answer, you need contextual information. The linked report has a bunch, but not everything you probably want to know***:

  • More PIs are submitting proposals. The number of PIs who submitted at least one proposal in 2011-2013 was 41% higher than in 2001-2003. UPDATE: This is from Fig. 14 in the linked report.
  • More proposals are being submitted: up 70% from 2001 to 2013 (UPDATE: Table 7 of the linked report). So the number of proposals is rising substantially faster than the number of PIs.
  • Per-proposal success rate is down from 2001, when it was 27%. But that number hasn’t really budged since 2005 (stimulus bump aside) and currently sits at 19% (UPDATE: Table 7 in the linked report).
  • UPDATE: Because this came up in the comments: data on mean and median size of research grants, in both nominal and real terms, are in Figs. 7 and 8. Both mean and median award sizes are increasing over time in nominal terms, with some ups and downs due to to things like stimulus funding (e.g., the median annual award size for research grants increased from ~$85,000 in 2002 to ~$130,000 in 2013). In real terms, mean and median award sizes are either roughly steady or increasing only slowly over time, with ups and downs (e.g., median annual award size in 2005 dollars was ~$90,000 in 2002 and ~$110,000 in 2013).
  • All of these numbers are for full proposals. An appendix presents data on the fraction of pre-proposals for which full proposals were invited, for those NSF units that have binding pre-proposals (it was ~25% in both 2012 and 2013). So per-proposal success rates are higher than they would be if you included pre-proposals. And it’s possible that per-PI funding rates would drop if you included pre-proposals, since it’s possible that some unsuccessful PIs have never been invited to submit full proposals.
  • Multi-PI proposals, where the PIs are from different institutions, are counted multiple times (see here). That certainly distorts the picture of per-proposal funding rates, since in reality a multi-PI proposal is a single proposal. But it doesn’t distort per-PI funding rates. If you’re a PI on a multi-PI proposal, and the proposal gets funded, you and the other PIs all get funded (ok, you probably don’t all get as much funding as if you’d all written successful single-PI proposals, but that’s a different question). As an aside, single-PI awards continue to outnumber multiple-PI awards, but the gap is slowly closing (UPDATE: Fig. 9 in the linked report). Single-PI proposals also have slightly higher success rates than multi-PI proposals, and those rates haven’t budged much over time (UPDATE: Fig. 11 in the linked report), so presumably the increasing proportion of multi-PI awards reflects an increasing proportion of multi-PI proposals.
  • These numbers are NSF-wide, they’re not specific to the NSF divisions (DEB and IOS) to which ecologists mostly apply. Which interacts with the previous two bullets, because the divisions to which ecologists mostly apply are the ones that brought in pre-proposals for their core programs in 2012, and because I think (?) DEB and IOS tend to receive a higher proportion of multi-PI proposals than some other NSF divisions.
  • These numbers include all categories of proposals, which I believe means they include things like DDIGs, conference support, and REU supplements (again, see here). Many of those categories have higher success rates than core research programs. Now, most of those categories involve small numbers of proposals and PIs, so won’t affect the per-PI funding rate too much. But DDIGs are more numerous. CORRECTION: The per-PI success rate data in Fig. 14 in the linked report are per-PI success rates for research grants. “Research grants” is a critical term here. This includes “typical” panel and mail-reviewed grants as well as EAGER and RAPID awards. (Aside: footnote 23 of the report notes that EAGER and RAPID awards have high success rates, but are only 1.4% of all proposals) The per-PI success rate quoted above does not include DDIGs, REU supplements, conference support, fellowships, equipment grants, Small Grants for Exploratory Research, most things funded by the education directorate, or big-ticket items like NEON construction. Similarly, all of the contextual data given above for number of PIs applying, per-proposal success rates, mean/median award size, are for research grants only. (Aside: some other figures and tables in the report do include stuff besides research grants, under the broader category of “competitive awards/actions”) Thank you to a correspondent from NSF for correcting me on this, and apologies for the error. In retrospect, I was reading too quickly–I missed the bit on the top of p. 19 in the linked report where NSF explains all this, clear as day.
  • There are no data provided on how often PIs apply, what their other funding sources are, or what type of institution they’re based at. When per-PI success rates are calculated, somebody who submits one proposal in 3 years and is unsuccessful counts the same as somebody who submits several proposals in 3 years and is unsuccessful on all of them. So you can’t tell from the data if, say, some of the growth in PIs and proposals is coming from people who don’t ordinarily seek NSF funding, or haven’t in the past, but who for whatever reason have decided to take an occasional crack at it.

Some take-home thoughts:

  • I think the 35% number quoted above likely is an upper bound on the current per-PI success rate over 3 years for faculty PIs seeking grants from the core programs of DEB and IOS. But I’m not dead certain.
  • I’m very surprised that the per-PI success rate hasn’t dropped much since 2001-2003, even though the number of PIs has increased 41%. Anecdotally, it had been my impression from reading social media that on a per-PI basis NSF funding had suddenly gotten much more difficult to obtain just in the past few years (not that it was easy to obtain before in an absolute sense). But if so, that doesn’t show up in these data, and I’m sure NSF’s numbers are correct. Now, you can probably tell a story about why a recent crash in the per-PI success rate wouldn’t show up in these data–but it’s not as easy as you might think. I’ve been trying and failing to come up with one (am I just being dense?)
  • More proposals per PI seems like a problem to me, since it suggests a sort of tragedy of the commons or Red Queen phenomenon–all these people writing more proposals just to keep up with all these people writing more proposals. All that time spent writing and reviewing proposals presumably could be spent doing something else, like science. And there’s of course all the stress and pressure boiled frogs PIs feel, which I suspect correlates more with per-proposal success rates (though in fairness, it’s not NSF’s job to make PIs feel happy…) All of which seems like a pretty good argument for limiting the number of proposals/PI/year, as noted by Chris in another comment. Indeed, DEB and IOS now cap pre-proposals/PI/year. Since the same rules apply to everyone, a cap on proposals/PI/year won’t affect the per-PI funding rate, and so shouldn’t reduce any PI’s chances of establishing or maintaining a research program.

You tell me, US colleagues–what do you think of these per-PI numbers? Are you surprised at how little they’ve changed over time? Do you find them encouraging or depressing? Useful or too hard to interpret? And what implications, if any, do you think the per-PI data have for issues like whether NSF should reduce average grant size or limit the number of active grants PIs can hold at once?****

p.s. The report also breaks down the data by self-reported gender, ethnicity, and disability status of PI. I encourage you to click through and read the report for details, but from my admittedly-quick skim the overall picture on this front is mostly (not entirely) a mix of good news and news that’s trending in the right direction. For instance, female PIs are if anything funded at very slightly higher rates than male PIs, and are submitting an increasing fraction of proposals. Having said that, to really interpret these numbers effectively so as to identify the root causes of any disparities, I think you’d want more contextual information than NSF provides (or could reasonably be expected to provide in this sort of report). Contextual information is really important.

*Actually, first of all thanks NSF!

**My god, that reference dates me, doesn’t it?

***Understandably. The purpose of the report is to summarize NSF’s merit review process for the National Science Board, not to allow individual PIs to estimate their own odds of obtaining NSF funding with high precision.

****Re: limiting the number of active grants PIs can hold at once in order to free up money for other PIs, the linked report has relevant data on that. In particular, the large majority of PIs with at least one grant just have the one, and very few have more than two. Now, some of the PIs with single grants are holders of things like DDIGs. But still, it’s possible that, if you ran the numbers, you might find that capping the number of active grants a PI could hold wouldn’t free up much money.

14 thoughts on “Not an April Fool’s joke: PI success rates at NSF are not dropping (much) (CORRECTED and UPDATED)

  1. “All of these numbers are for full proposals. An appendix presents data on the fraction of pre-proposals for which full proposals were invited, for those NSF units that have binding pre-proposals (it was ~25% in both 2012 and 2013). So per-proposal success rates are higher than they would be if you included pre-proposals.”

    This is a huge deal. If the per-PI success rate for full proposals is the same, but only 25% of preproposals get invited to submit full proposals, that’s a big difference. The exact effect on per-PI funding rates is hard to estimate, given that the reported rate includes DDIGs, REU supplements, etc., and that we don’t know how many people submit multiple preproposals and then have one invited. But my impression is that most people are not submitting multiple preproposals in a year. So, if only approximately 25% of PIs get to submit full proposals, that’s a big deal.

    (UPDATE from Jeremy: Meg’s comment here predates my correction to the post: the reported per-PI success rate does not include DDIGs, REU supplements, etc.)

    • Unfortunately, I think this is something you can’t get at with NSF-wide numbers. There’ve long been a few NSF programs with binding pre-proposals. But in 2012 there was a big jump in the number of binding pre-proposals because DEB and IOS implemented them. But I don’t think DEB and IOS core programs have enough PIs and proposals, relative to NSF as a whole, for any change in their policies to really show up in the NSF-wide per-PI success rate numbers.

      • @Meg:

        I hope so too. It’s my understanding from correspondence that it’s on their radar, but that it’s not easy to wrestle NSF’s systems into reporting the data that PIs care about. Understandably–those systems aren’t designed for that purpose, they’re designed for accounting and other purposes.

    • “The current success rate is in the one digit (<10%)…this is bad."

      You're referring to the per-preproposal success rate at DEB. I take it that means you don't buy the argument that per-PI success rates are what should be of most concern?

      Actually, I agree per-preproposal rates should be of concern too, but for different reasons than per-PI rates. Part of the reason I wrote this post is that, anecdotally, it's my impression that people often express concern about per-preproposal success rates when, based on their reasons for concern, they ought to be looking at per-PI success rates.

      "pre-proposal number manipulation"

      NSF isn't trying to manipulate the numbers; are you sure "manipulate" is the word you want? NSF isn't trying to fool PIs into thinking that the situation is anything other than it is. My understanding is that they report the numbers in the way they do because that's the way their reporting and information management systems work. After all, if NSF was trying to "manipulate" the numbers, why would NSF publish a blog and public reports that explain the "manipulation"?

  2. “These numbers include all categories of proposals, which I believe means they include things like DDIGs, conference support, and REU supplements (again, see here). Many of those categories have higher success rates than core research programs. Now, most of those categories involve small numbers of proposals and PIs, so won’t affect the per-PI funding rate too much. But DDIGs are more numerous.”

    (UPDATE from Jeremy: Josh’s comment here predates my correction to the post. The numbers quoted in the post don’t include DDIGs, conference support, etc.)

    In addition to the issue Meg raised with pre-proposals, above, this is inflating the success rate (likely significantly, especially the DDIGs). After reading this analysis, I am more convinced that those applying to DEB and IOS under the current structure are faring poorly, and that would be born out in the per-PI analysis of individual and group proposals.

    Also, Jeremy, sorry about the Twitter kerfluffle the other day. Wasn’t intended to come off so rude. And good post here.


  3. Like several others, I think the pre-proposal “failure” rates not being accounted for is huge. My default reading – not proven but not totally fact free – is that there is a long term downward trend that is reasonably steep but the end of TARP and the sudden removal of pre-proposal non-successes interact to hide this trend.

    In any case – some of the basic facts you repeat and which can be found in other NSF sources that: a) the # of PIs applying is way up, b) the total funding is flat, and c) the grant size in core programs is flat don’t add up to flat per PI funding (at least in DEB).

    • I hear you Brian. Though I’m not sure if your default reading is *quite* consistent with the data or not, at least at the NSF-wide level (as I said above, I don’t think you can hone in on what’s happening at DEB and IOS with the NSF-wide data). If per-PI success rate is going down in core programs because more PIs are applying but award size and total funding are flat, then to keep per-PI success rates flat across NSF *as a whole* (i.e. core programs plus other programs), what needs to be happening to the number (and identity) of people applying to other programs where per-proposal success rates are higher? And what needs to be happening to per-proposal success rates and average award sizes at those other programs over time? The answer isn’t immediately obvious to me. It’s not that I don’t believe per-PI success rates in core programs are declining. But if they are, I’m puzzled/curious about what’s changing in other programs over time so as to keep NSF-wide per-PI success rates pretty steady since about 2005, stimulus bump aside. My comment here predates the correction of the post and is based on a misunderstanding.

      The linked report includes data on the total NSF spend on grants (in both nominal and real terms), and average award size that would go at least some way towards addressing your default reading. UPDATE: I’ve now added those data to the post.

      This is starting to puzzle me enough that I might try to set up some sort of toy simulation of an NSF with two programs–a “core” program and another one (“DDIG”, say). Vary the allocation of money to each one, the number and identity of people applying to each one, the distribution of applications per-PI to each one (including multi-PI applications), etc. Play around with it and see what it takes to get time series of data that look roughly like those in the NSF report, without assuming or predicting anything too weird about stuff on which the report provides no data. My comment here predates the correction of the post and is based on a misunderstanding.

      Or maybe it’d be easier to do what Meg suggests and just hope that the DEBrief blog posts some data on per-PI success rates at DEB!

    • Ok, I did a baby simulation, which I think helps sharpen our intuitions a bit.

      It’s a 2-year simulation:

      In year 1, 1000 PIs each submit 2, single-PI proposals. 10% of proposals are funded, so 200 successful proposals out of 2000 (if you like, you can think of this 10% per-proposal success rate as the net result of a pre-proposal and full proposal stage). Use the binomial formula to figure out how many PIs have 0 successes out of 2, vs. 1 or 2 successes (so all PIs and all proposals have the same probability of success).

      In year 2, all the PIs unfunded in year 1 try again, as do X% of the PIs who were funded in year 1. We also add some new PIs to the system. Everybody submits 2 proposals again. There are still only 200 successful proposals, because mean award size and the funding agency’s budget are constant. So per-proposal success rate changes accordingly. All PIs and proposals once again have equal odds of success. Under those assumptions, figure out how many PIs get 0, 1, or 2 awards. And figure out how many of the successful PIs (those getting at least 1 award) already had at least 1 award from year 1.

      Total up how many PIs applied at least once, and how many were successful at least once. Divide the latter by the former to get per-PI success rate over this 2-year window. Fiddle with the % of PIs who come back for seconds after a year 1 success, and the number of new PIs entering the system in year 2, to see how per-PI success rates change as a function of the year-to-year increase in number of PIs applying.

      Results: if the number of PIs applying increases 20% from one year to the next, as opposed to remaining constant, that only drops the per-PI success rate from about 35% to 30%. The exact numbers vary a little bit depending on what fraction of successful PIs in year 1 return in year 2; I checked values from 50-90%.

      Obviously, there’s nothing special about the numbers I picked, you could pick different numbers and get quantitatively different results. But qualitatively, what I take home from this little exercise is that year to year growth in number of PIs applying doesn’t necessarily have a major effect on per-PI success rates over some multi-year window. Basically, the cumulative probability of having at least 1 success anywhere in the window isn’t that sensitive to the year-to-year growth rate in the number of people applying.

      I freely admit that’s a toy simulation. It could be done better–I just realized that this is effectively a stage-structured matrix model. So at some point maybe I’ll go beyond my little Excel simulation (yes, I used Excel) and do the proper stage-structured model, run it until you get a stable stage structure, and see what its properties are and how sensitive they are to changes in the various transition rates.

  4. Off the main topic, but in rereading the report to correct the post, I noticed that, although applications from women are if anything slightly more likely to be successful than applications from men, applications from those who withhold their gender are less likely to succeed. Not sure what to make of that, though I suspect it’s down to other attributes of applications that covary with the choice of the applicant not to provide gender information. For instance (and this is a purely made-up hypothetical), if it’s mostly junior women who choose to withhold gender information, applications with no gender listed might get funded at lower rates because applications from new PIs tend to succeed at lower rates. So that when looking at the success rates of applications from men vs. women, you’re comparing the average success rate of junior+senior men with the success rate of senior women.

    Again, the above is a pure hypothetical, there could be *many* other explanations for why applications with no gender listed are successful at lower rates.

  5. Pingback: DEB Numbers: Per-Person Success Rate in DEB | DEBrief

  6. Pingback: We asked, NSF answered: per-PI success rates at the NSF DEB | Dynamic Ecology

Leave a Comment

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s