A few weeks ago I posted on per-PI success rates at NSF, focusing on success rates per 3 year moving window. That is, what proportion of PIs applying anytime in a 3 year period got at least one grant in that period? I argued that this–not the more widely-discussed per-proposal success rate–is the most relevant measure of “success” if one is concerned about the ability of PIs to establish and sustain their labs. The data were pleasantly surprising, at least relative to the dire numbers one might’ve expected: the current per-PI success rate for core research grants per 3-year period is 35%.
In response, several readers quite reasonably questioned the relevance of those numbers to ecologists and evolutionary biologists. NSF-wide data aren’t specific to DEB and IOS, the NSF divisions to which ecologists and evolutionary biologists mostly apply. In particular, given that DEB and IOS brought in a pre-proposal system a few years ago, one might expect that per-PI success rates at DEB and IOS might be lower than for NSF as a whole.
Well, ask and ye shall receive, apparently! Heck, we didn’t even have to ask! A staff member at the NSF DEB saw my post and went to the trouble to crunch the relevant numbers for the DEB specifically. Here’s the very detailed DEBrief Blog post on the results.
The headline result: the 3-year per-PI success rates on core research grants at DEB (so not DDIGs, conference support, etc.) over 2012-2014 was 15.4%, or 17.5% if you exclude co-PIs. That’s the proportion of PIs who applied at least once anytime in 2012-2014 and were successful at least once. (UPDATE: This number includes authors of pre-proposals, unless I badly misread the linked post.) As I guessed, and as the linked post notes, that’s lower than the NSF-wide success rate. The 3-year per-PI success rate at DEB is down from about 28% back in 2006-2008.
The linked post also provides a lot of other data from DEB, which aid interpretation of the headline number. The most important contextual information is that stimulus funding and the onset of the pre-proposal system seem to have attracted lots of new PIs to DEB. That’s the main reason why per-PI success rate is dropping at DEB. There’s also other contextual information, like number of PIs holding more than one DEB grant at a time*. The data are broken down by gender. And the linked post also has a lot of discussion of caveats and limitations: the effects of different definitions of “PI” on preliminary vs. full proposals, the effect of moving some special programs into DEB from other NSF units, why the analysis “only” goes back to 2006,** why it’s so hard to include pre-proposals in the calculations,*** and more. And it’s all written in a clear, up-front, open style.
I’m curious to hear what commenters think of these data. In particular, that the advent of pre-proposals seems to have attracted more PIs to DEB was a surprise to me–was it a surprise to you? I think this highlights how difficult it can be for any funding agency to aim for a particular success rate. Any policy change you make that’s intended to increase success rates–and many that aren’t–might well have the unintended consequence of attracting more applicants and applications, thereby pushing success rates down. In particular, I wonder if the popular option of cutting award size to increase success rate would be at least partially self-defeating by attracting more applicants and applications.
In conclusion, a big thanks to the folks at NSF DEB for doing this. It’s a lot of work to pull an analysis like this together, using a database that’s not designed to give you these numbers. Nobody at the NSF DEB had to do this (they’ve got lots of other things to do). They just saw that some folks in the research community were interested in these numbers, and so went ahead and chased the numbers down. And published them immediately, in a blog post, for all to see. All of which is awesome.
*Very few. Even if you look over the entire 2006-2014 period, the large majority of PIs who got at least one DEB award got just the one award. So no, success rates at DEB are not low because Dr. Famous is hogging all the money.
**”[2006 was] chosen mainly because that was the span of complete years beyond which the server was getting angry with us for requesting too many records, #overlyhonestmethods”. Yes, that’s a quote. 🙂
***Briefly, because of the way NSF’s record-keeping database is designed.
Dynamic Ecology 
Here are the main take home messages in the NSF funding climate we are living in now (last three years, in particular):
4) Of 11,789 unique PI/CoPI applicants, only 2,970 (25% of all applicants) received any funding over the 9-year period examined. Of those 2,970 to receive funding, only 772 received multiple awards (26% of awardees, 6% of all applicants) that could potentially maintain continuous “funding” over this period. Any person applying to DEB’s competitive research programs is unlikely to be funded, and much less likely to maintain continuous support for a lab from this single funding source.
5) Coming back to our original motivation for this post, per-person success rates for funding in DEB were consistently ~10 percentage points lower than the NSF-wide submission and funding data in years leading up to the preliminary proposal system. The exclusion of preliminary proposals from NSF-wide statistics has only deepened the apparent magnitude of this disparity in recent years and has even altered the trajectory of PI participant counts for the agency as a whole.
I too am surprised to learn that per-PI success rates were running lower at DEB than NSF wide even before the advent of pre-proposals caused a jump in the number of applicants to DEB. I have no idea why that would be. I’m unsure how to interpret that or what if anything to do about it; open to ideas.
“The exclusion of preliminary proposals from NSF-wide statistics has only deepened the apparent magnitude of this disparity in recent years and has even altered the trajectory of PI participant counts for the agency as a whole.”
Can you elaborate? Because unless I misread something in my previous post, I think that’s going a bit too far. NSF divisions with pre-proposals are not a large fraction of all NSF divisions. And unless I’m misreading the NSF-wide data, I don’t see any major changes in NSF-wide data resulting from the onset of pre-proposals at DEB and IOS. Further, as I noted in the post, NSF does their best to publish the data they can with the tools they have. I get the sense that you’d like to see them revamp their database software?
I agree with you that 3-year per-PI success rates of 17% or so are probably lower than desirable, even for a project-based system like NSF’s*. And you’ve made clear that you consider even the pre-stimulus, pre-preproposal 3-year per-PI success rate of 28% to be far too low. So keep going–what should be done? Because as I noted in the post, I don’t see any obvious way to do a non-self-defeating ramp-up of success rates. How do you ramp up success rates in a project-based system like NSF’s, when anything you do to increase success rates might just attract more applicants and applications? Honest question.
*Although just to play devil’s advocate, as long as NSF DEB is receiving a sufficient number of proposals that are judged worthy of funding, shouldn’t they be happy? Don’t get me wrong–I agree with you that it’s a serious problem that it’s so difficult for PIs to maintain an ongoing, long-term research program by going back to NSF for funding for individual projects. But is it *NSF’s* problem? To put the question another way, does the current situation, in which an individual PI can only count on getting an occasional NSF grant, seriously degrade the collective research capacity of US ecologists? Either in general, or with respect to certain types of research (e.g., long-term work)? Honest questions to which I don’t know the answer.
Two things. First, for the last 3 years under the preproposal system the per PI success rate is around 10%, not 17%, and as NSF states, that means that most people cannot be funded by NSF and even people who get funded by NSF are very unlikely to be funded continuously.
Second, I am strongly in favor of highlighting these issues but not because I think that NSF can do much about them. The jump in numbers applying for money suggests that the underlying problem is much, much bigger and that is… there is not enough money to go around (big surprise, right?, no not at all). The problem is really even bigger than that because many people cannot apply for “other money” without fundamentally changing their research program. If you work on basic questions in ecology, systematics, ecosystem ecology, evolution, there may be some other pots of money that you could shoehorn your research to fit, but you are not likely to be competitive (any more than you are competitive at 10% funding rates at NSF). So where do you go? Well, many people change research topics, or leave the field, or go grantless (I have colleagues, tenured, of course, who have more or less been forced into this route, and I am facing my first gap year, too).
Brian has previously suggested a number of great ideas for rearranging the funding structure at NSF, but I think ultimately, that doesn’t solve the problem. I’m beginning to think that basic science needs to go the way that conservation went years ago – figuring out ways to influence state and federal legislators, i.e. lobbying, to get more funding for basic science. I know this goes on currently, but it does not occur at the level that it needs to. Basic research is an excellent investment for state and federal government, but we have done a really lousy job selling that to both the public and politicians. If you are asking me what we need to do it is two things (and don’t laugh): have more basic scientists pursue careers in state and federal government, and have more basic scientists, collectively!, work on grassroots efforts to get our message out to voters and legislators. Until more money goes to NSF on a regular basis, we’ll just be going in circles about how finely to slice up the same sized loaf of bread when there are more and more mouths to feed, so to speak.
Sorry, my first comment should have stated success rate per year. And I do note the 17.5% success for the 2012-14 window.
Just to elaborate on a point touched on in the post: One thing these data did is caused me to reject my mental model that there is some roughly fixed (or only very slowly changing) number of ecologists in the US, where “ecologist” means “people who apply to NSF DEB for funding”. It turns out that “ecologist” in that sense isn’t even a roughly fixed pool of people–lots of people are happy to apply to NSF DEB for money if there’s more money available (thanks to the stimulus), or if it becomes easier to apply (thanks to preproposals).
So if you want to raise success rates, maybe what you need to do is make applying for funding more onerous, so as to discourage some people from applying in the first place!
My take-away: diversify, diversify. It no longer works to look for funding in one place. One must apply to many pots of money. (It would be interesting to track down those pots that have *higher* success rates than the NSF average. They’ve got to exist to balance out the DEB’s…)
“diversify, diversify”
Indeed, that’s presumably what the many new applicants to DEB were doing in response to the stimulus and the advent of pre-proposals–diversifying their own funding sources to include DEB!
I’d be curious to see some survey data. How many funding sources does the average ecologist apply to, and get money from, over say the last 5 years? And how is it affected by–and how does it affect!–variables like what sort of institution employs you and what sort of research you do?
I’m also curious–how many people *don’t* actually sustain some continuous level of funding, whether by going to NSF or by cobbling together various sources? That is, how common is it to have a “boom and bust” lab–you ramp up when you get an NSF grant, do the project, and then ramp down until you get another one? Obviously, for many sorts of research this wouldn’t be possible, or would be far from ideal. But I imagine that some people do it (?), and speculatively I could imagine it might become more common in future. Much as an unpredictable environment can select for the ability to go dormant, but grow rapidly and reproduce in response to the rare good times. For instance, I could definitely keep my own microcosm work ticking over on a shoestring, but ramp up quickly to do some big one-off project if I ever got the money.
I guess in the end, to me, it doesn’t matter whether the number is 5% and headed south per proposal or 10% and headed south per PI each year or 15% and headed south per PI per 3 years. Those numbers are all way too low. You might say, since I’m a scientist on the dole, I’m biased. But I think they are too low in several objective senses:
1) Too low relative to any accurate evaluation rates. Once you drop below 25 or 30% you are basically introducing a lot of randomness into the decision making process. I’ve talked to a lot of people who feel like with 25-35% you are being very selective but fair. Lower numbers mean arbitrary luck of which reviewer gets it, etc.
2) Too low relative to the historical norms. Historical norms get locked in any number of ways. Deans in the US will regularly tell you during salary negotiations that they are assuming your real take home pay is 10/9 or even 11/9 of what they’re paying you because “of course you’re going to have summer salary, right?”. Tenure committees at R1 universities still have an expectation that you need an NSF (or NIH) grant to get tenure. Productivity in the US is assumed to be at a level based on having a technician but technicians are only funded by NSF grants which are no longer viable as a basis for a human being’s income.
As you say Jeremy it is not obvious there is an easy fix as the number of applicants varies according to conditions. But it is hard to believe that reducing grant sizes wouldn’t have some upward effect on accept rates.
And per Margaret’s point, there are no magic alternatives. Other federal grant programs (NIH, USDA primarily) have the same trends as NSF. State wildlife and forestry management money has disappeared. Federal land management money (e.g USFS, USFW) is trending down. Foundations may be a bit more of a player on the scene than the past in ecology but they’re still rare enough not to be having a huge impact. In short, NSF is typical, not a special case.
I really think we’re reaching a tipping point where the whole US research funding system is breaking down. The ultimate causes are probably not NSF (which has at least flat funded research for the past decade). It is probably the terrible job market (=over production of PhDs) and the pull out of states from funding research universities that have in combination led to ever increasing pressure on ever more people to apply for grants. In the old days a faculty member at an RII or an undergrad institution could reasonably expect to get a few thousand for field expenses or travel to a conference from their university and not to be under serious pressure to apply for grants. With the budget pressures the first is gone and with the view of indirect dollars as a plug to the budget, the 2nd is gone. And the people now being hired into those jobs have come out of and expect themselves to continue highly active research problems. Which all add up to rapidly growing numbers of want-to-be PIs.
But whatever the reason, its not a lot of fun to live through it!
Pingback: Links 5/11/15 | Mike the Mad Biologist
Pingback: Feeling the Tin Man’s Pain – The Loon Project