Frogs jump? researcher consensus on solutions for NSF declining accept rates

Dynamic Ecology’s readers have spoken in a clear voice! There is a clear consensus around what changes people favor to address the hopelessly declining grant award rates at NSF. In a post on Monday I described what I see as the long-term exogenous trends in our society (US specifically but as commenters noted probably largely applicable globally) that affect NSF. And that are putting in NSF in a tight squeeze leading to a current acceptance rate of 7.3% and every expectation it will go still much lower. Basically funding flat, many pressures on researchers to apply for more grants (both more applications from old hands and pressure on many to begin applying) lead to a trade-off, the only variables of which NSF controls is # of applications and grant size in $.

I had a reader poll on what choices readers would like to see NSF adopt. To be clear this poll is entirely unscientific in sample design – its whoever reads the blog and answers. It is presumably mostly academic ecologists, and our readership skews early career and male (although how much more it does so than academic ecology in general is unknown), but beyond that I couldn’t say what biases there are. There were 450 votes – I don’t know how many voters since each voter could vote up to 3 times and polldaddy doesn’t give me the details unless I pay them $200 (I know there are other choices – I’ll probably use them next time but polldaddy is so convenient from inside WordPress). But to a first approximation there were presumably about 160-175 voters (some voters likely voted for only 1 or 2 choices). The results as of 11:30AM EST Wednesday (in my experience the vast majority of people who will read the post have read it by now) are:

Results of survey on solutions declining accept rates at NSF

Results of survey on solutions for declining accept rates at NSF. Note since users could pick three choices the 450 votes probably maps to slightly more than 150 voters, perhaps 160-175 total voters, each picking 1-3 choices.

Basically there are three groups of answers. In the first group, nearly everybody who voted was in favor of two changes:

  1. Reduce the average grant size from the current $500K to something more modest ($200K was the example in the poll). This would immediately increase accept rates by 2.5x (last year’s 7.3% would have been 18.25%. That’s a pretty big difference. Several people noted cutting grant size would negatively affect graduate students (fewer RAships), faculty at institutions/departments without TAships, and postdocs. Presumably the choice for only a modest cut was partly driven by this. Personally I would take some of the money saved and put it directly into NSF predoc and postdoc fellowships (this money doesn’t come with indirects and so is more efficient and also tips the balance of power to the students which is desirable in my opinion).
  2. Limit the number of proposals by restricting multiple grants to one researcher in a fixed period. The example given in the main text was at most one grant per five year period (once you’ve been awarded you cannot apply again). There are of course devilish details – do coPIs count, SKP count (senior key personnel=people whos CV is submitted but no salary in the grant), etc. And 5 years from award date or end of grant? etc. And while there is no perfect solution – nearly every solution will unfairly penalize some deserving person – there are certainly multiple good solutions and this is not a reason to not implement this.

Again it is remarkable that nearly everybody who voted, voted for both of these options. These options together effectively amount for a vote to spread current funding around more widely. Also note that implementing #1 almost requires some version (possibly weaker than I proposed) of #2 or you just will compound the problem of more people submitting more applications to chase fewer dollars.

Three other choices were about evenly split. To a first approximation, almost everybody voted for the two choices above, and then split evenly among the following 3 choices with their 3rd vote. To wit:

  1. Reduce grant sizes even further to $50K (not the $200K from above). This would have allowed an acceptance rate of 73%. It would have also severely limited funding (after overhead it is about $35K so roughly 3 months of summer salary or 1 year of PHD or 1/2 year of postdoc). My guess is that the thinking here is that these grants would not mostly be used for such things and instead just cover the basics of fieldwork, travel to conferences, publishing papers, etc. In short not so different from the Canadian NSERC Discovery grant. To me it is striking that across choices #1 and #3 they got a combined 47% (recall 33%=everybody voted for it if everybody voted all 3 times). – presumably a non-trivial number of people felt so strongly about this they used 2 of their 3 choices to vote for reducing grant size.
  2. Limit number of proposals by only allowing “productive researchers” to submit – this of course begs the question of how you define productive researcher. I threw out the example in the main text of 15 papers published in the last 5 years. Like #2 above this will require an arbitrary definition that hurts some deserving people, but that alone is not a reason to avoid it – especially once the rules are clear people can manage their lives around the rules (and one could imagine exemptions for early career researchers, special circumstances, etc). One reason to like this option is that studies have shown that past research success is one of the best predictors of future research success (better for example than panel evalutions of projects).
  3. Limit number of proposals by a lottery – Again many details on how this works. Is there a lottery to apply? or just a lottery for the awards among those who applied? or just a lottery among qualified scientists however that is defined? Although the lottery seems absurd on the face of it, two recent studies cited in salient fact #2 of my original post suggest that, at least among those proposals ranked moderately high (30% in the DEB case), panel scores were not that different than a lottery in predicting research outcomes. Presumably this is true for some of those that were just below the 30% cuttoff and not true for the bottom 10-15% with the line somewhere in between. Thus the lottery has the great virtue of calling a spade a spade and removing stigma from losers in what currently has a large component of lottery already but cloakings of assessment.

Then there were two “no-hopers” – essentially nobody favored these choices:

  1. Business as usual – live with the low accept rates – this got only about 2% (perhaps 5-6% of voters), meaning about 95% of voters oppose business as usual with ever declining accept rates. In the metaphor of the original post, researchers are not frogs!!  In the original post and comments a number of problems in very low accept rates (beyond the fact it makes life tough for researchers) were identified including how it distorts the selection process (more conservative, more clique-driven and of course more random), the waste of time writing 15 page proposals (at least 1 month of researcher time) for 5% success, etc.
  2. Limit proposals to certain career stages – this was the absolute least favorite choice. We academics are an egalitarian bunch. It also is not obvious that any one stage is inherently more productive.

I said in my original post I would wait to share my opinions until the poll results were in to avoid driving the results. I’m sure my biases bled through in last post and this anyway, but hopefully not terribly. But personally, I agree with everybody else – I would be in favor of some combination of #1-#5 and opposed to #6 and #7. On the cutting grant size, I of course presented arbitrary discrete options of $50K or $200K, but to me the optimum would probably be about $100K*. Over 3 years that gives $22K of direct per year. That’s enough for field work (or computers equipment or what not for field), travel to conferences, publishing fees and some consummables each year with enough left over to give a bridge year to a student, a year to a postdoc, a year of tech etc. To make this viable, I would not put all of the savings into more grants (my $100K size gives an accept rate of 36.8% – I would aim for 20-25% accept rate and put the rest into more fellowships given directly to PhD and postdocs). The sublinear response of productivity/research outcomes to dollars input strongly argues we need to move down that curve to fewer dollars per researcher where the slope of the curve and hence marginal value of research productivity bought per dollar spent increases. By the same token, I think many feel, including, me that research dollars have gotten too concentrated in a few researcher’s hands (but I know of no data on this). There are good arguments for concentrating (see my post on Shockley and lognormal productivity), but really is a superstar lab with 18 students going to get more marginal value out of one more student than a very good lab that currently has 2-3 students? I doubt it.

I personally think #4 (limit by researcher quality) and #5 (limit by lottery) have more merit than people gave them credit for too, but they are more radical changes to the system.

It is worth noting that there is enormous consensus (at least among poll respondents) to reduce grant size non-trivially and put caps on number of grants per researcher. And these are things that NSF could, if it wanted to, implement immediately. No congress, no lengthy reform processes, etc would be needed. A year or two of appropriate advance notice to researchers would be good. But beyond that these are already within the purvey of program officers to adjust budgets and recall as a commentor did that a cap of max 2 proposals per PI was placed when the pre-proposals were introduced. It would probably require consensus across a unit to make the cap global and across multiple years, but that should be achievable. Finally, note that a single unit (say DEB just for example…) could implement these as an experiment while the rest of NSF watched to see how it worked (this already happened/is happening with the pre-proposal process too). Presumably the main dynamic opposing these changes are just innate conservatism/keep-it-like-it-is and lobbying by the few but powerful that are getting large chunks of money under the current system (although I would be curious to know how many of them really think the current system is optimal).

I think more meta-research is needed too. Just what can panels successfully assess or not? Although Sam Scheiner disagreed with me in the comments on my last post, I know of very little evidence that panels can do much more than distinguish the very worst proposals from the rest (please give my citations if you think I’m wrong). If that is true we need to be scientists and deal with it, not avoid doing the research to find out because the current system is comfortable. Kudo’s to Sam and Lynnette for their paper. Similarly the question of exactly how sublinear research productivity vs grant dollars is vitally important but not yet very clear.

I have no idea what the next step is, but it seems to me that the long term trends and outlook are so extreme something has to be done (only 5% favor business as usual). And there is such a strong consensus (nearly 100%, certainly *way* over 50%) on several concrete changes which would have big impacts but would not require major restructuring such that I would be disappointed to see nothing change over the next 3 years.

Here’s hoping the community can work together to find a way to turn down the heat on the pot we’re all in!

* I am not unaware that different subdisciplines cost different amounts to do research ($100K goes less far in ecosystem science work in the tropics than it does in simple trapping or counting experiments at a site close to home). The implications of this is a whole other topic, which I am not touching here. For this post if current DEB across all subprograms has a median of $500K then it can change to $100K with differences in funding between fields untouched.


22 thoughts on “Frogs jump? researcher consensus on solutions for NSF declining accept rates

  1. I guess I was in the small minority, then, of not favoring a cut in award sizes. But for work that is personnel-intensive — and, especially, for work where it makes sense to have a technician as a major player in the project — I just don’t see how you could get the work done with such small award sizes. In your $100K award scenario, there’s no money for technicians, right? Taking some of the funds and moving them to support grad students and postdocs doesn’t fully solve the personnel issue, unless I’m missing something. Given that that would be more like the Canadian system, maybe others will chime in with ideas for how it would work.

    • Meg – I agree – $100K or $50K (for 3 years) would probably not include techs. Most people in Canada do without them from my time at McGill – Jeremy can chime in if he’s had a different experience. I think you’ve put your finger on exactly why its tempting to just live with the current system/accept rates.

      Personally, I think big changes are going to have to happen, and they’re not going to be nice. Maybe it’s no techs. Maybe its accept rates of 3%. But we should still be proactive about recognizing they’re going to happen and figuring out the least worst path.

      I certainly don’t have the answers. I don’t think anybody does. But we ought to start talking about them instead of burying our head in the sand.

      *off soap box*

      • I would argue that moving to no techs would cause its own set of problems, including encouraging people to take on more grad students because they will be relatively easy to support under a system of small grants (especially if you move funds from grants to predoctoral fellowships). Do we really want to be training lots more grad students? What will they all do once they’ve finished their PhDs? Some of it is the nature of the work that I do (lots of culture maintenance, lots of field sampling), but a technician is a really efficient way to get the work done.

        You said, “Personally, I think big changes are going to have to happen, and they’re not going to be nice. Maybe it’s no techs. Maybe its accept rates of 3%. But we should still be proactive about recognizing they’re going to happen and figuring out the least worst path.” I agree. I just am trying to point out what I see as a major downside to the move to very small grants. It would be saying, implicitly, that there isn’t a role for technicians in our science. I think that’s wrong. And it would create an incentive to train more grad students than we currently do. I don’t think that’s the best path for ecology.

      • Meg – I totally agree with you about the fact that funding techs is an important piece of not training too many PhDs and creating a glut.

        In a perfect world, highly competitive grants, subject to renewal every 3 years, is not a great way to fund what is supposed to be a permanent position. In continental Europe they are often funded institutionally – i.e. through salary from the university (ultimately from the government). They actually used to be like that in the US too! (and still a few vestiges of that at land grant universities with experimental station dollars).

        The same thing, as an aside, has happened to secretarial/administrative support. Senior faculty used to have dedicated admin support (maybe not a full FTE, but part of one).I even see the same thing happen with computer support these days.

        The gradual backing out of funding universities by states and the transference to dependency on research dollars and student-paid tuition is very pernicious and destructive.

      • Brian, along the same lines, I’m confused by your statement of “with enough left over to give a bridge year to a student, a year to a postdoc, a year of tech etc.” How does the math work out if you have only $22K/yr to support a postdoc or a tech? How little do you pay them?!

      • Good catch – I meant that if you had a 3 year grant you could get one year of postdoc.

    • Techs (and postdocs) are indeed rare in Canada. That to me is the biggest downside to the most popular suggestions here. It’s easy for NSF to cut award sizes and cap the number of grants people can hold at once. It’s not easy for NSF (or anyone) to fund more techs and fewer grad students, or to get universities to do so.

  2. Unfortunately your poll design might be fatally flawed*. An astute reader will notice that the votes for the options exactly follows the order the options were presented in. Why would this happen? Well, each reader was allowed to choose no more than three options. Imagine most readers want to vote for 5 options. They start ticking boxes starting at 1, 2, 3 and then can’t tick any more. So they submit the vote.

    On another note: an option like South Africa’s was not actually presented (similar to Canada). What this would look like is as follows… Based on last 3-5 years productivity (I know *metrics*), give money awards. Everyone meritorious gets some amount, good people get lots. (Say 50K to 500K range.) Only a few get 500K (say 5%). Wait 3-5 years, repeat (except now you are judging based on what they did with their previous award).

    Also allow roaming student awards so that students can seek out advisors they want to work with, and be funded already, rather than advisors getting money and looking for students.

    *I think, but maybe I misunderstood this

    • Hi Trevor – not as bad as you fear. The order of choices was randomized to each reader, so I don’t think there are any flaws there.

      Great to hear about the South African system. I didn’t know anything about it, but it actually sounds very good.

  3. Not surprising that a poll aimed at an audience of (mostly) junior faculty would come to the conclusion that cutting the maximum grant size is the best strategy; most junior faculty don’t have much a chance of landing large grants (outside of career grants).

    Rather than squabbling about how to divide up the deck chairs on the Titanic, I think we need to address two basic issues:

    1. Increase total NSF funding SUBSTANTIALLY. Last time I checked, the entire NSF budget – to cover computation to astrophysics to geology to ecology – was less than the R&D budget for one company, Pfizer. The US owes its economic and political position in the world to our ability to innovate; the current starving of NSF and NIH by the (largely Republican) continuation of the sequestration is sapping our scientific lifeblood. If you haven’t written your senators and your congressman about how important this issue is, you need to do so today.

    2. The current response by DEB to cope with increasing numbers of proposals – to require pre-proposals on an annual cycle – is a disaster and must be replaced. The fundamental problem, in my view, is that the annual cycle can easily lead to labs shutting down, perhaps/probably forever, perhaps as a result of an unjustified comment by one reviewer. I’d suggest (a) dropping the pre proposals, (b) return to a twice-a-year cycle; (c) retain the ceiling on two proposals per cycles, and (d) make the narrative of full proposals no more than five pages. That would save NSF and the reviewers huge amounts of time, and reduce the chances of inadvertently killing off capable labs.

    • All good comments. I certainly agree with the “rearranging the chairs on the deck of the titanic” metaphor. I guess my only question is as an individual scientists should I be betting my career on the funding levels changing radically or on making the best of the current situation? There is of course no right answer and it probably depends a lot on personality.

      I do think it is no longer just a scare tactic to talk about how America is falling behind for not investing in research. My observation is that right now continental Europe is investing much more heavily and am starting to see a brain drain of some of the best in the US moving to Europe where the grass is greener.

      I also completely agree about shortening proposals. As both a writer and reader of NSF proposals, 15 pages is still not long enough to fully scope out what I will do and I invariably get reviews about how I didn’t adequately explain X (usually because I squeezed X because last time a reviewer complained about how I didn’t adequately explain Y). Again as both a writer and a reader I much preferred the Canadian NSERC length of 5 pages – it was plenty to get the gist of what somebody would do, short enough not to fool me into thinking I could get more, and much easier to write, and indeed was just the right length to cause reviewers to focus on strategic, big-picture issues instead of get bogged down in the details.

    • Re: increasing NSF funding substantially, there certainly are good arguments for doing that, and you’ve identified the best one, I think.

      I’d just note in passing that “in order to increase funding success rates” isn’t a good reason to increase the NSF budget. Recall what happened when the NIH budget was doubled over a period of a few years: every university just started hiring more biomedical researchers and building more biomedical facilities. So that within a very short period of time, success rates were back down to (or even below) where they’d been before the budget was doubled.

      Hopefully, arguments for increasing the NSF budget aren’t mutually exclusive with arguments for how best to spend that budget. I think the measures we’re discussing (reducing mean grant size, capping the number of awards someone can hold at one time, etc.) are important to discuss whatever is happening to total NSF funding. And conversely, as I said above I think the strongest arguments for increasing NSF funding are independent of declining success rates.

      • And in your 2nd paragraph lies the nub of what I think the core, core problem is (well along with society not funding educaiton and research any more) – the entire university is now organized around NSF and NIH indirects. Its the primary reason for adding medical schools to universities that don’t have them (a push, although unsuccessful, that I observed at both UMaine and Arizona State in just my last two homes). Its one reason for the increasing tilt to science over the humanities. And exactly as you lay out, its become a pyramid game – use indirects to hire more researchers to get more indirects to hire more researchers to …. This will blow up spectacularly! And it will blow up sooner if NSF and NIH are flat funded. Arguable it is starting to do so already.

        I also agree with your final point that discussing optimum grant size and acceptance rates is and should be orthogonal to the question of total funding. Discussing the optimal configuration of the details is a healthy debate to have ongoing and ought to be had intensively every decade or so.

  4. For what it’s worth, awards are already at the ~$200K level in some subdisciplines over on the Earth Science side of things. And our acceptance rates are similar to the current ecology acceptance rates. So I guess it seems like the idea of reducing award sizes is a temporary solution. Not that I have a clue what a good solution is.

  5. Pingback: a lottery system for grant proposals | coastalpathogens

  6. Pingback: Poll: What is your risk/reward preference in science funding? | Small Pond Science

  7. Pingback: Not an April Fool’s joke: PI success rates at NSF are not dropping (much) | Dynamic Ecology

  8. Pingback: We asked, NSF answered: per-PI success rates at the NSF DEB | Dynamic Ecology

  9. Pingback: Why grant funding should be spread thinly | Scientist Sees Squirrel

  10. Pingback: New to Science Blogging #10 – Scientist Sees Squirrel › From The Lab Bench

Leave a Comment

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s