Dynamic Ecology’s readers have spoken in a clear voice! There is a clear consensus around what changes people favor to address the hopelessly declining grant award rates at NSF. In a post on Monday I described what I see as the long-term exogenous trends in our society (US specifically but as commenters noted probably largely applicable globally) that affect NSF. And that are putting in NSF in a tight squeeze leading to a current acceptance rate of 7.3% and every expectation it will go still much lower. Basically funding flat, many pressures on researchers to apply for more grants (both more applications from old hands and pressure on many to begin applying) lead to a trade-off, the only variables of which NSF controls is # of applications and grant size in $.
I had a reader poll on what choices readers would like to see NSF adopt. To be clear this poll is entirely unscientific in sample design – its whoever reads the blog and answers. It is presumably mostly academic ecologists, and our readership skews early career and male (although how much more it does so than academic ecology in general is unknown), but beyond that I couldn’t say what biases there are. There were 450 votes – I don’t know how many voters since each voter could vote up to 3 times and polldaddy doesn’t give me the details unless I pay them $200 (I know there are other choices – I’ll probably use them next time but polldaddy is so convenient from inside WordPress). But to a first approximation there were presumably about 160-175 voters (some voters likely voted for only 1 or 2 choices). The results as of 11:30AM EST Wednesday (in my experience the vast majority of people who will read the post have read it by now) are:
Basically there are three groups of answers. In the first group, nearly everybody who voted was in favor of two changes:
- Reduce the average grant size from the current $500K to something more modest ($200K was the example in the poll). This would immediately increase accept rates by 2.5x (last year’s 7.3% would have been 18.25%. That’s a pretty big difference. Several people noted cutting grant size would negatively affect graduate students (fewer RAships), faculty at institutions/departments without TAships, and postdocs. Presumably the choice for only a modest cut was partly driven by this. Personally I would take some of the money saved and put it directly into NSF predoc and postdoc fellowships (this money doesn’t come with indirects and so is more efficient and also tips the balance of power to the students which is desirable in my opinion).
- Limit the number of proposals by restricting multiple grants to one researcher in a fixed period. The example given in the main text was at most one grant per five year period (once you’ve been awarded you cannot apply again). There are of course devilish details – do coPIs count, SKP count (senior key personnel=people whos CV is submitted but no salary in the grant), etc. And 5 years from award date or end of grant? etc. And while there is no perfect solution – nearly every solution will unfairly penalize some deserving person – there are certainly multiple good solutions and this is not a reason to not implement this.
Again it is remarkable that nearly everybody who voted, voted for both of these options. These options together effectively amount for a vote to spread current funding around more widely. Also note that implementing #1 almost requires some version (possibly weaker than I proposed) of #2 or you just will compound the problem of more people submitting more applications to chase fewer dollars.
Three other choices were about evenly split. To a first approximation, almost everybody voted for the two choices above, and then split evenly among the following 3 choices with their 3rd vote. To wit:
- Reduce grant sizes even further to $50K (not the $200K from above). This would have allowed an acceptance rate of 73%. It would have also severely limited funding (after overhead it is about $35K so roughly 3 months of summer salary or 1 year of PHD or 1/2 year of postdoc). My guess is that the thinking here is that these grants would not mostly be used for such things and instead just cover the basics of fieldwork, travel to conferences, publishing papers, etc. In short not so different from the Canadian NSERC Discovery grant. To me it is striking that across choices #1 and #3 they got a combined 47% (recall 33%=everybody voted for it if everybody voted all 3 times). – presumably a non-trivial number of people felt so strongly about this they used 2 of their 3 choices to vote for reducing grant size.
- Limit number of proposals by only allowing “productive researchers” to submit – this of course begs the question of how you define productive researcher. I threw out the example in the main text of 15 papers published in the last 5 years. Like #2 above this will require an arbitrary definition that hurts some deserving people, but that alone is not a reason to avoid it – especially once the rules are clear people can manage their lives around the rules (and one could imagine exemptions for early career researchers, special circumstances, etc). One reason to like this option is that studies have shown that past research success is one of the best predictors of future research success (better for example than panel evalutions of projects).
- Limit number of proposals by a lottery – Again many details on how this works. Is there a lottery to apply? or just a lottery for the awards among those who applied? or just a lottery among qualified scientists however that is defined? Although the lottery seems absurd on the face of it, two recent studies cited in salient fact #2 of my original post suggest that, at least among those proposals ranked moderately high (30% in the DEB case), panel scores were not that different than a lottery in predicting research outcomes. Presumably this is true for some of those that were just below the 30% cuttoff and not true for the bottom 10-15% with the line somewhere in between. Thus the lottery has the great virtue of calling a spade a spade and removing stigma from losers in what currently has a large component of lottery already but cloakings of assessment.
Then there were two “no-hopers” – essentially nobody favored these choices:
- Business as usual – live with the low accept rates – this got only about 2% (perhaps 5-6% of voters), meaning about 95% of voters oppose business as usual with ever declining accept rates. In the metaphor of the original post, researchers are not frogs!! In the original post and comments a number of problems in very low accept rates (beyond the fact it makes life tough for researchers) were identified including how it distorts the selection process (more conservative, more clique-driven and of course more random), the waste of time writing 15 page proposals (at least 1 month of researcher time) for 5% success, etc.
- Limit proposals to certain career stages – this was the absolute least favorite choice. We academics are an egalitarian bunch. It also is not obvious that any one stage is inherently more productive.
I said in my original post I would wait to share my opinions until the poll results were in to avoid driving the results. I’m sure my biases bled through in last post and this anyway, but hopefully not terribly. But personally, I agree with everybody else – I would be in favor of some combination of #1-#5 and opposed to #6 and #7. On the cutting grant size, I of course presented arbitrary discrete options of $50K or $200K, but to me the optimum would probably be about $100K*. Over 3 years that gives $22K of direct per year. That’s enough for field work (or computers equipment or what not for field), travel to conferences, publishing fees and some consummables each year with enough left over to give a bridge year to a student, a year to a postdoc, a year of tech etc. To make this viable, I would not put all of the savings into more grants (my $100K size gives an accept rate of 36.8% – I would aim for 20-25% accept rate and put the rest into more fellowships given directly to PhD and postdocs). The sublinear response of productivity/research outcomes to dollars input strongly argues we need to move down that curve to fewer dollars per researcher where the slope of the curve and hence marginal value of research productivity bought per dollar spent increases. By the same token, I think many feel, including, me that research dollars have gotten too concentrated in a few researcher’s hands (but I know of no data on this). There are good arguments for concentrating (see my post on Shockley and lognormal productivity), but really is a superstar lab with 18 students going to get more marginal value out of one more student than a very good lab that currently has 2-3 students? I doubt it.
I personally think #4 (limit by researcher quality) and #5 (limit by lottery) have more merit than people gave them credit for too, but they are more radical changes to the system.
It is worth noting that there is enormous consensus (at least among poll respondents) to reduce grant size non-trivially and put caps on number of grants per researcher. And these are things that NSF could, if it wanted to, implement immediately. No congress, no lengthy reform processes, etc would be needed. A year or two of appropriate advance notice to researchers would be good. But beyond that these are already within the purvey of program officers to adjust budgets and recall as a commentor did that a cap of max 2 proposals per PI was placed when the pre-proposals were introduced. It would probably require consensus across a unit to make the cap global and across multiple years, but that should be achievable. Finally, note that a single unit (say DEB just for example…) could implement these as an experiment while the rest of NSF watched to see how it worked (this already happened/is happening with the pre-proposal process too). Presumably the main dynamic opposing these changes are just innate conservatism/keep-it-like-it-is and lobbying by the few but powerful that are getting large chunks of money under the current system (although I would be curious to know how many of them really think the current system is optimal).
I think more meta-research is needed too. Just what can panels successfully assess or not? Although Sam Scheiner disagreed with me in the comments on my last post, I know of very little evidence that panels can do much more than distinguish the very worst proposals from the rest (please give my citations if you think I’m wrong). If that is true we need to be scientists and deal with it, not avoid doing the research to find out because the current system is comfortable. Kudo’s to Sam and Lynnette for their paper. Similarly the question of exactly how sublinear research productivity vs grant dollars is vitally important but not yet very clear.
I have no idea what the next step is, but it seems to me that the long term trends and outlook are so extreme something has to be done (only 5% favor business as usual). And there is such a strong consensus (nearly 100%, certainly *way* over 50%) on several concrete changes which would have big impacts but would not require major restructuring such that I would be disappointed to see nothing change over the next 3 years.
Here’s hoping the community can work together to find a way to turn down the heat on the pot we’re all in!
* I am not unaware that different subdisciplines cost different amounts to do research ($100K goes less far in ecosystem science work in the tropics than it does in simple trapping or counting experiments at a site close to home). The implications of this is a whole other topic, which I am not touching here. For this post if current DEB across all subprograms has a median of $500K then it can change to $100K with differences in funding between fields untouched.