What if NSF preproposals WERE the proposals?

I just finished my NSERC grant (hooray!), so thought I’d fire off a quick post with some thoughts on the difference between NSERC grants and NSF grants. At the end, there’s a radical suggestion for NSF grants: do away with full proposals and just go with the preproposals.

If you don’t know, NSERC is the Canadian federal government agency that funds non-biomedical research in Canada. It’s the Canadian equivalent of NSF (US) or NERC (UK). As I’ve discussed in the past, NSERC Discovery grants (DGs) are very different beasts than NSF grants (or grants for almost any other funding agency on earth, as far as I know). Briefly, DGs are 5 pages long, and you propose your entire research program for the next 5 years, not just one project. DGs are similar to NSF preproposals in terms of length, but even that’s not really a great comparison because NSF preproposals describe a single project rather than an entire research program.

As an example, here’s my previous DG from 5 years ago.* It’s not the greatest proposal (looking back, there are things I wish I’d done differently), but it’ll give you the flavor of what DGs are like. If you’re from the US and have no experience with the Canadian system, you may want to read it sitting down so you don’t hurt yourself when you faint from shock.🙂 As a US colleague of mine said when he read a draft of my latest DG: “This is like three NSF proposals worth of work in five pages!”

My US colleague continued by saying, “Once NSF gives you the money, they don’t want you to have to think.” That is, NSF (or at least their reviewers) wants you to have thought through every methodological detail, so that if you get funded you can just go out and do exactly what you proposed to do. “Here’s the basic idea, I’ll figure out the details later, trust me”, or words to that effect, is not something you want to say in your full NSF proposal. Whereas that’s more or less exactly what you say in an NSERC proposal.

One way to look at it is to say that NSF wants to pre-approve your methods, whereas NSERC is happy to “outsource” review of methodological details to journal referees and others who evaluate NSERC-funded research. Personally, I prefer the NSERC system. Your methods are going to be evaluated by journal referees whether or not they’ve been pre-approved by NSF, which makes NSF’s pre-approval seem a bit redundant. Plus, it’s not like NSF actually requires people to do exactly what they proposed. Now, you might say that NSF needs to evaluate your proposed methods in detail in order to identify the best proposals and avoid wasting money on proposals that won’t work. But if that were true, wouldn’t it imply that NSERC is wasting billions on infeasible or seriously-flawed work, thereby burdening journal referees with the job of weeding out lots of crappy Canadian science? To ask that question is to answer it. If that was happening, the recent international review of the DG program would’ve slammed the program rather than praising it.** The NSERC example suggests to me that a granting agency can let investigators say “we’ll figure out the details later, trust us” and not thereby waste lots of money on inferior or infeasible science.***

So here’s an idea: what if NSF preproposals were the proposals? After all, one way to encourage reviewers to focus on the big ideas rather than picky methodological details is to not provide them with any methodological details in the first place. Somewhat like how, if you’re designing an online matchmaking system and don’t want potential dates to focus on less-important things like height or weight, you need to design a system that doesn’t provide those details.

Just spitballin’ here, curious to hear what people think of this.

p.s. In an old post Meg proposed two stage peer review: you could get your methods approved before collecting the data. In a sense, that’s what NSF-style proposal reviews are: a methods review prior to data collection.

*Note that this is just the proposal, not all the supplementary information (the budget, my cv, etc.). That supplementary information is very important to the evaluation of DGs.

**The linked report is very interesting reading. For instance, some of the panelists apparently were worried that the NSERC DG success rate is too high, so that NSERC is wasting significant money on inferior science while underfunding the best stuff. But the panel couldn’t find any evidence that that’s the case (and note that other lines of evidence point the same way). This isn’t to say that NSF could or should simply adopt the entire NSERC model whole hog (though they could make some moves in that direction). I think the most plausible interpretation of international comparative evidence on scientific funding systems is that lots of different systems can work. But anyone who thinks that NSERC “must” be wasting lots of money on inferior science while underfunding excellent work will have a tough time explaining how Canada does as well or better than other advanced countries on metrics of scientific productivity and influence.

***Note that NSERC and their referees do care about your methods to some extent, and don’t just blindly trust investigators to figure everything out later. For instance, in that old proposal of mine, I got dinged for not describing and justifying the genetics methods for one of the projects in sufficient detail. And because I had no previous experience with genetics, the referees weren’t willing to trust me to figure it out later. And honestly, they were probably right not to trust me–looking back, that bit of the proposal wasn’t sufficiently well-developed in my own mind. The NSERC example merely shows that investigators can establish that they know what they’re doing while providing much less methodological detail than NSF ordinarily expects. And no, the only way to do this is not to limit your proposal to methods you’ve used before–people often propose to do things they’ve never done before in their NSERC proposals. They just need to make a better case than I did that they know what they’re doing.

10 thoughts on “What if NSF preproposals WERE the proposals?

  1. Re: NSF wanting to “pre-approve” your methods, it’s interesting to imagine what would happen if this *weren’t* redundant with journal review. That is, what if, when you submitted a paper, you could cite your funded NSF grant to say “These methods were pre-approved by NSF”? And that if you said that, referees weren’t ordinarily allowed to criticize your methods?

    This gets back to Meg’s old post proposing “two-stage peer review”: https://dynamicecology.wordpress.com/2013/10/21/two-stage-peer-review-of-manuscripts-methods-review-prior-to-data-collection-full-review-after/. And also two the various proposals for pre-registration of planned studies (here, getting an NSF grant funded and then doing exactly what you proposed to do functions as preregistration).

  2. As someone who is as-yet unfunded but applying for a lot of proposals, I have mixed feelings about this. Despite NSF’s hopes (and occasional claims to the contrary), pre-proposals do seem to have hurt early-career faculty, who have a lower success rate. I’ve seen some people suggest that this is due to (in some cases observed) situations where reviewers will give a senior researcher the benefit of the doubt if the methods aren’t as solid, because they have a known track record of productivity. In contrast, an early-career researcher may have a harder time proving themselves in a pre-proposal format. Maybe this is excessive paranoia, but what I’ve seen and heard does concern me a little.

    If we -do- do away with the full proposals, I’d like to see the cap of two pre-proposals either raised or eliminated for early-career folks at the very least.

    • Good point. The way NSERC deals with this is that new investigators can be funded with a lower evaluation score than investigators who’ve previously held an NSERC grant. This is to allow for new investigators having less of a track record. New investigators in the NSERC system do have a lower success rate than established investigators, but not massively lower (Plus, success rates in the NSERC system are still so high in an absolute sense that it’s hard to argue that the lower success rate for new investigators is a serious barrier to their career progress. Of course, making that true for NSF would require a much more radical change to how NSF operates.)

  3. Pingback: What we’re reading: The color of cichlids, projected genomes, and simplifying NSF proposals | The Molecular Ecologist

  4. NSF and NIH proposals in the U.S. can border on the farcical. It’s certainly reasonable for funding agencies to expect a coherent conceptual outline of methodology, but especially for new investigators, describing methods in the expected detail would require psychic powers. To develop new methodologies, one has to first think of what they might be, and sometimes that can’t happen until you take the first steps to investigate your hypotheses.

    • “Require psychic powers” or that you have already done the research – which is really closer to what is going on

  5. Pingback: Links 10/27/14 | Mike the Mad Biologist

  6. I think my opinion falls somewhere in the middle of the two extremes.

    I’d agree with drjuliebug that sometimes NSF proposals can enter the realm of the ridiculous with regard to nitpicking. However, my sense is that the nitpicking is actually a product of low funding rates more than anything else. When you’ve winnowed your pool down to only excellent proposals and you still have three times as many as you can fund, you have to find some way to make decisions.

    However, I’m still a fan of full proposals because there is a lot of value in thinking through ideas carefully. I never feel as immersed in my science as I do soon after writing a full proposal because it requires not just considering what is broadly interesting, but also forces you to extend those ideas to their logical conclusions with your system, and to think about how you’ll actually address those broad questions at a very practical level.

    • Interesting Bryan. So basically, you appreciate being forced to think everything through at the application stage, rather than having the freedom to think through the details after getting the grant.

      Re: nitpicking as necessary because you’ve got to choose among excellent proposals somehow, yeah, I suspect there’s a lot to that. One response to that is to note that nitpicking isn’t the only way to choose. For instance, you could winnow the proposals down to only the excellent ones, and then choose among the excellent ones via a lottery. There’s not much evidence that the nitpicking improves the science that gets funded–indeed, it’s arguably just a lottery by another name. If you’re going to have a lottery in practice, might as well be honest about it and actually *have* a lottery!

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s