I just finished my NSERC grant (hooray!), so thought I’d fire off a quick post with some thoughts on the difference between NSERC grants and NSF grants. At the end, there’s a radical suggestion for NSF grants: do away with full proposals and just go with the preproposals.
If you don’t know, NSERC is the Canadian federal government agency that funds non-biomedical research in Canada. It’s the Canadian equivalent of NSF (US) or NERC (UK). As I’ve discussed in the past, NSERC Discovery grants (DGs) are very different beasts than NSF grants (or grants for almost any other funding agency on earth, as far as I know). Briefly, DGs are 5 pages long, and you propose your entire research program for the next 5 years, not just one project. DGs are similar to NSF preproposals in terms of length, but even that’s not really a great comparison because NSF preproposals describe a single project rather than an entire research program.
As an example, here’s my previous DG from 5 years ago.* It’s not the greatest proposal (looking back, there are things I wish I’d done differently), but it’ll give you the flavor of what DGs are like. If you’re from the US and have no experience with the Canadian system, you may want to read it sitting down so you don’t hurt yourself when you faint from shock.🙂 As a US colleague of mine said when he read a draft of my latest DG: “This is like three NSF proposals worth of work in five pages!”
My US colleague continued by saying, “Once NSF gives you the money, they don’t want you to have to think.” That is, NSF (or at least their reviewers) wants you to have thought through every methodological detail, so that if you get funded you can just go out and do exactly what you proposed to do. “Here’s the basic idea, I’ll figure out the details later, trust me”, or words to that effect, is not something you want to say in your full NSF proposal. Whereas that’s more or less exactly what you say in an NSERC proposal.
One way to look at it is to say that NSF wants to pre-approve your methods, whereas NSERC is happy to “outsource” review of methodological details to journal referees and others who evaluate NSERC-funded research. Personally, I prefer the NSERC system. Your methods are going to be evaluated by journal referees whether or not they’ve been pre-approved by NSF, which makes NSF’s pre-approval seem a bit redundant. Plus, it’s not like NSF actually requires people to do exactly what they proposed. Now, you might say that NSF needs to evaluate your proposed methods in detail in order to identify the best proposals and avoid wasting money on proposals that won’t work. But if that were true, wouldn’t it imply that NSERC is wasting billions on infeasible or seriously-flawed work, thereby burdening journal referees with the job of weeding out lots of crappy Canadian science? To ask that question is to answer it. If that was happening, the recent international review of the DG program would’ve slammed the program rather than praising it.** The NSERC example suggests to me that a granting agency can let investigators say “we’ll figure out the details later, trust us” and not thereby waste lots of money on inferior or infeasible science.***
So here’s an idea: what if NSF preproposals were the proposals? After all, one way to encourage reviewers to focus on the big ideas rather than picky methodological details is to not provide them with any methodological details in the first place. Somewhat like how, if you’re designing an online matchmaking system and don’t want potential dates to focus on less-important things like height or weight, you need to design a system that doesn’t provide those details.
Just spitballin’ here, curious to hear what people think of this.
p.s. In an old post Meg proposed two stage peer review: you could get your methods approved before collecting the data. In a sense, that’s what NSF-style proposal reviews are: a methods review prior to data collection.
*Note that this is just the proposal, not all the supplementary information (the budget, my cv, etc.). That supplementary information is very important to the evaluation of DGs.
**The linked report is very interesting reading. For instance, some of the panelists apparently were worried that the NSERC DG success rate is too high, so that NSERC is wasting significant money on inferior science while underfunding the best stuff. But the panel couldn’t find any evidence that that’s the case (and note that other lines of evidence point the same way). This isn’t to say that NSF could or should simply adopt the entire NSERC model whole hog (though they could make some moves in that direction). I think the most plausible interpretation of international comparative evidence on scientific funding systems is that lots of different systems can work. But anyone who thinks that NSERC “must” be wasting lots of money on inferior science while underfunding excellent work will have a tough time explaining how Canada does as well or better than other advanced countries on metrics of scientific productivity and influence.
***Note that NSERC and their referees do care about your methods to some extent, and don’t just blindly trust investigators to figure everything out later. For instance, in that old proposal of mine, I got dinged for not describing and justifying the genetics methods for one of the projects in sufficient detail. And because I had no previous experience with genetics, the referees weren’t willing to trust me to figure it out later. And honestly, they were probably right not to trust me–looking back, that bit of the proposal wasn’t sufficiently well-developed in my own mind. The NSERC example merely shows that investigators can establish that they know what they’re doing while providing much less methodological detail than NSF ordinarily expects. And no, the only way to do this is not to limit your proposal to methods you’ve used before–people often propose to do things they’ve never done before in their NSERC proposals. They just need to make a better case than I did that they know what they’re doing.