Poll: Should grant applicants be evaluated relative to how much funding they’ve received?

When I apply for an NSERC Discovery Grant, 1/3 of my evaluation score is based on my scientific productivity over the previous six years. NSERC calls this “excellence of the researcher”. Reviewers look at the quality, impact, and importance of my papers and other contributions to science. Many funding bodies do something similar, though the details vary.

NSERC instructs reviewers not to treat funding level as an indicator of excellence. You’re not supposed to infer that someone who has lots of funding is doing great science, or that someone who has little or none is doing weak science. But of course, funding is correlated with scientific productivity. No perfectly correlated or even linearly correlated, of course, but correlated. That’s the whole point of giving scientists money—so that they can produce more and better science! Which is exactly what any scientist will do, if given more money.

So here’s my question: when evaluating “excellence of the researcher”, should reviewers evaluate excellence relative to the amount of funding the researcher had? So that researchers who’ve been very productive—but also very well funded—would be evaluated less well than they would be if reviewers were just asked “how productive has the applicant been?”

I think there’s a strong argument that grant applicants should be evaluated relative to their previous funding level, although NSERC doesn’t provide any instruction to reviewers one way or the other.* Indeed, I know of at least one person who does this when reviewing NSERC Discovery Grants. But I don’t know how common it is. And it’s kind of difficult to do, for various reasons. For instance, how do you allow for differences in cost among different research approaches? How do you allow for the fact that probably every researcher’s productivity is some nonlinear, decelerating function of their funding, making it likely that researchers will less funding will be more productive on a per-dollar basis than researchers with more? And how do you allow for the fact that the height and shape of those nonlinear functions presumably varies among applicants? Although presumably such difficulties are mitigated in a system like NSERC’s, in which reviewers only make fairly coarse distinctions among applicants (scoring their “excellence” on a 6-point scale).

What do you think? As a conversation starter, here’s a little poll:

Looking forward to your comments.

UPDATE: I forgot it was a US holiday when I put this post up yesterday. #amateurhour So we didn’t get many votes. But of the 78 votes we got, 56% agreed that grant applicants should be judged relative to their previous funding level, 35% disagreed, and 9% weren’t sure. So based on this admittedly small and non-random sample, there’s a lot of disagreement on this issue!

*Do other funding agencies provide explicit instructions on this?

p.s. I’m intentionally not getting into the issue of whether funding agencies should look at applicants’ track records at all, or whether they should make use of that information in some different way than NSERC does (say, only asking if the applicant has the background and experience needed to carry out the proposed research). Those are interesting questions, but I’m setting them aside for purposes of this post.

14 thoughts on “Poll: Should grant applicants be evaluated relative to how much funding they’ve received?

  1. Absolutely. If a lab has a bazillion dollars and has a bunch of papers, then that really isn’t all that special. I’ve been on NIH study sections where a hot shot lab pitched a high budget project to go along side their other high budget projects with funding from NIH and private foundations. They claimed they were very productive when they were really just well funded. Usually the problem is the converse where a lab has had a 5 yr project and only has one or two papers.

    NIH did a study of funding vs. productivity with some interesting results: http://www.nature.com/news/2010/101116/full/468356a.html. It’s pretty clear that productivity asymptotes with funding, albeit at a relatively high level ($750K for NIH). I also suspect that the “Number of grant-linked publications” is easily biased upwards. Because study sections and program officers want to know that their money is well spent, it’s in my best interest to attribute multiple sources of funding to the same paper. Also, even if a grant has expired, I can attribute that grant to a paper claiming that the project started while the grant was ongoing.

    I do think productivity is an important factor since we should value people that know how to stretch a $. I’m pretty adamant that productivity should be measured relative to the past levels of funding. Although NIH doesn’t direct us one way or another on this, it certainly enters my personal calculations.

  2. I voted yes, but the complexities noted in post are well taken:

    “How do you allow for differences in cost among different research approaches? How do you allow for the fact that probably every researcher’s productivity is some nonlinear, decelerating function of their funding?”

    What would happen if we did indeed evaluate researchers based on a metric of productivity per dollar? For argument’s sake, how about Impact Factor / $: add up the IFs for all publications and divide by reported funding. With this approach, scientists using inherently high-cost approaches would receive less funding, and (assuming the decelerating function above) later-career scientists who are “sticking to their guns” would also receive less funding. The consequence would be to compel science to seek cheaper approaches or shift away from their historical foci and plow more fertile fields. Is this a bad thing? If higher IF means greater scientific progress, then these policies would lead to greater progress per dollar. Given a limited amount of dollars, that means greater progress.

    I think I’m for it, but I would love to hear counterpoint.

    • I think the counterpoint is that we want to fund a mix of science on all sorts of topics. You don’t want to just give up on, say, particle physics and astronomy (and the super-expensive particle smashers, neutrino detectors, satellites, and telescopes they require).

      Put another way, comparing “productivity” across fields, or even subfields within fields, isn’t really sensible. How many discoveries in ecology equals one Higgs boson?

      Of course, decisions on funding allocation do still get made–they have to be. But it’s not really sensible to try to make those allocations by trying to optimize bang for the buck, because there is no single measure of “bang”; it doesn’t exist.

      • Hi Jeremy, the statements “comparing productivity across fields isn’t sensible” and “funding allocations still get made” don’t sit very well together. So, why is there so much more money for health research than ecology? Isn’t it because there is a general perception that humanity will benefit more from health research than ecology research? If it’s not then what is the rationale? When valuation is implicit in every decision that gets made I think it’s a mistake to avoid explicit valuation (just because it’s difficult). And I would say valuation is implicit in every funding decision that gets made. Best, Jeff.

      • We don’t disagree, at least not much. Valuation is indeed implicit in every funding decision. But it’s still difficult or impossible to make it explicit.

        Whether the attempt to make it explicit is helpful is an open question. This is where we might disagree a bit. For instance, trying to make the criteria for a complex judgement explicit can easily lead to distortions, by leading to an undue focus on measurable outcomes.

      • Jeff- I can relate to your comment concerning the level of health funding. Prior to going into medicine I was somewhat resentful of the disparity between health and environmental science funding. But after a dozen years in medicine (I returned to ecology in 2009) my perspective very much changed. People are people, and the evolutionary processes giving rise to familial bonds are profound, as are the bonds themselves.

        So at the level of policy I believe, anyway, those bonds very much influence decisions. It is also important to understand that with modern society as complex as it has become, disruptions to human health can be devastating on all fronts. Imagine, if you will, running out of effective antibiotics for things like streptococcus. In 2004 I was stricken with an antibiotic-resistant form of strept on my skin that I contracted at my “health” club, of all places. It was a touch & go ordeal and I nearly faced amputation of both legs. Medicine, and the gazillions we spend on it spared me that fate.

        Perception is reality in politics and allocation of public funds. I predict however that when climate change reaches the point of disrupting food supply our focus will change. It is unfortunate it will take an event like this for the public to understand survival involves things other than a CT scan & a pill, but that’s the hand we were dealt.

    • First to your point, Jeremy – I’ve never been clear on how implicit criteria can ever be better than explicit criteria. Some person or group of people is going to make decisions about the relative amounts to be given to different disciplines and I just can’t see how it will be done better if we have vague and ill-defined criteria versus explicit, concrete criteria. The ‘problem’ with explicit criteria is they are easy to criticize – the target is clear. The only advantage I see of implicit criteria is that they are tough to criticize because the target is fuzzy. And you’re right – explicit criteria implies ‘things that can be measured’. But I’m not very comfortable with criteria that can’t be measured. Is there an argument to be made in favour of using outcomes that can’t be measured as criteria for decision making? But, I also get that this can result in a rigid system that doesn’t allow for differences in valuation among individuals. This is not a simple problem
      Dave, I wasn’t trying to imply that health sciences gets too large a share of the pie (although I admit that it sounded like that). I don’t think there is any doubt that over the last 100 years people have benefited far more from health research than ecological research and that’s a pretty good reason to give health research more money. Your personal story is a good example – over the past 100 years health research is way more likely to have saved somebody’s life than ecological research. My point is that disciplines that get a lot of money should have to make an explicit case – I think medical research would have a pretty easy case to make. Jeff H

      • I think if you want to judge which research to do based on outcome-related criteria that are explicit and specific enough to be useable, you’re pretty much not going to have any fundamental research. Which in the long run is a very bad idea, even though in the long run it will hard to trace how we’re all worse off because of lack of fundamental research that wasn’t done.


        FWIW, the world is moving in the direction you suggest. Most national governments and funding agencies are moving in the direction of supporting applied research (including subsidizing private R&D) for which the case for relevance is easy to make.

  3. Related to this point, should you compare the output of a PhD (say for post-doc applications) in light of funding levels?
    If two students produce roughly the same output in the same amount of time (say 4 papers in roughly equivalent journals), but one TA’ed throughout, while the other had a fellowship, then which student would get the post-doc award?
    It seems that most agencies would go with the fellowship student (i.e. Matthew effect) even though the TAing student arguably did more during their PhD and is better prepared for a faculty job (teaching + research vs. just research). The reasoning behind one student getting funding and the other not can be pretty trivial.

    • That’s a good point. Though it also highlights the limits of the logic of the post. For instance, what if those two students don’t produce roughly the same output? Say the one who TAed a lot produced less? How much less does it have to be before one would prefer the student who had the fellowship for the postdoc?

  4. I think your comment about different approaches really matters a lot. If people were to be reviewed relative to funding, I think it would be imperative to see budgets when making the review. If the majority of budget is going to supplies and equipment (versus lab manager / grad student / postdoc salaries), I would suspect that there isn’t much added productivity.

    For example, it costs ~$10,000/month to run the Serengeti Lion Project. Most of this amount is accounted for by the cost of petrol in remote Tanzania and the constant maintenance required on vehicles that drive off-road day-in and day-out. It’s a very expensive project, but unique in what it does. Its scientific output (papers, talks, etc.) is not higher than a lab-based group or a modeling-based group, though they tend to be high-impact.

    And then what do you do about non-direct funding — or funding from “other” sources? For example, if you work at a LTER (or NEON in the future) site that has lots of base measurements then you, as the PI, don’t have to worry about paying for that data collection. Or, what if you get a grant from a state or municipal government, or non-profit group, or crowd-fund a chunk of your research? Would you ask for people being reviewed to provide a listing of all their funding sources?

    I chose “not sure”, because while it seems reasonable at the outset, it’s really a complicated thing to implement in a fair way.

    • Yes, good examples. Cost differences among research programs are a really challenging issue here. Same for other non-direct funding.

      My own feeling is that reviewers can and should take all this into account in a broad-brush way. Not aiming to make over-fine distinctions among applicants, but aiming to identify those people who’ve been impressively productive or impressively unproductive, given the resources available to them, what it costs to do the sort of work they do, and any alternative ways of answering whatever question they’re asking.

      • Hmm… then maybe would it make sense to “bucket” people somehow?

        These twenty people had more-or-less the same amount of resources and do the same sort of work. Do any of them stick out as particularly productive or unproductive?

        (But then, the “particularly unproductive” makes me worry too… What about new parents (especially mothers)? What about a new chronic health condition? What about those times in life when everything bad seems to happen all at once? Many (most?) people will run into a string of a few years where they are less productive due to the stochastic of life.)

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s