In (tentative) praise of shopkeeper science (UPDATED)

I grew up around grocery stores. My grandfather on my father’s side started Fox’s Market in Middletown, a small town in rural central Pennsylvania. He ran the business along with my dad and my aunt. Fox’s Markets was a success, eventually growing to four stores. No mean feat, given how competitive the grocery business is. Most people in the US have more than one grocery store within a few miles, even in a small town like Middletown. So profit margins are small, and to stay in business as an independent grocer like Fox’s you have work hard and effectively. Growing up, I remember my dad often working in his home office late into the evening. I also remember how in touch he was with everything going on in the stores. He knew all the employees by name, and when walking through the stores he’d often talk to them about all sorts of little details of the business. Some of this was him gathering information and staying in touch with how things were going (e.g., how well was a particular item selling), and some of it was him making sure things were running as they should (e.g., were the shelves neatly stocked). Because my grandfather, father, and aunt were so closely involved with every day-to-day aspect of the business, Fox’s Markets was very much a reflection of them. Some of items sold in the deli were our family recipes. Fox’s was very involved in supporting the local community, for instance by buying a lot of lambs from local farmers at the annual 4H auction at the Farm Show. Even the look of the stores–my grandfather was a serious antique collector, and so the stores were furnished with his antiques. Old advertising signs, farm implements, mannequins…One store even had an antique carousel for children to ride.

My dad and aunt sold Fox’s Markets to another local independent grocer when they retired. My brother and I both had both gone into other lines of work by that point. But while I ended up becoming a scientist rather than a grocer, in many ways I think I run my lab in the same way my grandfather, dad, and aunt ran Fox’s Markets. I don’t run a big lab. I have three graduate students, just down from four because my first one just finished. I consider that an ideal lab size–just large enough to have the critical mass for a “group”, but small enough that I have time to meet one-on-one with every student as often as needed (typically weekly during the fall and winter). In the summers I usually hire a couple of undergrad research assistants, and I supervise the occasional undergrad honors student (I’d supervise more but it’s hard to convince most undergrads that they can do ecology indoors…). While I wish I had the funds for a technician and/or postdoc (I’ve had both in the past, but infrequently), I don’t think I’d want several of them, as then I wouldn’t be able to stay in sufficiently close touch with everything that’s going on in my lab. And while I can imagine research projects I could spend massive amounts of money on, I don’t mind not having massive amounts of money, as I have plenty of good ideas for cheap research projects. Not all really good science is expensive science. To get access to large amounts of money, I’d probably have to join some kind of big collaborative project. I’d rather have the freedom to do the science I want to do, as I want to do it, contingent only on my own ability to convince others that it’s good science. So if I want to work in microcosms, or write a blog, or let my students work in systems totally different than mine, I can do that, just so long as it leads to good science.

In short, I do what might be called “shopkeeper science”. I run my lab more or less the way a shopkeeper might run his shop.

Shopkeeper science is under increasing pressure. Funding agencies in many jurisdictions are funding fewer individual investigators, in order to fund only the very best grants, or to fund “star” investigators and expensive collaborative projects (for instance, see here for an excellent summary of the Canadian situation). I think that’s a bad idea. It amounts to putting too many eggs into too few baskets, and it wastes money by throwing too much money at researchers who are probably not money-limited.

A recent analysis of Canadian data seems to back me up on this, although as I’ll discuss below the analysis has some limitations, and the policy implications are not as straightforward as they might seem. Fortin & Currie (open access) have documented impact of researchers funded by the Canadian Natural Science and Engineering Research Council (the equivalent of NSF in the US or NERC in the UK) in relation to their funding. They find that impact is only weakly related to funding, for several different measures of impact. They suggest that this is because really high impact work is rare (indeed, it’s rare almost by definition), not necessarily highly expensive, and difficult to predict in advance. Further, the impact-funding relationship is decelerating, indicating that per-dollar impact is lower for larger grant holders. And researchers who also hold grants from other agencies are not any more productive on average than those who hold only NSERC grants. The simplest interpretation is that, beyond some relatively low level of funding, researchers become increasingly limited by other factors (presumably time) rather than money. And while other measures of impact, such as number of students trained, probably do scale more tightly with funding, presumably taking funding away from big labs and giving it to smaller labs would lead to about the same number of students being trained, just by different people. So for a funding agency with a finite budget, seeking to maximize “bang for its buck”, Fortin & Currie suggest that the best policy is to give smaller pots of money to more researchers.* (UPDATE: Thanks to Carl Boettiger for bringing Fortin & Currie to my attention)

I’ll note in passing that other lines of evidence indicate that there’s deceleration in the impact-funding relationship even among highly-funded “elite” ecologists. And Fortin & Currie cite a previous study of the US NIH that found a decelerating relationship between impact and funding. So Fortin & Currie’s results aren’t necessarily just due to lack of a sufficient range of variation in funding levels, and seem like they could generalize to other funding agencies in other countries.

Fortin & Currie’s analysis does have some limitations (first noted to me by a Calgary colleague). First, their data come from a time period when NSERC funding was set very differently than it is today. It used to be the case that there was a lot of inertia in funding levels. Your funding level was ultimately determined by the amount of your first successful application, since at every renewal after that you could expect your grant to increase by about 9%, no matter what the impact of your work. Under the current system, in which there is little inertia in funding levels and in which 1/3 of your grant score depends on your track record over the past 6 years, there might actually be a stronger association between impact and funding. Second, the fact that they’re using data from the old funding system may explain why they fail to find much relationship between change in funding levels and change in impact. Observed changes in funding levels are concentrated around small positive values, due to the high inertia in the old system. Third, there may often be longer time lags between when work is funded and when it is published than Fortin & Currie’s analysis allows for. You could do more sophisticated analyses involving covariates to try to get at some of this. But offhand, I don’t see how any of those caveats would explain the observed deceleration in the impact-funding relationship.

A more important caveat is to do with the policy implications they draw. Fortin & Currie explicitly assume a world in which the impact-funding relationship for every investigator can be described by exactly the same curve (plus residual error), and use the data to estimate whether that curve is straight, accelerating, or decelerating. But of course, different people’s research impact (however measured) might scale with their funding level in different ways, for all sorts of reasons. For instance, some investigators might just be better scientists than others, able to produce more impact for any given level of funding. Or maybe the impact of some investigators increases slowly as their funding increases from low levels, but saturates at a high level, whereas the impact of other investigators increases rapidly as their funding increases from low levels but saturates at a low level. Etc. Let’s call all this “investigator heterogeneity”. Investigator heterogeneity is a really important thing to know about when it comes to optimally allocating funding. Even if the impact-funding relationship is decelerating for every individual investigator, and decelerating overall, it is not necessarily true that reallocating funding away from highly-funded investigators towards other investigators will increase bang for the buck.

Here’s a simple hypothetical illustration of that point. The figure below shows the relationship between impact and funding for three different hypothetical investigators: Dr. Green (highest line), Dr. Red (middle line), and Dr. Blue (lowest line). Every line takes the same power law form y=ax^b assumed by Fortin & Currie. All the relationships are decelerating (b=0.5 for all investigators). But I introduce investigator heterogeneity by allowing the parameter a to vary among investigators (a=1, 2, 3 for Drs. Blue, Red, and Green respectively). Allowing a to vary is an arbitrary choice; I could’ve varied b, or both a and b, to make the same point.

The filled black points show the funding levels for each investigator, and the corresponding impact. Notice that these points define a decelerating curve. This (not the green, red, and blue curves) is the sort of decelerating curve Fortin & Currie found. The fact that the green, red, and blue curves don’t all fall right on top of the curve defined by the black points represents investigator heterogeneity, which matters for reasons I’ll explain in a second. In this example, the three investigators share $50,000 of funding and produce a total impact of 822.1, for an impact/funding ratio of 0.016

The open points show what happens if we reallocate that $50,000 equally to all three investigators. Obviously, the funding and impact of Drs. Red and Blue increases, while the funding and impact of Dr. Green drops. But here’s the key point: because of investigator heterogeneity, the increased impact of Drs. Red and Blue doesn’t fully compensate for the decreased impact of Dr. Green, much less overcompensate as it would if there were no investigator heterogeneity. Total impact drops to 774.6, for an impact/funding ratio of 0.015. Oops.

Put another way, if all investigators were identical as Fortin & Currie assume, changing their funding levels would shift their impact along the curve defined by the black points. But investigators aren’t all identical, and so that’s not how their impact shifts.

Now this is an arbitrary example. I could’ve constructed an example in which allocating funding equally among investigators dramatically increases total productivity–or decreases it even more. And I could’ve shown an example in which funding is reallocated in some other way that doesn’t equalize everybody’s funding. But none of that matters, because my point is simply that investigator heterogeneity hugely complicates the task of optimal allocation of funding.

If I’m not mistaken, the optimal thing for a funding agency to do, at least if all investigator’s impact-funding curves are decelerating, is to fund investigators so that they all produce the same impact per dollar (correct me if I’m wrong!) (UPDATE: in the comments, Ric Charnov reminds me of something I should’ve remembered: the optimal funding allocation equalizes the marginal productivity of each investigator–the slopes of the curves.) But you can’t do that unless you know the shape of the impact-funding curve for each investigator, and I doubt you can reliably estimate those shapes from the available data.

So where does this leave the argument for shopkeeper science? I’m not sure. I still think that giving a few investigators large grants (or multiple large grants!) while giving most investigators nothing is a really bad idea. High impact science is sufficiently unpredictable that putting all your eggs in a few baskets just seems really unwise. But even if you grant me that point and agree that lots of people ought to get at least small grants, there’s still a huge range of possible allocations of funding among those people. And I have no idea how to choose among those allocations. Plus, there’s clearly much more that goes into determining the collective scientific impact and “bang for the buck” of a country’s scientists besides the funding allocation system used by its granting agencies.

p.s. Fortin & Currie find that a researcher’s own past impact is a much stronger predictor of their future impact than is their funding. So that changes in funding for individual researchers aren’t associated with changes in researcher impact. This could be, and probably is, for various reasons.

p.p.s. The policy implications of Fortin & Currie’s results actually are unclear for other reasons besides those discussed above. One could argue for redistributing the pot: cutting funding from highly-funded researchers and redistributing it to researchers who have less, or have nothing. I take it that’s more or less what Fortin & Currie would argue for. But as a Calgary colleague of mine suggested to me, one could also argue for shrinking the pot: just cut funding from highly-funded researchers and don’t spend it on science at all! On the grounds that, if scientists’ impact is only weakly related to their funding levels, you can just cut everyone’s funding down to some low level without causing much damage. My point about investigator heterogeneity might provide one way to push back against this argument for shrinking the pot, but I’m not sure.

*Not that there might not be reasons to do the opposite. For instance, here’s a provocative suggestion that giving out many modestly-sized grants leads to lots of false results because researchers can only afford modest sample sizes.