The story and lessons of the NutNet experiment: an interview with Elizabeth Borer

The Nutrient Network (“NutNet“) is a long-running, pioneering globally distributed field experiment. It now involves hundreds of researchers at hundreds of sites around the world, it’s published a bunch of important and influential papers, and it’s served as the inspiration and model for many other distributed experiments. I’m a huge fan of NutNet, not just because of all the great science that’s come out of it, but because it’s a new and very interesting model for how to do science (at least, new to me…) I’m also a fan of NutNet because it’s a very different sort of science than anything I do or would have even thought of doing. When I look at someone else’s protist microcosm experiment, I look at it with the eyes of a connoisseur, because I do protist microcosm experiments too. But when I look at NutNet, I’m just amazed, it’s like I’m looking at a magic trick or the Sagrada Família. How does someone do that? How does someone even think of doing it?

To answer those questions, I asked one of the someones who did it. 🙂 Elizabeth Borer is one of the co-founders of NutNet. In the same spirit as my interview a few years ago with Rich Lenski about his Long-Term Evolution Experiment, I emailed Elizabeth a bunch of questions about NutNet and she was kind enough to answer them. I hope you find her answers as interesting as I do (seriously, they’re super-interesting)!

1. How did you and the other founders come up with the idea for NutNet? 

The idea for NutNet came about during a working group meeting at the National Center for Ecological Analysis and Synthesis (NCEAS) in Santa Barbara. A group of us wanted to better understand the interactions among multiple changing factors – when two or more factors change at the same time, do we simply get additive responses that we could predict from each alone? Or are responses stronger or weaker than we’d expect? This was a really important gap in our knowledge: humans are changing the cycling and inputs of nitrogen and phosphorus and other elements while concurrently changing the densities and identities of predators and herbivores. It’s possible that plants will respond more strongly to nitrogen if phosphorus has been added. Or, for example, if added nutrients make plants more nutritious, herbivores will prefer them and eat even more. But if the plants invest some of the added nutrients in defense structures or compounds, maybe herbivores will eat less. And we wanted to understand how general the responses were across ecosystem types – do freshwater or marine systems with primarily unicellular producers act differently than terrestrial systems, or do they all show similar responses?

To answer these questions, we used meta-analysis, pulling together a library of experimental papers that had either manipulated multiple different kinds of nutrients or crossed nutrient supply with either a predator or herbivore exclusion treatment. We’d been working on analyses and finding really interesting, general responses across ecosystem types (freshwater, marine, terrestrial, etc). But a big challenge with meta-analysis is that every study is done differently – different spatial scales, rates of nutrient addition, timing of data collection, etc – so it is difficult to separate out real biological patterns in the responses from patterns due to differing methodology. Meta-analysis is a great tool, but, in addition to problems of inference from the studies that *are* in the literature, we were finding that scientists tend to ask and answer (and publish) certain questions in certain ecosystems; in some ecosystems there simply were no data because these types of experiments had not been done.

During a coffee break, we were talking about these shortcomings of meta-analysis. We (jokingly) agreed that all scientists should be required do their experiments the same way so that we could have stronger inference when we combined them. It was in this moment that we were inspired to do an experiment to fill this gap. We would all use the exact same experimental design and treatments and we’d use the same methods to collect standardized response data. Our own experiences – our work within the US LTER program, our own experimental research replicated across a handful of distributed sites, and our comfort with collaborative science – as well as the context of science at the time – the NEON program was in the planning phase with a goal of a spatially-distributed observational network – likely helped us take the next step of dreaming up this collaborative experimental network. Across a few days, during breaks, we met and designed a pair of experiments to test hypotheses about multiple nutrients and herbivores in grasslands.

We decided we’d collect three different datasets but keep time and costs as low as possible without sacrificing data quality. The first dataset would be an observational one that would simply allow us to compare species diversity, productivity, composition, and soils across each of these sites. The second would be a nitrogen x phosphorus x potassium-and-micronutrients factorial combination – to test questions about nutrient interactions. The third would take advantage of the all nutrient and no nutrient (control) plots from the second experiment and add onto this the same treatments, but with herbivore fencing around them to exclude medium and large-sized herbivores. Fertilizer is pretty cheap, and building fences is a lot of work, but just a one-time expense.

2. How hard was it to organize NutNet and get it up and running? As someone who struggles as soon as he tries to collaborate with more than, like, two people at a time, or run a lab with more than, like, 5 people in it, the soft skills involved in organizing and running NutNet seem like superpowers to me!  

NutNet started with just a few of us – working at sites in California, Oregon, Minnesota, Kansas, and North Carolina – and one site in Germany because a working group member was from there and interested in contributing. We all had experience and knowledge of plant ecology, so it was relatively easy to agree on the treatments, the data we’d collect, and how we’d sample our plots. Two of us had just gotten faculty jobs at the same university, so we offered to coordinate and store all of the data as a shared side project.

After that, we agreed we’d write up a letter describing our vision for this project and send it to other scientists we knew. This is when things got exciting and a little crazy – the letter got forwarded on to others… and forwarded from there… We had unknowingly struck a chord for our science peers – many were ready to join in this side project. By the time the data started rolling in after the first sampling year, there were more than 25 scientists, many of whom I’d never met, sending data from 5 different countries. So, yes. This was a moment when I wondered what we had done, especially since I was an assistant professor with a new job, new baby, new grant to do something entirely different from this, and new classes to teach.

In the decade and a half since then, NutNet has grown steadily each year, and is now being replicated at about 140 sites in 26 countries. We’ve spent a lot of time encountering challenges with this scale of collaboration and thinking about the many aspects of how to effectively collaborate so that everyone wins. We’re empiricists, though: still trying new things and learning about how to be inclusive and supportive and do large-scale collaboration well. So, yes, I guess it benefits from soft skills; I’d say that most solutions arise from listening, empathy, and creativity.

3. If you did a meta-analysis of the NutNet experimental data, would the among-site heterogeneity be much lower than if you did a meta-analysis of independently-conducted experiments addressing similar questions in grasslands? 

Great idea! We now have data to answer this – datasets like the original meta-analysis database we assembled (and others) and the NutNet dataset. Do you want to do the analyses to answer this question, Jeremy?

4. Something I find very interesting and powerful about NutNet is that it’s a mix of “top-down” and “bottom-up” science. The core NutNet experiment seems like very top-down, centrally-coordinated science. If you sign up for NutNet, you have to run that experiment, using exactly the same protocols every other NutNet site uses. But there’s also a strong bottom-up element, where individual investigators can piggy-back their own creative studies on NutNet. And the top-down element actually facilitates the bottom-up stuff. The centrally-coordinated experiment provides a scaffolding that makes possible a lot of creative studies by individual investigators that might not be possible otherwise. Is that synergy of top-down and bottom-up science something you and the other founders foresaw from the get-go? 

I’d like to say we were visionary, but I think the reality is a little different. When we were designing the project, we thought about what would make an experiment attractive to us. First, it had to overcome past limitations by having standardized treatments and measurements. After that, it had to be cheap and easy (remember the new job, new baby, etc?). All of us were early career researchers, so needed to have ground rules for data use and paper co-authorship. But we also realized that our nutrient treatments (rates, elements we chose) might not be ideal for some sites. We thought that we might want to do something new (rainout shelters? silica addition? disturbance? insecticide?) at some future point. We also wanted to be able to do work with graduate or undergraduate students in the plots, but not compromise the cross-site experiment. So, we designed every experimental plot to have 4 subplots: the ‘core’ where only the cross-site sampling takes place; a ‘site’ subplot where we could work with students on additional projects; and two ‘future’ subplots where we could do a more destructive sampling (like many or deep soil cores) or add a new factor across the whole network of sites at some future point in time.

So, the reality is the design reflects our thoughts about what we wanted to be able to get out of the effort. It turns out that our needs, interests, and motivations are a lot like others’, so this design has worked well. Honestly, whenever we think about any new sampling or change, we first check in on whether it would compromise having identical treatments or sampling across sites, and this has allowed us to maintain the experimental integrity as a scaffolding for answering lots of questions we didn’t originally foresee.

5. A follow-up to the previous question: does NutNet provide any generalizable lessons for how to design “top-down” research so that it also facilitates bottom-up creativity? Because not every top-down research program facilitates, or even allows for, much bottom-up creativity from individual investigators. Think of the Manhattan project, for instance, or the Large Hadron Collider. 

There are probably a lot of parts to this that include the people, the network culture, and the project design, but I think an important part of this recipe is the insistence on identical treatments across sites – this is what makes this project an improvement on meta-analysis – while also making space for people to pursue site-level projects that can involve measuring new response variables. The treatments, themselves, are also general and sit at the intersection of basic and applied questions. As a group of scientists, we are excited for new ideas and leadership on projects and papers by grad students and postdocs; we’ve worked to foster a research environment that is supportive, egalitarian, and welcoming.

6. Another follow-up on top-down vs. bottom-up science: there’s an argument to be made that ecology as a field needs more top-down, centrally-coordinated work, and less bottom-up work led by individual investigators doing their own thing. If too many people go out and conduct their own small-sample studies on their own questions in their own noisy, variable study systems, at the end of the day nobody’s really learned much of anything with any confidence. Better to put all our eggs into a few centrally-coordinated, high-powered baskets. Get a few good answers to a few questions, rather than weak answers to many questions. What do you think of that argument? 

I disagree. We have a lot of tools in our toolbox as ecologists, and I think that distributed experiments are just one of these. Others include mathematical modeling, distributed observations, single-site experiments, and meta-analysis. Meta-analysis, for example, can help us generate hypotheses and identify gaps in data and knowledge. Coordinated, large-scale observations can ask whether we find patterns predicted from local work, meta-analysis, or theory. Single-site experiments can test theory and potentially provide a demonstration that the biological responses to environmental change are at least concordant with outcomes predicted by theory. Coordinated, distributed experiments can ask about the generality of these outcomes – determining whether the responses are contingent on site conditions. None of these tools gives us perfect knowledge or ability to predict the future, but each can provide important insights into how the world works.

7. A little while back we did a reader poll on Dynamic Ecology about different approaches to “generality” in ecology (https://dynamicecology.wordpress.com/2019/11/04/poll-results-the-many-ways-ecologists-seek-generality/). Distributed experiments, meta-analysis, simple theoretical models, and others. Distributed experiments really stood out in the poll as an approach that many respondents think is important to pursue, but that very few respondents actually use themselves. Why is that? Why don’t more ecologists participate in distributed experiments, given that everybody thinks they’re great? 

Everyone I know is part of one. 😉

Ok, for a more serious answer, it takes some commitment. Not everyone is as crazy or stupid or naïve as we were when we started NutNet, so while there are several groups using our NutNet model as an inspiration, there isn’t a distributed experiment for every question. And, even as a participant, this scale of collaboration is not for everyone. The challenges to this kind of approach would differ by the ecosystem, treatments, and response variables, too – but funding for this kind of project can be a challenge. I also think that the culture of ecology plays a role, but this is probably big enough for a whole other blog post. (Jeremy adds: I’ll just leave this here. 🙂 )

8. Are there any ecological questions that you think are crying out to be addressed with distributed experiments? Or more broadly, are there certain kinds of questions that are good candidates to be addressed with distributed experiments? For instance, “it has to be a question you can address with an inexpensive experiment”.

My answer to this relates to my previous answer. I do think that costs are one important consideration. Early career investigators and those in countries where science is not as well funded cannot be involved in work that is too expensive. While cost determines who can be involved, which is an important consideration, it is somewhat separate from which questions are appropriate for distributed experimentation. Being able to standardize treatments is another important consideration. For example, we discovered that the company we used for our micronutrient fertilizer mix began marketing a different mix of elemental nutrients on different continents under the same product name. From their perspective, farmers needed a different mix to amend the soils, but from ours, this would have led to a confounding of treatments by continent. Before we had collaborators in this experiment from many continents, checking the percentages of each element in a product by location would never have occurred to us. Maybe kind of obvious, but experiments are strongest if they are hypothesis driven, and distributed experiments provide the additional layer of uncovering the site-level conditions under which they are supported. If this kind of answer isn’t exciting for a particular question, then this approach may not be worth the additional effort. Finally, because the inference from a distributed experiment is strongest if everyone implements it exactly the same way, working within a single ecosystem – with organisms of similar size, similar abiotic constraints – is a consideration, but probably not a deal-breaker. If someone wants to start one, I’d be happy to chat about the specifics.

9. How difficult has it been to maintain NutNet funding over the years? Do you think it will become more or less difficult, the longer NutNet runs?

Ah, funding. We originally conceived the experiment, itself, to be low cost and low time investment by any single investigator. This means that we’ve been able to ask every investigator to self-fund the work at their site, and that has not proven prohibitive to many. The expenses have come in two forms: expensive response variables and data management.

None of the core (plant, light) response variables are expensive to collect, but for expensive response variables, we have taken two approaches. First, two of us have spent our faculty startup funds on generating data on soil texture and chemistry using standardized methods, so that we can have this key variable for all network analyses. While we sought US NSF funding for this on a few different occasions, we never received this (in one case with a review along the lines of, “I thought NutNet was great because it was basically free, so you shouldn’t need money to do any of this work.”) More recently, we received a moderate amount of funding from USDA for soils processing, but as our funds have run down, we have asked site PIs to pay for the analyses, if they can. For ‘non-core’ response variables, the scientists leading the study pay. For example, two different groups have generated data on soil and plant metagenomics across the network. Other investigators have generated data on plant chemistry and foliar traits. 

The biggest expense, though, is the coordination and data management, with a somewhat smaller expense of hosting annual meetings. We were lucky to get a US NSF Research Coordination Network grant early on that allowed us to hire a postdoc to help manage data and people and provided funds to help with bringing people together for an annual working meeting. We have kept this position funded through various grants and opportunities since then (though I’m still hoping someday a foundation will invest in this!), and we have combined resources (funds, finding free space, etc) to hold annual meetings. The data manager/research position and the meetings are the heart of the network and keeping these funded has been a focus (and worry) of mine for years. And, to answer your last question, I have not found it getting easier with time.

10. Knowing what you know now, is there anything about NutNet you wish you could go back in time and change?

While we’re in pretty good shape now, it would have been great if we’d started at the outset by developing a relational database for storing and querying network data. While it took quite a bit of effort and time to switch over from storing our data in flat files – we specifically hired a really talented postdoc to create this and transition the network over – this was absolutely the right investment for the long-term.

11. How long will NutNet run? Would you like to find some way for it to run forever, much as Rich Lenski wants the Long-Term Evolution Experiment to run forever

We originally decided that if we (remember the 6 of us?) managed to do this for a decade, we’d decide at that point whether to continue or be done. In years 7-9, the network had grown in ways we never anticipated, so I did a few-year process of asking (a) whether people thought we’d answered all the questions we could or wanted to continue with the experiment and (b) if they wanted to start a new treatment. The answer to the first was a resounding vote to continue because there were so many more questions that had not yet been answered. So, it looks like we’ll keep NutNet going for another 10 years, at this point.

The answer to the second was more complicated. After a lot of listening, discussion, and hard experimental design work by a few, we eventually settled on a new experiment looking at disturbance and nutrients, Disturbance and Resources Across global Grasslands, or DRAGNet (https://nutnet.org/dragnet). This new experiment, involving fertilizer and rototilling, is designed to test hypotheses about community and ecosystem responses to disturbance and disturbance cessation.

12. What advice would you give to someone looking to start their own distributed experiment?

Read this blog post and give me a call if you want additional advice.

3 thoughts on “The story and lessons of the NutNet experiment: an interview with Elizabeth Borer

  1. Re: that review saying “I thought NutNet was great because it was basically free, so you shouldn’t need money to do any of this work.”, that’s the trouble with being low-cost or efficient. Funders think “low-cost” and “efficient” are synonymous with “free”. So they “reward” you for your efficiency by not giving you any money!

  2. Really great to hear the inside story. I’ve been part of a lot of working groups but they (obviously) never went beyond metaanalysis to experiment. Its really interesting to see parallels and differences. Everybody talks about how “big data” is changing science, but I think at least as big a change is “big science” (large teams). There’s not nearly enough discussion, training, etc on this change in our lives.

  3. On how ‘big science=large teams” (sensu Brian above) will change scientific result acquisition, testing and decision making, I speculate that the COVID19 situation may give rise to interesting insights in global terms.
    In recent and less recent human history, I can not think of anything similar to the COVID19 situation in terms of knowledge exchange, data sharing and trust building in such a narrow time period. Even if this takes place spontaneously or due to an emergency sitiation and even though it occasionally creates conflicts, we still have so many things to marvel at this global human transboundary effort.

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.