Charley Krebs suggests we call a moratorium on microcosm studies in ecology, because their results don’t generalize to nature. He refers to this as “Volkswagon Syndrome”, claiming that, like Volkswagon cars, microcosms don’t perform they way their real world versions do.
I have huge respect for Charley, but he’s way off base on this. He’s making a common mistake: assuming that the purpose of all microcosm studies is to reproduce or predict the behavior of some particular natural system, or ‘nature’ in general. And implicitly, he’s making the common mistake that that’s the only possible useful purpose of microcosm studies. In fact, microcosm experiments have various useful purposes (just like experiments in general have various useful purposes). Charley’s objection to microcosms only applies to certain microcosm experiments, conducted for certain purposes (e.g., to estimate the value of some rate parameter in some specific natural system), and then only if the experiment in question is in fact insufficiently realistic.
Further, Charley appears to have overlooked cases in which the results of microcosm studies do generalize to nature. For instance, Fox (2007 Oikos) shows that the average strength of trophic cascades in protist microcosms is almost scarily similar to their average strength in field experiments. And Smith (2005 PNAS) shows that microcosm and mesocosm phytoplankton communities fall on exactly the same species-area curve as natural phytoplankton communities.
In passing, Charley notes that ecologists can’t wait around for another century’s worth of data to test many predictions of interest. Which is a puzzling point to raise in this context. One motivation for microcosm and mesocosm studies of small organisms with short generation times is to collect many generations of long-term data in a reasonable human time frame. Better some long-term data from studies of organisms with short generation times than no long-term data at all, surely?
Here’s my old post on objections to microcosms in ecology, and their answers, which anticipates Charley’s objection (and others). Many of the points I make in that post have been made in older peer-reviewed literature as well (e.g., Lawton 1996 Ecology). More broadly, here’s an old post of mine arguing for the value of model systems in ecology. I’d welcome the opportunity to discuss these issues with Charley in the comments, as they’re near and dear to my heart, and because comments on his blog seem to be closed now. I’ve enjoyed our previous exchanges on related issues.
More broadly, and at the risk of reading into Charley’s post something that’s not there, it seems like Charley’s objection to microcosms springs from a concern with prediction in ecology in general. His view seems to be that predicting the behavior of natural systems is so overwhelmingly important that anything that doesn’t lead directly to successful predictions of the behavior of natural systems should be discarded. Without wanting to deny that predicting the behavior of natural systems is hugely important, and that ecologists often lack what Brian’s called a “problem solving mentality”, I’d suggest that there’s more to doing good ecology than predicting the behavior of natural systems. For instance, I want understanding and explanation as well as prediction (e.g., this old post).
Further, there are many ways in which “unrealistic” models and experiments can help us understand and, yes, predict what happens in more realistic cases. False models are useful because they’re false. Unfortunately, a single-minded emphasis on predicting the behavior of natural systems shades easily into discarding lots of useful tools that help achieve that goal, but in indirect or non-obvious ways. Charley seems to have an unduly narrow view of where predictions might come from in ecology, and doesn’t note how one can combine different research approaches in order to make predictions. Taken to an extreme, a single-minded emphasis on predicting the behavior of nature leads to Robert Peters-style instrumentalism, an unsuccessful approach ecologists shouldn’t emulate. I’d rather see ecologists emulate Britt Koskella, who has a great guest post on how she combines microcosm experiments with theory, field experiments, and field observations to really nail how her study system works. See also Meg’s and Brian’s posts on the power of combining diverse research approaches in ecology, where “diverse” includes microcosm experiments. And see Meg’s post on how she won the Mercer Award by powerfully combining microcosm experiments with other approaches. Most Mercer Award winners combine different research approaches. Ruling out any approach–microcosms, field experiments, field observations, theoretical models of various sorts, etc.–seems like a bad idea. It’s like obliging someone to fight with one hand tied behind their back. But it’s possible that I’m misreading Charley’s views on prediction.
As always, looking forward to your comments.
Dynamic Ecology 
I agree with you, Jeremy, and I’d add that it’s odd to see someone rejecting simplified microcosm experiments without simultaneously rejecting simplified theoretical models. I see one purpose (among others) for microcosm experiments as a sort of biological computing, analogous to a simple model like Lotka-Volterra, where we strip away all kinds of complications to check our understanding of a fundamental underlying mechanism. If microcosm experiments are too simple to teach us anything about nature, then surely so are virtually all of the mathematical models underlying ecology and evolution!
“without simultaneously rejecting simplified theoretical models.”
Well, as I read him, he’s rejecting simplified theoretical models too. At least the ones that don’t reproduce the behavior of some natural system or other.
Yes, I guess that’s what he means at the beginning of his second paragraph, but he brushes by it rather quickly. I guess if one is going to toss out big chunks of our research methodology, one might as well go big?
“Yes, I guess that’s what he means at the beginning of his second paragraph, but he brushes by it rather quickly.”
Yeah, I think the whole piece suffers from trying to cover too much ground. Charley starts out with microcosms but then moves on to theoretical models, and the importance of prediction in general, and how ecology isn’t making progress…Even someone who’s been reading him for a while and so has a sense of where he’s coming from (as I have and do) is likely to have trouble parsing this particular post. Which happens; I certainly have my share of posts where even regular readers couldn’t tell what the heck I was talking about.
Well, I’ll admit I read him too quickly the first time, and coming from your post, I was primed to be reading about microcosms. So I’m not going to blame Charley for my too-shallow read. Although I’m more gobsmacked, not less, knowing that he really IS railing against models too….
@ScientistSeesSquirrel:
Not *all* models, actually. For intance, in other pieces, Charley’s said he’s a big fan of ratio-dependent predator-prey models. Which I suspect reveals something about what he looks for in a theoretical model. My reading is that he wants theoretical models that have few parameters, that have *some* sort of mechanism or other in them, and that are capable of reproducing simple patterns in natural data (or capable of fitting data from many different systems). But on my reading, he doesn’t actually care much about whether or not the model has the *right* mechanism. Or maybe he does care, but mistakenly thinks that the ability of ratio-dependent models to fit the data that they do is strong evidence for the correctness of the assumed mechanism.
But in saying this, I freely admit I’m reading a lot into Charley’s pieces. Charley’s point of view is sufficiently different than mine that I sometimes struggle to understand where he’s coming from. I struggle to understand how anyone who cares at all about the internal consistency of their views could hold the combination of views he appears to hold. This is why I’m hoping he’ll comment, as he has in the past. It’s quite possible that some of the apparent disagreement here would be resolved if I better understood his views and his reasons for holding them.
Jeremy, bravo. I was dismayed to read the same old (moldy) chestnut arguments criticizing microcosms. Your thoughts, and the blog posts you link to, make clear the need for multiple approaches to understand nature. I think there is an additional, important point also to make. Krebs suggests that microcosms fail because they often fail to predict phenomena in natural systems BUT he also argues that people use them to establish general theories. I think this reveals a bias particular to only some sub-fields of ecology. For example, as we both know from our days at Imperial, in population ecology ‘general theories’ are often developed from microcosm work. I have no major beef with this, as long as those theories are tested using multiple approaches. However, as an ecosystem ecologist/ biogeochemist I constantly hear from more population and community focused people that my subfield has no robust theory. There is certainly some truth to this argument, but I would argue we more typically use microcosms to elucidate and test mechanisms, as opposed to develop general theory. Those mechanisms can then be incorporated in models and model projections tested against field data. Although the emphasis right now seems to be on getting predictions correct, I think the real power of this approach is improving confidence in projections (even if uncertainty in the span of projections remains high) because one is more confident that the models are representing mechanisms we believe to be important. The basis of model inter-comparisons in the physical sciences is often not on reducing uncertainty in the spread of projections but on improving confidence that our models capture the relevant mechanisms and so that the true answer probably lies within the projected uncertainty (as opposed to far outside it). I think ecologists need to face up to the fact that in a very complex world, a near-term emphasis on prediction certainty is misplaced until we can be confident that the increasing certainty we have arises from actually understanding mechanisms. That may mean that prediction uncertainty remains large, whereas confidence in that span of predictions increases markedly. That is, we get it right because we understand the system. Here, microcosms make a huge contribution. Regards, Mark.
Good comments Mark. Yeah, I confess I too was disappointed to see Charley just give a new name to an argument that’s been refuted many times. He’s of course entitled to his views, but I was hoping to see a guy as widely read as Charley defend his views by engaging with those refutations. Offer some counter-counter-arguments.
It’s not just Charley. I’ve never seen *anyone* opposed to microcosms engage in a serious way with arguments like yours, or mine, or Britt’s, or Lawton’s. Closest I’ve seen were some comments from Jeff Houlahan on that old post of mine, and even he began by conceding I had a point. I’d love to see somebody defend blanket opposition to microcosms as an approach to go through my post point by point and refute it with counterarguments. Without misreading it, skipping any points, or just repeating arguments that I refuted (unless it’s to say “we’ll have to agree to disagree on this point.”). Come on, microcosm opponents–I took your arguments seriously and made counter-arguments. Counter my counter-arguments!
“The basis of model inter-comparisons in the physical sciences is often not on reducing uncertainty in the spread of projections but on improving confidence that our models capture the relevant mechanisms and so that the true answer probably lies within the projected uncertainty (as opposed to far outside it). I think ecologists need to face up to the fact that in a very complex world, a near-term emphasis on prediction certainty is misplaced until we can be confident that the increasing certainty we have arises from actually understanding mechanisms. ”
Well said ! I have a background surface geochemistry and contaminant transport and I can’t honestly understand why microcosms or models should be controversial. It seems that we should be using as many complementary tools as we can to get a handle on the complex mechanisms involved and avoid getting stuck in the century old dichotomies that have plagued ecology since the beginning (arguments about field vs lab, models vs. experiments, qualitative vs quantitative etc.).
Mark, I’m not sure I understand exactly the point you are making – are you saying that we shouldn’t worry if a model makes very bad predictions as long as we are sure that we have identified the true causal mechanisms and have them in the model? Jeff
Hi Jeff, I think your query is important and relates to what qualifies a prediction as “very bad”. For me, “very bad” is when a prediction is based on a very uncertain mechanistic understanding. As we gain confidence that we have the mechanisms correct, then the prediction becomes much better (at least in my book). This is ‘true’ even if we do not decrease the uncertainty bounds of the prediction, because we’re more confident the true answer lies within those bounds. Hence, microcosms are a tool to help us elucidate mechanisms that we can use in building predictive models of natural phenomena, which we can test against real world observations and then circle back to refine mechanisms, again potentially using microcosms as a powerful tool to help do this. If we get to the example that you raise – a model with true causal mechanisms that gives bad predictions – I think we’d be a in a great position and concerned with the more minor details of improving parameter estimates. But I’m a little skeptical that in ecology we’ll ever get a model with all the true causal mechanisms…
I too found Krebs’ comments to be short-sighted. I see microcosms and mesocosms as part of a continuum which spans from our very simple theoretical models to the most complex of natural ecosystems. While it is true that we applaud those whose work finds generalities that span across these divides, it is also true that our failures to generalize can lead to great insight (often with less applause…). In my experience as a theoretician, the challenge is not to build a model that captures the reality of an experiment or natural dataset, but to figure out which elements of the model are essential and which are cursory to capturing that reality – and of course, it is never quite so simple as to independently accept or reject each element. Echoing Mark Bradford’s comment above, I believe microcosms are paramount to helping us understand which ecological processes transcend the different degrees of ‘natural’ in our ecosystems. Without this it is hard to see how each and every natural ecosystem is anything but a self-contained case study.
Agree with all of this Dave, but wanted to highlight this bit:
“Without this it is hard to see how each and every natural ecosystem is anything but a self-contained case study.”
Spot on. As I said in my old post defending microcosms, if you think that microcosms should behave like nature, well, *nature* doesn’t behave like nature! That is, any given natural system behaves differently in many respects from any other given natural system, and the behavior of any given natural system changes a lot over time. So “nature” is not any one state of affairs, it’s a huge range of states of affairs. Which isn’t to say there aren’t any generalizations, of course; natural systems and microcosms also behave *like* one another in various *other* respects. Again, Smith 2005 is one good example.
Charley Krebs and others who think as he does can’t possibly be unaware of the variance among natural systems, of course–Charley actually brings it up later in his post. Which only makes his post even more puzzling.
David, the struggle I have is – how do you decide which elements of the model are essential and which are cursory without comparing it to experimental or observational data? Jeff
@Jeff:
Why do you think that Dave thinks we can decide that without looking at data? Is that what you think his phrase “as a theoretician” meant?
Plus, there *are* some ways in which one can tell which assumptions of a model matter for which purposes without looking at data. One can for instance look at whether different models making different assumptions make the same predictions? For instance, just to pick the first example I thought of, Dave Tilman has an old (1990?) book chapter in which he takes a very simple model of resource competition from his 1982 book and adds in various complications to make it more realistic and specific to terrestrial plants. And shows that all of the complications he introduces don’t eliminate the R* rule–the competitor with the lowest R* for the shared limiting resource still competitively excludes all the others at equilibrium.
But all this seems pretty obvious to me, which makes me surprised you’re raising it and makes me suspect I’m misunderstanding you, Jeff. Can you elaborate?
Jeremy, I thought that because Dave wrote
“the challenge is not to build a model that captures the reality of an experiment or natural dataset, but to figure out which elements of the model are essential and which are cursory to capturing that reality”
Maybe I’ve misinterpreted the point Dave was making but I read that to mean that you could figure out what was essential and what wasn’t without worrying about well the model fit the data.
And as for telling which assumptions matter without resorting to the data – I don’t think you misunderstand me – I just think that you value a contribution like showing that a prediction is robust to varying model assumptions more than I do. I think there is some value to identifying that predicted outcomes are robust to varying assumptions but if despite that, the predictions the models make don’t describe what happens in nature we’re still falling pretty far short of what our goal is – to explain how nature works. It may help guide us towards the correct model but it is telling us little about how the world works.
Jeff
@Jeff:
“Maybe I’ve misinterpreted the point Dave was making but I read that to mean that you could figure out what was essential and what wasn’t without worrying about well the model fit the data. ”
Without wanting to put words in Dave’s mouth, I think what he meant was that making a model that fits the data is easy. Making a model that fits the data for the right reasons–in particular identifying the “core” assumptions that are most crucial for fitting the data–is harder.
Jeremy, I’m left with the same question –
” Making a model that fits the data for the right reasons–in particular identifying the “core” assumptions that are most crucial for fitting the data–is harder.
Can you do this without using data as the ultimate arbiter? I’m guessing you can because you’ve suggested it as an alternative interpretation of Dave’s point but it’s just not clear to me how you would do that.
I confess I’m pleased with the positive reaction to this post so far on Twitter, including from senior ecologists. For instance, here’s Shahid Naeem & co:
And the tweet of this post, or RTs thereof, have been “liked” by Don Strong and Chris Klausmeier, among others, which I assume constitutes endorsement.
I would never invoke proof by authority. Maybe it’s me and everyone who agrees with me who is way off base! Wouldn’t be the first time. But if you think so, you’ve got to make the argument. Disappointed to not yet see Charley or anyone else come forward to do so yet, but it’s early so I remain hopeful.
Am now trying to anticipate how one would make the argument. What’s the strongest argument for abandonment of microcosms as an approach, that addresses my counterarguments while conceding as little to them as possible? Here are a couple of opening bids:
-Argue that the overwhelmingly most important goal for ecology is to predict and manage specific natural systems. Generalizations, ecological “laws”, mechanistic insight–all of that stuff is nice, but it’s an optional extra. What’s really important is for us is to be able to tell management agency X *exactly* how to achieve management goal Y on *particular* bit of land or water Z. Argue that microcosm studies contribute little to this goal, and only contribute that little if said microcosms are designed to realistically mimic the behavior of particular bit of land or water Z. In other words, concede that microcosm studies can contribute to achieving various goals, but argue that most of those goals are comparatively unimportant. (EDIT: this argument basically responds to Dave Vasseur’s comment above by saying “yes, each system is indeed a unique special case, at least for overwhelmingly-important management purposes”)
-Concede that microcosm studies done well can help us predict and understand nature, but argue that people who use microcosms well are rare exceptions. Ecology as a field can’t rely on everybody being as sophisticated in their inferences from microcosm data, and in the way in which they combine microcosms with other lines of evidence, as Peter Morin, Meg Duffy, Shahid Naeem, Britt Koskella, etc. Support your claim by citing a few examples of people overextrapolating from microcosm studies or ignoring conclusion-altering artifacts created by enclosing some bit of nature in a walled container. Argue that, for ecology to advance, it needs to be based on simple, “off-the-shelf”, “crank the handle”-type research approaches that anyone can apply fairly mindlessly in his or her own system. Top people doing sophisticated, difficult-to-emulate research are worthy of praise, but don’t actually advance the field that much because their sophisticated, “custom designed” research strategies don’t scale.
-EDIT: here’s a third: argue that, since ecologists still have very little ability to generalize or make predictions about nature, every single approach or combination of approaches that we’ve ever taken must be inadequate to the task. We shouldn’t just abandon microcosms, we should abandon all of ecology as currently practiced and start from scratch. Or maybe admit defeat and stop doing ecology entirely. This is basically Lawton’s famous “community ecology is just a stamp collection of special cases, so let’s abandon it” argument applied to all of ecology and every approach to it.
I of course disagree with these arguments. But they at least have the virtues of internal consistency and plausibility, I think. Neither is *obviously* logically flawed, or *obviously* contrary to empirical evidence, or based on *obviously* bad premises.
To be clear, I’m not suggesting that Charley Krebs, or any other microcosm opponent, would make these or other imaginary arguments (and they are imaginary–I’ve never heard anyone make them). But since I’ve never seen any *actual* counterarguments to the points I’ve made in this and previous posts, I’m forced to imagine my own. And it seems most useful to imagine the strongest ones possible. If nobody else is going to push me in a serious way, I’ll just have to push myself.
A related question: what’s the strongest *non-blanket* criticism of microcosms in ecology? By which I mean, what’s the most common or important *real* problem with *some* microcosm studies in ecology? After all, it’s not as if all microcosm studies are great! And as with any research approach, whatever flaws that do crop up from time to time will tend to run to type (“failure modes”).
Personally, I’d give two answers to this question:
-there are some microcosm studies (“some” meaning “a minority”) that come too close to being “rigged”, meaning that the design of the experiment is such that there’s basically only one possible outcome which is obvious in advance. Failure to observe that outcome wouldn’t be an interesting surprise, it would mean you screwed something up and didn’t actually impose the treatments you intended or something. This problem actually comes up more in evolution than ecology, actually. Rich Lenski refers to such experiments as being like drawing colored marbles out of urns–you already know from sampling theory what’s going to happen, and if you don’t get that it means you didn’t sample randomly.
-some microcosm studies don’t take full advantage of the data that microcosms can generate. For instance, only running an experiment for a couple of dozen generations when you could easily run it for hundreds. Or only analyzing time averages of one’s data, thereby missing the opportunity to address additional questions, and/or better address one’s chosen question, by not averaging away lots of the information in the population dynamics.
And above, I see that Jeff Houlahan has now given a version of the argument I added in my edit…
Jeremy, your edit doesn’t completely capture my position. I certainly wouldn’t say give up. I also wouldn’t say abandon how we do ecology and start from scratch. What I would say is that prediction is the only way to demonstrate/measure understanding and so we should put prediction at the center of ecology. We haven’t done that. And to clarify – I’m not saying ‘general predictions’. They can be very local – understanding can be local or general. And I’m not saying that assessing understanding using prediction isn’t problematic – it’s just that if it is the only way then we have to deal with the problems.
So, if I was to make one prescription it would be – any claim to increased understanding of how the world works should be measured by its contribution to better predictions. Where microcosms allow us to make better predictions of the natural world (and as you’ve pointed out, there are several examples) I couldn’t be more admiring. Jeff
Oh, and insofar as his real concern isn’t so much with microcosms per se as with coming up with generalizations from anywhere we can get them: don’t you want microcosms as one tool in your toolbox precisely *because* they differ from natural systems in various ways? That is, don’t you *want* to study the greatest possible range of systems, so as to be able to construct better generalizations, and so as to better understanding the limits of those generalizations? Analogous to how, if you’re trying to estimate the true regression relationship between a dependent variable and an independent variable, you want as wide a range of variation in the independent variable as you can get.
But now I’m starting to repeat stuff I said in that old microcosm post here in the comments, so I should probably just stop commenting until I get something to reply to.
Jeremy, while I disagree with Charley’s assumption that simple models or microcosms can’t tell us anything about how nature works I also get a little frustrated with the position that ‘all approaches are equally important and valuable’. It always feels like an attempt to avoid offending. The fact is, we are all trying to figure out how nature works and my sense is that ecologists are doing a relatively bad job of this. That may be because, as Steve Heard has suggested elsewhere, ecology is hard. But it’s also possible we’re not doing our jobs very well. I think there is a need to step back and identify approaches that have been most productive for increasing our understanding of how the world works – any approach that isn’t ultimately aimed at describing how the natural world works and testing that description on independent data (i.e. data that weren’t used to develop the description) is unlikely to move ecology forward. My biggest problem is that I’m not sure what approaches have worked best – what I’m pretty sure is that some approaches are better than others. Charley seems convinced that simple models and mesocosm experiments haven’t helped much – I haven’t seen the evidence for that opinion. There certainly isn’t anything inherently wrong with mesocosm experiments and there may be a lot right – it seems like they are potentially a useful way to examine ‘large-scale’ population and community dynamics in a controlled manipulative way. But, the proof of the pudding is in the eating – for any approach to ecological research the key question is “How has that approach helped us to understand how the world works?” This sounds like an empirical question.
Best, Jeff Houlahan
*
“‘all approaches are equally important and valuable’.”
I never said that. But I also doubt something you seem to implicitly assume: that there is a global answer to the question “which approaches are most valuable”. I think that question only has a local answer. It depends on your question, system, and goals. I certainly don’t think that any and all approaches, alone or in combination, are equally good ways to address any given question. Don’t think I’ve suddenly gone all non-prescriptive! I just think that blanket, global attempts at prescribing how to do ecology are mostly doomed to fail.
Jeremy, while I agree that there are lots of ways to do effective science I also think most scientists have a common objective – to better understand how the natural world works. And regardless of your question or system or whether you are looking for local or global understanding there is only one way to demonstrate understanding of how the world works – that’s with prediction. So, I think the hallmark of all effective science is that it ultimately results in good predictions of how the natural world works. And that the value of an approach can be measured by how much it contributes to increased understanding of the natural world (as measured by improved predictions).
“any approach that isn’t ultimately aimed at describing how the natural world works and testing that description on independent data (i.e. data that weren’t used to develop the description) is unlikely to move ecology forward.”
Jeff, I’d hope you’d recognize that I’ve already answered your empirical question. Go look at Britt Koskella’s post and associated papers. Go look at Meg’s Mercer Award winner. Go look at Smith 2005. Do I need to keep going? How many examples do I need to give to establish in your mind, beyond question, that microcosms can help us understand how nature works? Why are you still using weasel words like “potentially”?
When it comes to microcosms, your empirical question already has an affirmative answer.
Jeremy, I didn’t use ‘potentially’ to imply that the jury was still out on whether microcosms ever could help us understand how nature works – I used it to suggest that for some questions it could and for others it may not be able to. And the empirical question is not whether microcosms can help us understand nature or not, it’s how they fare relative to other approaches. And let me emphasize – I have no idea what the answer to that question is. Jeff
“And the empirical question is not whether microcosms can help us understand nature or not, it’s how they fare relative to other approaches.”
Since different approaches often are complementary to one another rather than substitutes, I don’t think that’s the right question to ask. I do think there are slightly different questions that are more productive to ask. More on this in future posts soon, hopefully.
Pingback: On Ecological Rants and Microcosms | Jabberwocky Ecology | The Weecology Blog
Pingback: Why functional trait ecology needs population ecology | Dynamic Ecology