The uncanny valley of theoretical models and microcosm experiments

In robotics, “uncanny valley” refers to the fact that robots that look and act somewhat human seem creepy. It’s better for them to either look and act not at all like humans, or to look and act exactly like humans. The latter hasn’t yet been achieved as far as I know.

In ecology, theoretical models and microcosm experiments also have an uncanny valley, when they are designed to mimic a particular natural system only in one or two particular respects. In all other respects they are unlike that particular natural system and not intended to mimic it.

I don’t want to cite examples since I don’t want to single anyone out or pick on anyone, so here’s a made-up example that should give you a “search image”. Imagine a protist microcosm experiment purporting to test the effects of phosphorus enrichment on species diversity in lakes. The size and shape of the protist microcosms are chosen so as to mimic the frequency distribution of surface area-to-volume ratios of N. American lakes, and the range of total phosphorus (TP) concentrations used is chosen so as to match the range of TP concentration in N. American lakes. In all other respects, no effort is made to make the microcosms anything like real N. American lakes. The microcosms are kept in a constant environment in the lab, the growth medium is an artificial medium, they only contain algae, bacteria, and bacterivorous protists that were purchased from a biological supply company, etc. But the experiment is described as revealing the effects of enrichment on species diversity in lakes, specifically. As opposed to (or perhaps in addition to) testing some general theoretical hypothesis about enrichment and diversity that might apply in an approximate way to many different systems.

One sometimes sees the same thing in theoretical models. Made up example: a spatially-explicit metacommunity model in which the patch sizes and connectivity are chosen so as to match a particular natural metacommunity. But the species dynamics within patches are described by a Lotka-Volterra competition model of some arbitrarily-chosen number of species, with arbitrary parameter values. And yet the model is discussed as if it explains the observed features of that particular natural metacommunity.

Which seems…odd. At least to me. Those sorts of models and experiments fall in “uncanny valley” for me. Maybe it’s just me, but don’t understand why it’s useful for a model/experiment that’s unlike any particular natural system in most respects to be matched to a particular natural system in one particular respect. Especially since that one respect usually seems to be chosen arbitrarily. In my admittedly-anecdotal experience, there’s never any case made that the one respect in which the model/experiment matches the natural system is the overwhelmingly important one. As in, if you get that bit of the design right, nothing else matters, because that one bit is going to completely dominate the behavior of the model/experiment. I rarely see anyone test whether their modeling or experimental results are sensitive to the realism of the one particular aspect of their model/experiment that’s supposed to be realistic, by including some unrealistic “controls”. And I rarely see anyone test whether their modeling or experimental results are sensitive to the lack of realism of other aspects of their model/experiment.

To be clear, I see an important place for models and experiments that mimic particular natural systems! I’m thinking for instance of the cattle tank experiments on anuran assemblages that folks like my former advisor Peter Morin used to run (e.g., the Mercer Award-winning Morin 1983). Those cattle tanks aren’t exactly like natural ponds in every respect. But they’re enough like natural ponds in enough respects that their results absolutely do give a lot of valid information about the community ecology of anurans in natural ponds. As a second example, think of Dolph Schluter’s (1994) classic experiment using artificial ponds to show that resource competition selects for character displacement in threespine stickleback. Those artificial ponds were, and were intended to be, reasonably representative of the natural ponds first colonized by ancestral threespine stickleback ~10,000 years ago. As a third example, when Meghan assays the virulence of different parasite genotypes in her lab, I’m sure those assays provide valid indices of the virulence of those particular genotypes in nature, even though the lab environment is unlike the natural environment in various respects. As a fourth example, check out this excellent old guest post from Britt Koskella. To return to the uncanny valley analogy, those sorts of studies are like a robot that looks and acts sufficiently human not to creep people out.

Conversely, as I’ve discussed before and won’t repeat here, I see an important place for theoretical models and experiments that test general principles, but that make no attempt to mimic any particular natural system, although they share some features with many natural systems (e.g., protist microcosms exhibit trophic cascades of a strength typical of trophic cascades in nature). Those studies are like robots that make no attempt to look and act human, although they can do some of the things that humans do.* But a model or experiment that carefully matches some particular natural system in one particular respect while not bothering to match that system in any other respect falls between two stools into uncanny valley.**

To be clear, if some feature of your model or experiment has to be chosen arbitrarily (as is often the case for models and experiments that discover and test general principles), the arbitrary choice to mimic some particular natural system isn’t necessarily any worse than any other arbitrary choice. But it’s not any better, either. And it might be worse if it fools you into thinking your experiment is more “realistic” than it really is.

I recognize that there are times when one wants to mimic a natural system in certain particular respects but not others. Schluter (1994) provides an example. In that experiment, Schluter used interspecific hybridization to create a focal population of fish harboring an unnaturally wide range of genetic and phenotypic variation for selection to act on, so as to maximize his chances of detecting selection. Which is fine! Sometimes to get information about how natural systems work, you have to create unnatural conditions.*** But in the cases I’m thinking of, the many mismatches between the model/experiment and the particular natural system of interest aren’t deliberately chosen so as to increase statistical power or test a scientific hypothesis.

I also recognize that it can be a debatable empirical issue whether a model or experiment that’s designed to mimic a particular natural system actually does so sufficiently well in the relevant respects that its results apply to that natural system.

My advice: if you want your model or experiment to test general principles, just go ahead and do that. It’s fine! You don’t need to give your model or experiment a thin veneer of “realism” by matching one particular aspect of it to some particular natural system. Conversely, if you want your model or experiment to give insight into the behavior of some particular natural system, mimicking that natural system in one particular respect probably won’t cut it. And if you want both, the way to do that is to design your model or experiment to match the particular natural system of interest, and then use other approaches and arguments to draw out the broader implications of your system-specific case study (that’s “approach to generality #5” in this list).

I bet this post is going to be controversial. Indeed, I bet at least a few of my friends will seriously disagree. As always, looking forward to your comments. Especially if you think I’m full of it. 🙂

*Which is why criticizing them as “unnatural” or “unrealistic” totally misses the point. That’s like criticizing Robby the Robot for not looking human.

**I tried to work a third metaphor into this sentence but couldn’t quite manage it.

***Many ecologists fail to appreciate this point, in my admittedly-anecdotal experience.

 

13 thoughts on “The uncanny valley of theoretical models and microcosm experiments

  1. Jeremy, I’m not sure … I share the feeling that researchers sometimes design experiments / models for “realism without reason”, but it’s difficult to discuss the sense of a research design without knowing what its purpose is.

    Using your protist microcosm example: if a study wants to test if a particular emergent pattern could originate from the surface area-to-volume ratios and (TP) concentrations of N. American lakes, and if other factors that are neglected should not influence this pattern, then the experiment makes total sense to me.

    I guess the general question is if things that are left out can be assumed to act more or less orthogonal to the factors you are looking at, or if you have to describe them in detail to describe / test the process you are interested in.

    I was also wondering about your preference for “simple models” over “partially realistic models”. I see no big difference between simple and partially realistic models regarding their forward utility (if I assume this, then I get …), and regarding inference (could this process be an explanation for pattern X?), wouldn’t any reservations against partially realistic models also apply to simple models? OK, you can say that simple models are more general, but if I happen to be interested in N. American lakes, would disregarding lake sizes really help me in any way?

    • “it’s difficult to discuss the sense of a research design without knowing what its purpose is.”

      I agree. But that’s what the introduction of the paper is for. It’s the author’s job to convey the purpose of the study and explain why their study design is fit for purpose. I wrote this post because when I read these papers I’m left unclear as to the purpose of making the experiment match a particular natural system in one particular respect.

      “if a study wants to test if a particular emergent pattern could originate from the surface area-to-volume ratios and (TP) concentrations of N. American lakes, and if other factors that are neglected should not influence this pattern, then the experiment makes total sense to me. ”

      As I said in the post, if you want to test the effect of realistic feature X, you need an unrealistic control. If you think (e.g.) that some naturally-observed correlation between lake SA:volume ratio and lake TP is key to explaining the response of species diversity to TP in natural lakes, you need a control that breaks that correlation. Which I can’t recall ever seeing.

      “I guess the general question is if things that are left out can be assumed to act more or less orthogonal to the factors you are looking at, or if you have to describe them in detail to describe / test the process you are interested in. ”

      Yes. I just don’t think anyone’s entitled to assume that. I don’t think you’re entitled to assume that (say) a few protists that you got from biological supply companies and grew in protozoan pellet medium in a constant environment in the lab will respond to a TP gradient the same way as the microbes, plankton, fish, insects, ambphibians, etc. that live in natural N. American lakes under natural N. American environmental conditions. Unless of course you think that some very broadly-applicable ecological principle governs the qualitative of species diversity to enrichment (the more individuals hypothesis, say). But if you think that, there’s no reason to match any aspect of your experiment to N. American lakes specifically, and nothing gained by doing so. Now, I can imagine cases where there could be reasonable disagreement over whether an experiment matches a particular natural system sufficiently well, in enough respects, to be taken as representative of nature. But for that to even be debatable, I think the experiment would have to be much more closely matched to a particular natural system than the ones I described. But perhaps we’ll have to agree to disagree on this.

      • “I agree. But that’s what the introduction of the paper is for. It’s the author’s job to convey the purpose of the study and explain why their study design is fit for purpose. I wrote this post because when I read these papers …”

        I just wanted to point out that it’s difficult to discuss these points without knowing who “these papers” are, and what they write in their introduction. With the experimental setup that you describe, certain groups come to mind, but I wouldn’t agree that all or the majority of those work as naively as portrayed in the text. But OK, I understand that you don’t want to single out anyone in particular here.

  2. Jeremy – this is very helpful and thought provoking. I see this post is “uncategorized” which will make it hard to find in the future when I’ve forgotten the details but want to direct students to it in a methods class. How easy is it to add Category or Tags to posts so that teachers/mentors/students/researchers can more easily find posts related to specific categories of the kinds of posts you have here?

    • I used to categorize the posts into quite broad categories, but stopped doing so because I didn’t think anyone used them to find old posts (sorry!) People mostly find old posts via search engines, and categorizing posts doesn’t help search engines as far as I know.

      Also, even back when I was categorizing posts, the categories were quite broad (e.g., “teaching”). So I’m not sure they were that helpful. People who want to find old posts mostly seem to want to find specific old posts, or old posts on topics much narrower than the categories I used to use.

      Meghan sometimes uses detailed tags, but doing a bunch of detailed tags for every post is kind of a pain, sorry.

      • No apologies, I get it. WordPress should have an AI indexing widget that automates the generation of categories and tags

      • Purely anecdotally (because I have no idea what WordPress’ financials look like), I don’t think WordPress spends a lot on product development. My impression is that they’re a pretty modest operation, that devotes modest resources to trouble-shooting and tech support for the existing product and not much to making it better.

        I suspect they make most of their money from paid hosting (doubt they make much from selling ads), and AFAIK website hosting is a low margin, highly competitive commodity business.

        We pay them a bit of money every year to keep our site ad free. But honestly, the main reason I pay is because I worry that one of these years they’re going to vanish, and that will be the end of Dynamic Ecology. Or at least a tremendous pain when I attempt to migrate all the old posts and comments to a new platform. So I pay them a bit of money in the hopes that helps keep them a going concern.

        Hopefully I’m wrong to worry and their finances are just fine! But this is why I download all the posts and comments onto my hard drive once a year as a huge XML file. It’s a hedge against WordPress suddenly going under.

      • My dissertation was written in a now dead Adobe software — try opening those files! Given this experience and some since, I now use text based tools for everything (R, latex, rmarkdown, text editors, etc. etc.). For what it’s worth, I’ve spent the last few weeks investing a bunch of time developing a blog-website using rmarkdown + rstudio + Hugo + github + netlify. The advantage is every post is written in a simple text file (rmarkdown) and archived on github or whatever and so will never disappear nor is there some weird proprietary format that will disappear. The disadvantage is some cool things that seem very easy in wordpress are really hard with the geek route unless you know html + css + java — none of which I know. I don’t really see the rmarkdown/Hugo route taking off even though the tools that people have created are really amazing.

  3. Jeremy, do you see the uncanny valley of theory/microcosms as an issue of salesmanship (similar to presenting an exploratory study as hypothesis driven) or as a more fundamental problem with the design of the study?

  4. As a mathematical/theoretical E&E person, I’ve gotten people who question the validity of my results since it’s not “real life”. I think many ecologists fall into the trap of because it’s not tested experimentally esp. in the field that it cannot yet be taken as ‘true’. So a lot of people who do simple experiments or mathematical work feel the need to justify their results by latching it onto “real systems” even if they come nowhere close to accurately representing real systems.

    At least that’s my experience…

  5. Very interesting post Jeremy.

    It made me remember Paul Dayton’s article “Dayton, P. K. 1973. Two cases of resource partitioning in an intertidal community: making the right prediction for the wrong reason. The American Naturalist 107:662-670” and needing to be careful about oversimplification and misinterpretation using ecological models.

  6. Here’s a common area in which this comes up in the theoretical literature: modeling the population and community dynamics of species embedded in interaction networks such as food webs or plant-pollinator interaction networks. Often, the network topology–who eats whom in a food web, who pollinates whom in plant-pollinator network–is realistic. It’s an observed network topology, or a topology generated by some theoretical model designed to generate realistic network topologies. But the other aspects of the population/community dynamics typically are described with a Lotka-Volterra model or some other simple model, with more-or-less arbitrary parameters (except perhaps for some constraints on parameter choice that prevent the resulting population dynamics from causing a bunch of species to go extinct).

    Sometimes I think this sort of work is fine, sometimes it falls in uncanny valley for me. It really depends on the question of interest. For instance, if your question is “do realistic network topologies lead to more stable population dynamics than unrealistic topologies, all else being equal?”, then obviously you need to study the dynamics of species embedded in realistic interaction networks (and the dynamics of species embedded in unrealistic interaction networks). And obviously, it’s fine for you to describe those dynamics with some simple model like a Lotka-Volterra model, that loosely approximates the dynamics of many particular natural systems without being a close approximation to any of them. Because your question is about the effects of network topology, independent of the details of the population dynamic model. But I can imagine other questions for which the choice to use realistic network topologies is an arbitrary choice, no better or worse than some other arbitrary choice. And I can imagine still other questions for which the choice to use realistic network topologies would put the study in uncanny valley. Deliberately silly made-up example: a model of a multispecies fishery that uses a realistic food web topology, but that just uses a Lotka-Volterra model with arbitrary parameters for the population dynamics. Merely getting the food web topology right would not make that model is good tool for managing that particular fishery.

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.