Why don’t more ecologists use strong inference?

Platt (1964 Science) is a classic practical statement of philosophy of science by a scientist. Briefly, Platt argues that some fields of science progress faster than others, and that this is neither an accident nor attributable to variation in intrinsic skill or brilliance among scientists in different fields. Rather, he says that workers in some fields routinely use a method that more or less guarantees progress: strong inference. They line up a bunch of competing alternative hypotheses, and then conduct the decisive observations and experiments to distinguish among those hypotheses. So that in the end, they either identify the correct hypothesis, or know they have to go back to the drawing board and come up with some new hypotheses.

At least according to one small sample, ecologists mostly don’t do strong inference.* Why not?

tl;dr: I think Brian’s wrong and that the usual excuse for not doing strong inference in ecology is mostly bogus. I think ecologists could, and should, do a lot more strong inference than they do.

The usual excuse for not doing strong inference in ecology is that ecology is too noisy and complicated. Platt was a molecular biologist working with highly-controlled, low-noise systems in the lab, making decisive experiments feasible. And the alternative hypotheses of interest in molecular biology at the time were mutually exclusive, and either right or wrong. Ecologists have to deal with many sources of variation, including but not limited to massive sampling error. Our alternative hypotheses often aren’t mutually exclusive and often are matters of degree. And experiments are infeasible at some spatial and temporal scales. That’s why Quinn and Dunham (1983), and Brian, say you can’t do strong inference in ecology, except perhaps in some unusual special cases.

That all sounds very plausible. It’s also wrong. As evidenced by the many, many ecologists who’ve done strong inference. Way too many for them to all be unusual special cases with no lessons to offer the rest of the field. Just off the top of my head, supplemented with one casual Web of Science search:

  • McCauley et al. 1999 is the culmination of series of experiments by Ed McCauley and colleagues to nail down the explanation for why Daphnia in nature don’t exhibit the paradox of enrichment. As Meg will be happy to confirm to you if you doubt it, Daphnia dynamics in nature are noisy. And the five explanations for the absence of the paradox of enrichment in natural Daphnia dynamics aren’t mutually exclusive. But yet, it was still perfectly possible to do a decisive series of experiments testing each of them in turn.
  • Kendall et al. 1999 show how to do strong inference about the causes of population cycles, at least in well-studied species.
  • Schluter and McPhail 1992 developed a now-standard checklist of assumptions and predictions that need to be tested in order to demonstrate character displacement and rule out alternative possibilities such as divergence in allopatry followed by secondary contact. Several field systems have now been shown to tick every box on the list. The Schluter and McPhail checklist is a particularly telling counter-example to the claim that you can’t do strong inference in ecology because ecology is “noisy”. Ruling out “chance” or “noise” is the first item on Schluter & McPhail’s checklist.
  • Simons 2011 develops a checklist for what you need to show, and what alternative hypotheses you need to rule out, to demonstrate bet hedging. As with character displacement, a few studies tick every box on the list, though most don’t. And if you say that it’s only possible to develop checklists and get others to use them after a long history of failed attempts to test the ideas in question, well, so what? Saying that “we can do strong inference in ecology, it just takes a while to figure out how to do it in any particular case, and then get people to actually do it” is totally different from saying “we can’t do strong inference in ecology”. I’m fine with the fact that, when ecologists first start exploring some big question, they may not be in a position to do strong inference–just so long as we work our way up to doing strong inference eventually.
  • Ford and Goheen 2015 call for studies of trophic cascades involving large carnivores to be based on strong inference, and explain in detail how this could be done (Adam Ford’s done it himself in his own work). Even though large carnivores are infamously difficult to study in controlled replicated experiments at the relevant spatial and temporal scales, and even though the alternative hypotheses aren’t mutually exclusive.
  • Downes 2010 does the same for studies of effects of stressors in streams. Notably, she uses the noisiness and complexity of stream systems as an argument for strong inference, not an argument against it.
  • O’Connor et al. 2015 reviewed studies of climate change impacts and found that only a minority stated prior expectations and tested alternative hypotheses about drivers of change. But that some of them did shows that strong inference of climate change impacts is possible. Failure of many climate impact studies to use strong inference has nothing to do with the inherent noisiness or complexity of the problem.
  • Similarly, Dochterman and Jenkins 2011 note that behaviorial ecologists often have tested multiple alternative hypotheses, and suggest ways for them to do so more often and more effectively.
  • Many recent Mercer Award winning papers test the hypothesis of interest from all angles, powerfully combining different research approaches and lines of evidence to show that the hypothesis passes a severe test (meaning: a test that a correct hypothesis would pass with high probability, and an incorrect one would fail with high probability). Which seems to me to be close enough to strong inference for government work.
  • Similarly, in a guest post here, Britt Koskella talked about how she combined microcosm experiments and other lines of observational and experimental evidence to cut through the complexity of nature and test alternative hypotheses about parasite-host coevolution.
  • Krebs’ informal survey, linked to above, found that 22% of papers in the first two issues of Journal of Animal Ecology in 2015 had explicit hypotheses and alternatives. Okay, you’d have to look more closely at those papers to see if they actually were successful implementations of strong inference. But surely 22% is too high a fraction to just dismiss strong inference as obviously impossible in ecology.
  • Other examples that I’m sure must exist. 🙂

My point isn’t that strong inference is the infallible recipe for progress Platt made it out to be–it’s not. But the notion that ecology is somehow not amenable to strong inference is just false.

So if ecologists could do more strong inference, why don’t they? I don’t know, but here are some hypotheses:

  • Strong inference can be hard. Simons 2011 notes that the studies of bet hedging that only tick some of the boxes almost invariably tick the easiest boxes. The same is true for studies of character displacement.
  • Inertia. If there’s an established way of studying X, it’s quite likely that lots of people will keep following that approach, even if it doesn’t involve strong inference. It’s that sort of inertia that those review/perspectives pieces arguing for strong inference are pushing back against.
  • Self-fulfilling prophecy. If you think that the world is really noisy and complicated, so that you don’t expect strong inference to work, maybe you never try it, or expect anyone else to try it.
  • The oft-remarked on tendency of many scientists to conflate statistical null hypothesis tests with tests of substantive scientific hypotheses. Some people think they already have alternative hypotheses–the statistical null hypothesis and the statistical alternative hypothesis.
  • An understandable but overly-narrow interpretation of strong inference as nothing but replicated, controlled, manipulative experiments. Ok, that was Platt’s original formulation. But I think it’s more useful to define strong inference more broadly, as I have in this post. The point is to check your hypotheses from every angle (both assumptions and predictions) so as to subject them to severe tests.
  • Conversely, some of it is failure to recognize cases in which experiments are perfectly possible and would be really useful. Too many ecologists mistakenly think that, because the phenomena they’re interested in predicting or explaining occur on spatial or temporal scales on which experimentation is impossible, that relevant experiments are impossible. For instance, regarding the causes of local-regional richness relationships. Just because you’re interested in some “large scale” or “long term” phenomenon does not mean that small scale, short term experiments are uninformative! On the contrary, they’re often hugely informative, since many large-scale, long-term phenomena represent the cumulative outcome of small-scale, short-term events and processes that happen everywhere, all the time. Always remember: macroecology is community ecology all the way down. Yes, the large-scale, long-term implications of small-scale, short-term processes can be difficult to suss out–but you’re voluntarily discarding evidence if you don’t try. (Aside: in researching this post, I was struck by how many review/perspectives pieces on strong inference in various areas of ecology make this same point: that ecologists are missing opportunities to do strong inference by not drawing on research approaches and lines of evidence they could be drawing on.)
  • Unwillingness to eliminate seemingly-intractable complexity by asking higher-level questions. Nobody in evolutionary biology complains that the stuff they study is intractably noisy and complicated. Not because evolution is inherently any “simpler” than ecology, but because evolutionary biologists ask “higher level” questions than ecologists typically do.

Entertaining refutation from Brian in 3…2… 🙂

p.s. Just to be clear: I think ecologists could do more strong inference than they do, but I don’t think 100% of ecology could or should be strong inference. There’s more to science than inference about hypotheses.

*It would be very interesting to enlarge this sample and write a paper about what you found. That paper would definitely have a shot in a top journal. Grad students: instead of participating in a conventional reading group next term, rope in some of your friends and do this instead as a side project! You’ll still be reading a lot, but you’ll also get a paper out of it.

11 thoughts on “Why don’t more ecologists use strong inference?

  1. tl;dr – I think Jeremy is wrong. 🙂 Or at least has a different definition of strong inference than I do. I very much take the Quinn & Dunham view and haven’t seen anything that refutes it. Namely strong inference as discussed in Platt is about mutually exclusive choices DNA is the genetic material or proteins are. Decisive test tells us DNA is the genetic material. We have ratcheted forward in our knowledge.

    But if I do an ecological study of what is controlling the number of deer. It could be predation. It could be hunting. It could be ticks/disease. It could be cold temperatures. It could be snow depth preventing browsing. Those are NOT mutually exclusive how can I do strong inference? Simply saying hunting does or does not have an effect on deer population is a much weaker statement and not strong inference in my book (or I think Platt’s).

    Jeremy, how exactly do you define strong inference in a multicausal world?

    • Fair response.

      I’ll note in passing that my final bullet point is a response to your deer abundance example.

      But more broadly, perhaps we don’t disagree *that* much, because I do indeed prefer to define “strong inference” more broadly than Platt originally defined it. I basically think of it as severe testing–you check *all* the alternative hypotheses from *every* angle, checking both their assumptions and predictions. It bugs me that the standard arguments against strong inference sensu stricto in ecology work equally well as arguments against severe tests more broadly. So Brian: in a multicausal world in which strong inference sensu stricto is impossible, how do we do severe testing? I know you’re bothered by what you see as ecologists’ collective lack of a “problem solving mentality”, their collective unwillingness to ever totally reject an idea. How do you adopt a “problem solving mentality” in a multicausal world?

      p.s. I have a follow-up post in mind about what I see as under-discussed obstacles to strong inference (and severe testing more broadly) in ecology. So if you wait a little while, I might refute myself. 🙂

      • Fair enough. To me “strong inference” is strongly tied to Platt and I hold pretty much irrelevant to ecology. But Mayo’s “severe tests” or my preferred touchstone from Lakatos of “bold predictions” are definitely part of just “good science” which you can definitely do in ecology. Indeed I would agree with most of your examples as being examples of “that” whatever you call it.

        I do think the much older paper by Chamberlain that talks about competing hypotheses is more relevant than Platt but gets much less attention (Chamberlain doesn’t assume mutual exclusiveness like Platt does).

        I think the problem is, as we’ve both said before, there is no simple recipe for good science. It often has hypotheses, theory, multiple competing hypotheses, a mixture of methods and empirical modes of testing, etc. But there is no sine qua non. There are lots of valid rigorous approaches. The hard part is clear thinking and lots of work assembling lines of evidence.

  2. Another possible reason ecologists don’t seem to use strong inference is that they spread tests of multiple alternative hypotheses over more than one paper. In other words, strong inference might happen at the level of a whole research program, but not at the level of individual papers.

  3. You raise an excellent point, Jeremy. The testing of multiple, potentially non-exclusive hypotheses is difficult to say the least. My personal preference has been to attack hypotheses one at a time, regardless of the issue of potential overlap.

    One area I find problematic for many scientists concerns the issue of varied methodologies to test a hypothesis. Obviously time and funding can put a damper on that, so I encourage folks to build the approach into grant proposals, rather than struggle post hoc.

    With some regularity I see hypotheses supported or refuted in the literature based upon singular approaches to testing them. I would point to those examples as ‘weak inference’. Alternatively, a study that triangulates a hypothesis is far more rigorous, and often reveals that which singular approaches cannot.

    A good example of this in ecology relates to spatial resolution. So for example, among a community of plant species, only a singular spatial resolution will reveal the true distribution of two or three species, at best, due to over- or under-sampling of patches relative to the landscape. Ergo, a lot of information is excluded as it pertains to most species distributions in the ecosystem.

  4. Pingback: Check out Jeremy Fox’s blog about strong inference in ecology – ECOL 8000 – Fall 2016

  5. Pingback: Friday links: academic pariahs (except not really), publish and perish, and more | Dynamic Ecology

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.