The power of “checking all the boxes” in scientific research: the example of character displacement (UPDATED)

Brian’s written a lot on this blog about the importance and power of making and testing predictions (here’s his most recent post on this). I agree, but I also worry that, like any good thing, an emphasis on testing predictions can be taken too far (and I’m not the only one who thinks so). For instance, I’ve talked in the past about the importance of testing the assumptions as well as the predictions of one’s models, as a means of severely testing one’s models and as a way of learning from error. If your goal is to figure out how the world actually works, then you need to subject your hypotheses or theories about how it works to severe tests–tests that true hypotheses or theories would will pass with high probability, and that false hypotheses or theories will fail with high probability. You also need to figure out why your hypotheses or theories are false, because that tells you how to modify and improve them. You need to be able to localize the sources of your errors, like a programmer debugging code or an auto mechanic narrowing down the reason a car isn’t running properly.

I wanted to return to those themes by highlighting research on character displacement. This is a line of ecological and evolutionary research where researchers are doing a good job of testing their assumptions, thereby subjecting their ideas to severe tests and localizing exactly where their ideas fail.

In particular, I want to highlight a key paper on character displacement, Schluter and McPhail (1992). Their paper is a review of the evidence for character displacement in threespine stickleback in postglacial lakes in British Columbia. But the real importance of the paper is that it lays our a checklist of six criteria that you have to satisfy is you want to demonstrate character displacement, which I’ve briefly paraphrased below:

  1. The apparent pattern of character displacement isn’t just due to chance (i.e. it really is a pattern, not just noise or a coincidence)
  2. Differences in the trait of interest between sympatric and allopatric populations are genetically based
  3. The pattern is the result of evolution in sympatry, not divergence in allopatry followed by secondary contact
  4. A shift in the trait of interest is associated with an appropriate shift in resource acquisition
  5. There is competition for resources, and individuals with more similar traits compete more strongly, so that individuals with rare trait values have the highest relative fitness
  6. Sympatric and allopatric sites have similar resource availability, and are similar in other abiotic and biotic factors affecting trait evolution

There are several things I like about this checklist:

  • It’s a checklist! We often speak loosely of our admiration for very thorough, careful research. Research that considers the hypothesis from every angle, that “checks all the boxes”. Well, thanks to Schluter and McPhail 1992, research on character displacement literally does have to “check all the boxes”! It’s now standard for researchers on character displacement to refer to this checklist and specify the boxes that have been checked off (e.g., Beans 2014). (UPDATE: And see Stuart and Losos 2013 for a checklist-based review of the evidence for character displacement in animals. I was trying to remember this paper when I was writing the post, but couldn’t recall the authors or the journal. Thanks to a commenter for pointing it out). And it’s my sense that researchers on character displacement are acutely aware of just how few putative examples of character displacement check all the boxes. This kind of checklist is a reallyย effective way to prevent people from accepting a hypothesis that hasn’t actually been severely tested. It’s also a really effective way to focus everybody’s attention on the boxes that haven’t yet been checked–those are the most important lines of research to pursue (e.g., box #5 seems to be the hardest box to check in character displacement research). I can think of lots of areas of research in ecology and evolution that would really benefit from this sort of checklist! For instance, Simons (2011) structured his review of the empirical evidence for “bet hedging” life history strategies around a checklist of 6 different lines of evidence for bet hedging, ordered in terms of increasing strength. He found that the vast majority of empirical evidence for bet hedging came from the weakest lines of evidence.
  • It’s mostly about testing assumptions, not predictions. Character displacement theory doesn’t predict that individuals with more similar traits will compete more strongly, or that shifts in trait values lead to shifts in resource acquisition, or that allopatric and sympatric sites have similar resource availabilities, or etc. It assumes that all that stuff is the case, and then from those assumptions derives the prediction of character displacement. If those assumptions hold, character displacement must have occurred. And conversely, you cannot just test the predictions of character displacement theory and thereby show that the theory holds. Because those predictions (basically, larger interspecific phenotypic differences in sympatry than in allopatry) invariably have alternative explanations. Rather, you need to show that the predictions hold for the right reasons–which is what testing assumptions lets you show.
  • It forces you to specify your assumptions. There mere act of trying to list the assumptions on which your theory is based can be really valuable. It can force you to make your assumptions precise and explicit.
  • It raises the bar for ecological and evolutionary research. Schluter and McPhail’s checklist is similar in spirit to doing strong inference on character displacement. Ecologists and evolutionary biologists sometimes think of Platt’s idea of strong inference as desirable in principle, but impractical outside the realm of old-school molecular biology. But Schluter and McPhail’s checklist suggests that strong inference, or something quite like it, is more feasible in ecology and evolution than we sometimes think.

Of course, checklists won’t solve all the world’s problems. For instance, they don’t prevent researchers from checking off boxes on weak grounds. For example, if you try to use observational rather than experimental evidence to check off box #5, there are almost certainly going to be tears before bedtime.๐Ÿ™‚ But still, I do think it would be very helpful if we had such checklists in many other areas of research.*

So what do you think? Do we need more research checklists in ecology and evolution?

*Protip for grad students: I have just given you free advice for writing a great research proposal, which you can then turn into a high-impact critical review paper! Identify an area of research that could really use this sort of checklist, develop one, and then review the literature to see which studies check off which boxes. And then in your proposal, propose a study in which you’ll check all the boxes.๐Ÿ™‚

6 thoughts on “The power of “checking all the boxes” in scientific research: the example of character displacement (UPDATED)

    • Ah, thank you for the memory jog! When I was writing the post I was trying and failing to remember that paper. I knew there was a recent review using the checklist, but couldn’t remember who wrote it or in what journal. They say the memory is the first thing to go…๐Ÿ™‚

  1. This to me is an apples and oranges comparison Character displacement is a concept around for decades, modelled to death, demonstrated to death and meta-analyzed. This is just a list of blind alleys that have confused people and to be careful about.

    But I don’t think it is realistic to expect that list to appear right away for a really novel concept with one model and one test in one group of organisms. That is more the domain of prediction. And prediction did play a role in the early days of character displacement.People thought competition should lead to character displacement. They built models to confirm this intuition and then they went out and found them. Prediction! Success! Dozens of careers launched. Many pats on the back for the successful predictions of ecology. Then endless debates about whether this example really was character displacement. Then, much later, this list.

    What I would agree with is that such a checklist is a sign of a very mature topic that is long past the bandwagon stage. Doesn’t really comment on prediction or not prediction in my opinion.

    • Hmm, you’ve surprised me here Brian! And puzzled me as well. No so much in the broad strokes–I knew I wasn’t going to talk you out of an enthusiasm for prediction!–but in the details. You and I clearly have a different reading of the history of character displacement research, in the role of checklists like this one, and perhaps in other things.

      I confess I’m not sure how to respond besides noting my puzzlement, as unlike in past cases where we have friendly disagreements it seems like in this one I’ve just talked past you and you didn’t really “get” it. Which is probably my bad, I probably should’ve written it differently. Though I confess I’m not sure how.

      Let me try a different angle, picking up on (and gently pushing back against) your suggestion that checklists like this one are a sign of a mature line of research. So you don’t think it would’ve been possible to produce a checklist along these lines in, say, the 1970s for research on resource partitioning as a mechanism of competitive coexistence? And that maybe doing so would’ve saved ecologists from going down some blind alleys, and maybe even prevented development of some zombie ideas (like limiting similarity, which arguably traces at least in part to a misunderstanding of the assumptions underpinning some of MacArthur’s work)? Honest questions.

  2. Correct me if I’m wrong, but what I think Brian means is that you could not develop such a checklist in the 1970’s for the theory of character displacement. It was too new and had been subject to too few tests. Only in the 1990’s could Schluter and McPhail survey the collective wisdom built after a couple of decades of research, extract the key assumptions, and build such a checklist. During those earlier decades, the theory was built based on a back and forth between testing predictions, modifying assumptions, testing new predictions,

    • Yep – exactly. Most of that checklist is a list of rat holes we went down – places where people claimed character displacement then others disputed them based on one of the checklist points. Now did MacArthur/others anticipate many of those ratholes – in many cases yes. But did that prevent science from going down them? No.

Leave a Comment

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s