The power of “checking all the boxes” in scientific research: the example of character displacement (UPDATED)

Brian’s written a lot on this blog about the importance and power of making and testing predictions (here’s his most recent post on this). I agree, but I also worry that, like any good thing, an emphasis on testing predictions can be taken too far (and I’m not the only one who thinks so). For instance, I’ve talked in the past about the importance of testing the assumptions as well as the predictions of one’s models, as a means of severely testing one’s models and as a way of learning from error. If your goal is to figure out how the world actually works, then you need to subject your hypotheses or theories about how it works to severe tests–tests that true hypotheses or theories would will pass with high probability, and that false hypotheses or theories will fail with high probability. You also need to figure out why your hypotheses or theories are false, because that tells you how to modify and improve them. You need to be able to localize the sources of your errors, like a programmer debugging code or an auto mechanic narrowing down the reason a car isn’t running properly.

I wanted to return to those themes by highlighting research on character displacement. This is a line of ecological and evolutionary research where researchers are doing a good job of testing their assumptions, thereby subjecting their ideas to severe tests and localizing exactly where their ideas fail.

In particular, I want to highlight a key paper on character displacement, Schluter and McPhail (1992). Their paper is a review of the evidence for character displacement in threespine stickleback in postglacial lakes in British Columbia. But the real importance of the paper is that it lays our a checklist of six criteria that you have to satisfy is you want to demonstrate character displacement, which I’ve briefly paraphrased below:

1. The apparent pattern of character displacement isn’t just due to chance (i.e. it really is a pattern, not just noise or a coincidence)
2. Differences in the trait of interest between sympatric and allopatric populations are genetically based
3. The pattern is the result of evolution in sympatry, not divergence in allopatry followed by secondary contact
4. A shift in the trait of interest is associated with an appropriate shift in resource acquisition
5. There is competition for resources, and individuals with more similar traits compete more strongly, so that individuals with rare trait values have the highest relative fitness
6. Sympatric and allopatric sites have similar resource availability, and are similar in other abiotic and biotic factors affecting trait evolution

There are several things I like about this checklist:

• It’s a checklist! We often speak loosely of our admiration for very thorough, careful research. Research that considers the hypothesis from every angle, that “checks all the boxes”. Well, thanks to Schluter and McPhail 1992, research on character displacement literally does have to “check all the boxes”! It’s now standard for researchers on character displacement to refer to this checklist and specify the boxes that have been checked off (e.g., Beans 2014). (UPDATE: And see Stuart and Losos 2013 for a checklist-based review of the evidence for character displacement in animals. I was trying to remember this paper when I was writing the post, but couldn’t recall the authors or the journal. Thanks to a commenter for pointing it out). And it’s my sense that researchers on character displacement are acutely aware of just how few putative examples of character displacement check all the boxes. This kind of checklist is a really effective way to prevent people from accepting a hypothesis that hasn’t actually been severely tested. It’s also a really effective way to focus everybody’s attention on the boxes that haven’t yet been checked–those are the most important lines of research to pursue (e.g., box #5 seems to be the hardest box to check in character displacement research). I can think of lots of areas of research in ecology and evolution that would really benefit from this sort of checklist! For instance, Simons (2011) structured his review of the empirical evidence for “bet hedging” life history strategies around a checklist of 6 different lines of evidence for bet hedging, ordered in terms of increasing strength. He found that the vast majority of empirical evidence for bet hedging came from the weakest lines of evidence.
• It’s mostly about testing assumptions, not predictions. Character displacement theory doesn’t predict that individuals with more similar traits will compete more strongly, or that shifts in trait values lead to shifts in resource acquisition, or that allopatric and sympatric sites have similar resource availabilities, or etc. It assumes that all that stuff is the case, and then from those assumptions derives the prediction of character displacement. If those assumptions hold, character displacement must have occurred. And conversely, you cannot just test the predictions of character displacement theory and thereby show that the theory holds. Because those predictions (basically, larger interspecific phenotypic differences in sympatry than in allopatry) invariably have alternative explanations. Rather, you need to show that the predictions hold for the right reasons–which is what testing assumptions lets you show.
• It forces you to specify your assumptions. There mere act of trying to list the assumptions on which your theory is based can be really valuable. It can force you to make your assumptions precise and explicit.
• It raises the bar for ecological and evolutionary research. Schluter and McPhail’s checklist is similar in spirit to doing strong inference on character displacement. Ecologists and evolutionary biologists sometimes think of Platt’s idea of strong inference as desirable in principle, but impractical outside the realm of old-school molecular biology. But Schluter and McPhail’s checklist suggests that strong inference, or something quite like it, is more feasible in ecology and evolution than we sometimes think.

Of course, checklists won’t solve all the world’s problems. For instance, they don’t prevent researchers from checking off boxes on weak grounds. For example, if you try to use observational rather than experimental evidence to check off box #5, there are almost certainly going to be tears before bedtime. 🙂 But still, I do think it would be very helpful if we had such checklists in many other areas of research.*

So what do you think? Do we need more research checklists in ecology and evolution?

*Protip for grad students: I have just given you free advice for writing a great research proposal, which you can then turn into a high-impact critical review paper! Identify an area of research that could really use this sort of checklist, develop one, and then review the literature to see which studies check off which boxes. And then in your proposal, propose a study in which you’ll check all the boxes. 🙂