Technical statistical mistakes are overrated; ecologists (especially students) worry too much about them. Individually and collectively, technical statistical mistakes hardly ever appreciably slow the progress of entire subfields or sub-subfields. And fixing them rarely meaningfully accelerates progress. The rate of scientific progress mostly is not statistical mistake-limited.
Don’t agree? Try this exercise: name the most important purely technical statistical mistake in ecological history. And make the case that it seriously held back scientific progress.
Go ahead, I’ll wait. 🙂
Off the top of my head, here are some candidates for most important purely technical statistical mistakes in ecological history. None of which seem all that important in the grand scheme of things, honestly:
- Regressions of local species richness on regional species richness that failed to account for the boundedness of the operational space, biasing the results.
- Using the wrong null models as baselines against which to test for density dependence in population dynamics. Dennis & Taper 1994 review some of this history.
- Overfitting niche models by failing to allow for spatial autocorrelation in abiotic environmental conditions and species’ occurrences/abundances.
- If you want to argue that failure to account for detection probability is a purely technical statistical mistake that has seriously held ecology back, Brian will fight you. 🙂
- I don’t think that widespread adoption of generalized linear models over general linear models applied to transformed data fixes a technical mistake, or that it makes a big enough difference to the scientific conclusions in enough cases to constitute an important advance to ecology.
- I suppose you could follow Nate Silver and argue that null hypothesis significance testing is a technical mistake that has massively held back all scientific progress, including in ecology. I don’t buy that argument. But if you want to make the case for a statistical mistake appreciably limiting the progress of an entire scientific field, that’s probably the magnitude of mistake you’re looking for
Look, I’m very glad that technical mistakes the ones on my little list got fixed. I think those fixes were real advances. I’m just not convinced they were big advances. For instance, I don’t get the sense that fixing how we regress local richness on regional richness had any appreciable effect on the direction of that research program. My sense is that that research program always had more fundamental conceptual problems, and was already petering out or changing direction anyway by the time the statistical problem was fixed. As another example, see Brian’s old posts arguing that estimating detection probabilities hasn’t really much advanced or altered our understanding of wildlife ecology.
Of course, what constitutes a “purely technical statistical mistake” is open to debate. One common way that people who prefer statistical approach X argue for X is to argue that not using approach X is a purely technical mistake, rather than a defensible choice with upsides and downsides. That argument has been deployed for instance by defenders of estimating detection probabilities. As another example of the fuzzy line between “purely technical statistical mistake” and “other stuff”, consider the use of randomized null models of species x site matrices to infer interspecific competition or facilitation. Is that a purely technical statistical mistake? Or an ecological mistake–a mistaken ecological interpretation of technically-sound statistics? I dunno, honestly. I’d say ecological mistake, but I could imagine someone arguing otherwise.
I do think that technical statistical advances aid scientific progress. But I think they do by combining in complicated ways with all sort of other changes in the constantly-evolving ecosystem of scientific practice. And I think we’d be better off if we all recognized that and emphasized it more to our students. Students who come to me for statistical advice invariably worry far more about technical statistical issues than they need to, to the exclusion of more important issues like being clear about what scientific question they’re trying to address in the first place. Yes, you should get your technical statistics right enough (that last word is important). But it’s more important for you to think hard about what scientific question is worth asking, and all the things besides statistics that go into answering it.
I’m deliberately taking the strongest defensible position I can take on this, in the hopes to spark an interesting thread in which people come up with counterexamples and explain why I’m wrong. Plus, I only spent about 10 minutes trying to think of examples. I figured that this isn’t the sort of thing one can exhaustively research, so I might as well toss out a few examples and then open the floor. So have at it! 🙂
p.s. This post does not say or imply that pre-publication peer reviewers should stop caring about technical soundness, or that we should stop teaching our students statistics! Part of the reason technical mistakes don’t much inhibit scientific progress is because we teach our students not to make them, and because peer reviewers catch them. We, and they, should keep doing those things, because otherwise many more technical statistical mistakes would get made. We should just try to do those things in a way that doesn’t cause the possibility of technical statistical mistakes to take on an outsize importance in our minds. And I don’t think we need to do much more than we already do to prevent or correct statistical mistakes.