Different fields and subfields of science have different methodological traditions. Standard approaches that remain standard because students learn them.
Which to some extent is inevitable. Fields of inquiry wouldn’t exist if they had to continuously reinvent themselves from scratch. You can’t literally question everything. Further, tradition is a good thing to the extent that it propagates good practices. But it’s a bad thing to the extent that it propagates bad practices.
Of course, it’s rare that any widespread practice is just flat-out bad. Practices don’t generally become widespread unless there’s some good reason for adopting them. But even widespread practices have “occupational hazards”. Which presumably are difficult to recognize precisely because the practice is widespread. Widespread practices tend to lack critics. Criticisms of widespread practices tend to be ignored or downplayed on the understandable grounds of “nothing’s perfect” and “better the devil you know”.
Here’s one way to help you recognize when a widespread practice within your own field may be ripe for rethinking: look at whether the practice is used in other fields, and if not, what practices those other fields use instead to address the same problem. Knowing how things are done in other fields helps you look at your own field with fresh eyes.
As an example from ecology, consider randomization-based null models. By which I mean models that start with some observed data, and try to figure out what the data would have looked like in the absence of some ecological process (e.g. interspecific competition) by randomly shuffling the observations under some constraints. The idea is that the random shuffling will remove all and only the effects of the process of interest, while the constraints will retain the effects of other processes. If the randomized data resemble the observed data, that shows that the process of interest is absent or unimportant. See here, here, here, here, and here for discussion of various examples, such as mid-domain effect models.
Longtime readers will know that I’m critical of randomization-based “null” models as a tool for inferring process from pattern; I think they mostly don’t work for that purpose (follow the links above for discussion). But for purposes of this post, I’m more interested in how researchers in other fields approach the same problem. Do researchers in other fields use randomized “null” models the way ecologists do? If not, what do they use instead?
Researchers in other fields do often start with observed data and then use some model-based approach to subtract out the effects of some particular process on those data, thereby revealing what the data would have looked like in the absence of that process. But in every case I can think of (which I admit isn’t that many), the model is not a constrained randomization of the observed data. In many cases, it’s some independently-validated theoretical model of the process of interest. I’m thinking for instance of how astronomers use the Boltzmann equation and the Saha equation to subtract out the effects of ionization from observed stellar spectrograms in order to infer what stars are made of (see here and many other websites; this is a standard topic of undergraduate astronomy courses).* In other cases, the subtraction is based on independently-validated background knowledge. Think for instance of how opinion pollsters correct for the fact that some people are more likely to respond to polls by weighting their data based on independent knowledge of the population demographics (e.g., census data).
In looking into how other fields try to subtract out the effects of particular processes from their observed data, I’ve been struck that they never say or assume that the resulting data will be “random”. As far as I can tell, it’s only ecologists who ever think that correctly subtracting out the effects of process X from observed data would result in random data.
But maybe that just shows my ignorance of how other fields work. So here’s my question: do you know of any field besides ecology that uses randomization-based null models to try to subtract out the effects of particular processes from observed data?
I emphasize that I’m not talking about using randomization-based null hypothesis tests in statistics. Plenty of fields use those. I’m talking about attempts to use the same “logic” to test a substantive scientific hypothesis.
*And if you say that ecologists can’t use that approach because we lack that sort of quantitative theory, how can you have any confidence that your randomized null model is actually working as intended? If you can’t write down an explicit quantitative model of the effects of the process you’re trying to subtract out from your data, how do you know that constrained randomization is the correct implicit model?