During a long and interesting post on storytelling in science, Andrew Gelman makes the following remark about some famous statisticians and the techniques they’ve developed:
The many useful contributions of a good statistical consultant, or collaborator, will often be attributed to the statistician’s methods or philosophy rather than to the artful efforts of the statistician himself or herself…Rubin wielding a posterior distribution is a powerful thing, as is Efron with a permutation test or Pearl with a graphical model, and I believe that (a) all three can be helping people solve real scientific problems, and (b) it is natural for their collaborators to attribute some of these researchers’ creativity to their methods.
I think this is right. Techniques and approaches, statistical or otherwise, aren’t powerful except in narrow and, in the grand scheme of things, rather unimportant senses. If techniques were really what mattered, science could be reduced to a recipe that anyone could follow.*
What matters most, I think, is how and to what the technique is applied, and how the results are interpreted and linked to other evidence and ideas. None of that can be automated or routinized. Which means that what really matters is the ability of the scientist using the technique or approach, where “ability” is a fairly nebulous concept encompassing much more than just technical knowledge of the technique or approach. Rich Lenski wielding a vial of bacteria is a powerful thing, as is Mathew Leibold with a simple food web model, or Jon Losos with a phylogeny, or Tony Ives with an ARMA model**, or Peter Morin with a jar of protists, or
me Steven Frank with the Price equation. But not because Rich Lenski has more technical knowledge of E. coli than other people, or because Steven Frank knows more math than other people, or etc.*** This is why the best papers in ecology rarely follow a recipe that anyone can follow without much thought.
Grad students: one implication of this is that you probably worry more than you need to about purely technical matters like whether to use GLS or OLS, and less than you need to about more important issues like whether you’re asking a good question in the first place.
*Francis Bacon thought this was possible (“But the course I propose for the discovery of sciences is such as leaves but little to the acuteness and strength of wits, but places all wits and understandings nearly on a level.”) There’s much that he got right in “The New Organon” (1620), but I think this bit is wrong.
**or Tony Ives with many other things. Multa novit Tony Ives.
***Obviously, they do all have enough technical knowledge about their chosen approach to use it effectively. Meg has an old post on the value of sticking with systems and approaches you know well, because you’re less likely to make silly mistakes due to lack of technical knowledge.
Footnote: this is a lightly-revised version of a post that originally ran on the Oikos blog four years ago.