Techniques aren’t powerful; scientists are

During a long and interesting post on storytelling in science, Andrew Gelman makes the following remark about some famous statisticians and the techniques they’ve developed:

The many useful contributions of a good statistical consultant, or collaborator, will often be attributed to the statistician’s methods or philosophy rather than to the artful efforts of the statistician himself or herself…Rubin wielding a posterior distribution is a powerful thing, as is Efron with a permutation test or Pearl with a graphical model, and I believe that (a) all three can be helping people solve real scientific problems, and (b) it is natural for their collaborators to attribute some of these researchers’ creativity to their methods.

I think this is right. Techniques and approaches, statistical or otherwise, aren’t powerful except in narrow and, in the grand scheme of things, rather unimportant senses. If techniques were really what mattered, science could be reduced to a recipe that anyone could follow.*

What matters most, I think, is how and to what the technique is applied, and how the results are interpreted and linked to other evidence and ideas. None of that can be automated or routinized. Which means that what really matters is the ability of the scientist using the technique or approach, where “ability” is a fairly nebulous concept encompassing much more than just technical knowledge of the technique or approach. Rich Lenski wielding a vial of bacteria is a powerful thing, as is Mathew Leibold with a simple food web model, or Jon Losos with a phylogeny, or Tony Ives with an ARMA model**, or Peter Morin with a jar of protists, or me Steven Frank with the Price equation. But not because Rich Lenski has more technical knowledge of E. coli than other people, or because Steven Frank knows more math than other people, or etc.*** This is why the best papers in ecology rarely follow a recipe that anyone can follow without much thought.

Grad students: one implication of this is that you probably worry more than you need to about purely technical matters like whether to use GLS or OLS, and less than you need to about more important issues like whether you’re asking a good question in the first place.

p.s. It remains to be seen if “me wielding zombie jokes” turns out to be powerful. 😉 If it does, this is what I’m going to say. 😉

*Francis Bacon thought this was possible (“But the course I propose for the discovery of sciences is such as leaves but little to the acuteness and strength of wits, but places all wits and understandings nearly on a level.”) There’s much that he got right in “The New Organon” (1620), but I think this bit is wrong.

**or Tony Ives with many other things. Multa novit Tony Ives.

***Obviously, they do all have enough technical knowledge about their chosen approach to use it effectively. Meg has an old post on the value of sticking with systems and approaches you know well, because you’re less likely to make silly mistakes due to lack of technical knowledge.

Footnote: this is a lightly-revised version of a post that originally ran on the Oikos blog four years ago.

13 thoughts on “Techniques aren’t powerful; scientists are

  1. I disagree. All the examples of powerful scientists you gave would have been much less so without the benefit of techniques developed by previous researchers. No one person (other than Newton maybe) has the ability to develop all their techniques from scratch, and therefor they benefit from techniques developed by others. Differential equations and methods for their analysis, culturing techniques, etc. all gave the researchers a toolkit they in turn used to make new techniques and discoveries. I get your point that the best science isn’t formulaic. Applying a technique that has been used to do great science won’t necessarily make your science great. But good techniques let people (even Tony Ives) do science that they otherwise would not have been able to do.

    • You don’t disagree with me Ben, you’re asking a totally different question. Nothing in the post says or implies that Tony Ives (or whoever) could’ve done equally good science without any technical knowledge, or that great scientists are those who develop all of their own techniques and approaches from scratch! Please give me a little credit–I would never claim something as silly as that.

  2. Jeremy, interesting post, and of course it’s hard to disagree with the notion that a technique has no power or utility when not coupled with appropriate knowledge and understanding. And I think that the spirit of your message is here is meant to be empowering, but I’m a bit worried that it may send the wrong message by associating the approach directly with their creators or stars. No one should read this post and think, “well, I’m not Brad Efron so maybe I shouldn’t be using the bootstrap.” Some techniques are powerful especially because they can be employed very successfully by knowledgeable and critically thinking individuals, and I believe the bootstrap is one of them. Our new Data Science curriculum ( at UC Berkeley has done an excellent job this semester in teaching this bootstrap to undergraduate freshmen. The result has been far more critical thinking in application and far less rote recipe-following than I’ve seen in traditional introductory statistics courses.

    Not all techniques are created equal; obviously some of the most powerful techniques are things we already consider part of a general education, while others can be successfully applied only by a small and very advanced population of individuals (though I would suggest that there’s no truly powerful scientific technique that’s useful *only* in the hands of its creator — perhaps that distinguishes science from wizardry.) It is worth the scrutiny to identify what techniques we do not teach to a broad audience for good reasons, and which, like the bootstrap, we omit from lower-level courses for purely historical reasons (like lack of access to a computer).

    • “And I think that the spirit of your message is here is meant to be empowering”

      I didn’t intend it as empowering or disempowering. I just think that what distinguishes the best science from other science is mostly not a matter of technique. (given that every working scientist has a lot of technical knowledge, of course. Thought I’d better make this explicit to prevent serious misunderstandings like Ben’s above.)

      “No one should read this post and think, “well, I’m not Brad Efron so maybe I shouldn’t be using the bootstrap.””

      I agree. I hope that’s not the message people take away. Your comment here raises issues we’ve discussed in various old posts:

      “It is worth the scrutiny to identify what techniques we do not teach to a broad audience for good reasons, and which, like the bootstrap, we omit from lower-level courses for purely historical reasons (like lack of access to a computer).”

      Which techniques to teach to beginning students isn’t a topic I intended to raise in the post. But it’s an interesting and important topic and I’m happy for the conversation to move in that direction. I agree with your example–bootstrapping absolutely can be taught to beginning undergrads. We teach it in our intro biostats course now. I agree with you that the topic coverage in many intro biostats courses (including Calgary’s) is in need of updating, in various ways. For instance, it’s for historical reasons that we teach t-tests, regression, and ANOVA as separate things rather than just teaching general linear models from the get-go. Which seems like a bad idea, pedagogically, as it forces students to memorize a bunch of seemingly-unrelated tests and their associated jargon.

  3. I’m going to be a bit of a devils advocate here, and say that both techniques and the people who apply them scientists fall along a gradient. Both can be powerful. Both can be unpowerful. The best combination is a powerful scientist using his/her preferred powerful method. But a powerful scientist using an technique that isn’t powerful is not going to get anywhere. So I think it’s sort of an illogical argument to make that only one or the other of techniques and people can be powerful.

    And I’m going to be a diversity devils advocate here and ask: any powerful female scientists come to mind? Eh?

    • “Both can be powerful. Both can be unpowerful. ”

      I agree. But in practice, I think many ecologists–especially students–tend to worry too much about the technique and too little about larger issues. Hence statistical machismo, for instance. But I freely admit this impression isn’t backed by data.

      “And I’m going to be a diversity devils advocate here and ask: any powerful female scientists come to mind? Eh?”

      Fair comment. As is probably obvious, I just listed some names that came to me off the top of my head. They’re all people who happen to have come up on old posts on related topics, which is probably why they popped into my head. Yes, that is an illustration of how just going with the first names to pop into your head often will lead to a biased list of names. And yes, Meghan Duffy wielding a flask of Daphnia (really, Meg wielding a combination of approaches) is a powerful thing. Sally Otto wielding a simple mathematical model is a powerful thing…

  4. Oh well. I think science MUST be reducible to a recipe. If it can’t be repeated by others its not science.

    “powerful” scientists are the scientists who follow the recipe and/or are able to modify it to make a new recipe. I’ve always been surprised by the number of scientists who DONT follow the recipe well.

    • With respect, you misunderstood the post, Jim.

      For instance, once someone like Rich Lenski has come up with the idea for the Long-Term Evolution Experiment (LTEE), absolutely, anyone could replicate it. But not just anyone could first decide to ask the questions the LTEE asks, and then come up with the idea of the LTEE to answer them.

    • However, there is a difference between being able to write down a recipe after the fact, and working from a recipe before the fact. Science is arguably about producing new “recipes”, by mixing and matching components of old recipes with ideas, hypotheses and creativity. The recipe metaphor should probably not be taken too far though.

  5. Pingback: While I was away… some recommended reads | The Lab and Field

  6. Pingback: Statistik dan Arus Informasi dalam Ekologi: Apa Saja yang Harus Dipelajari? | Cuma Ide

  7. Pingback: Friday links: another retraction for Jonathan Pruitt, journal cover art, and more | Dynamic Ecology

Leave a Comment

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.