Hurlbert’s Classic Pseudoreplication Paper & Using MythBusters to Teach Undergrads About Experimental Design

As Jeremy revealed yesterday when announcing the top papers of the 70s, 80s, and 90s, Hurlbert’s 1984 paper on pseudoreplication is not only the most cited ecology paper of the 1980s, it’s the most cited from that whole window, having been cited 4,157 times to date. 4,157! (My number differs slightly from Jeremy’s; it was cited 11 times between when Jeremy looked up the numbers and when I did this past weekend.)

I first read this paper as a grad student, and it was one that came up often during my training as a grad student. I view it as a foundational paper that all students should read, and give it to all of my incoming students. So, when I first got to Georgia Tech, I was a little surprised to realize that it wasn’t routinely assigned; Chamberlin’s “Method of Multiple Working Hypotheses” was the more typically assigned “how to do science” sort of paper among the ecologists. That made me wonder if there are subfield differences in whether people view this as a seminal paper; there may well be (and, if readers have thoughts on this, I’d love to hear them in the comments), but clearly lots of people view Hurlbert as worth reading and citing!

I also assign this paper to undergraduates in classes where we read a lot of the primary literature. (My upper level courses have very little lecturing, and instead have students reading and discussing 1-2 papers from the primary literature each class period.) I usually spend the first class period going over basics of experimental design, as students need a basic understanding of experimental design to be prepared to read and evaluate the primary literature. For that class, I assign Hurlbert (and sometimes also Chamberlin). Students always really love Hurlbert’s paper – I think in part because he goes into specific examples from the literature (which students usually find a little surprising). Plus, who doesn’t like the idea of demonic intrusions? Well, in theory; no one likes demonic intrusions when they occur in his/her own experiment! (In case you aren’t familiar with the paper (in which case – go read it!!!), Hurlbert recommends “eternal vigilance, exorcism, human sacrifices, etc.” as means of reducing or eliminating the effects of demonic intrusions on experiments. He also cautions that “If you worked in areas inhabited by demons you would be in trouble regardless of the perfection of your experimental designs.” Sad, but true.) More seriously, students also like the paper because they finally start to understand a bit more about proper controls and replication. I had multiple students tell me they wish they had read the paper when they were freshman, since they feel like it would have helped with all their lab coursework.

I like to pair reading Hurlbert with an activity that gets students to think more carefully about experimental design, and also to consider constraints. I think it’s very important to talk about what can be done realistically; otherwise, it’s easy for students to have unrealistic expectations about what can be done, which can lead to discussions getting bogged down on experiment critiques. My favorite activity involves taking part of an episode of MythBusters. The episode focuses on whether cockroaches will inherit the earth after a nuclear holocaust. We first watch just enough to know what the question is. I then have the students work in small groups to design how they would do the experiment, both if they have access to a radiation source and if they did not. The latter is where we get into issues of feasibility – yes, the ideal study in this case directly manipulates radiation, but that might be possible. How else could this question be studied? I’ve found that students are remarkably creative – and also that none of them go for what I would have done, which would have been a study based in Chernobyl. (Now that I think about it, I wonder if, when I next do this activity, some students will suggest going to Fukushima.) After we discuss their proposed studies, we watch how the MythBusters crew did it and discuss whether what they did was pseudoreplicated. (This takes more discussion than you might initially imagine.) We then watch the rest of the episode (well, the parts that deal with this experiment). In all, I’ve found it a very fun way to discuss experimental design in an accessible way, and students really enjoy it.

I’d be curious to hear what others do. Do you have undergrads read Hurlbert? How do you teach basics of experimental design (in a class that is not explicitly focused on experimental design or statistics)?

Related post:
Hurlbert rips Sokal & Rohlf