tl;dr: No, my paper’s not having much impact. But that might be because there’s no potential impact for it to have, because the IDH is already a “ghost”.
Let me back up. For those of you who are just joining us, the intermediate disturbance hypothesis (IDH) says that species diversity is, or should be expected to be, a humped function of disturbance frequency or intensity. It’s a widely-cited, highly-influential idea that features prominently in various ecology textbooks, but it’s been strongly criticized on empirical and theoretical grounds (e.g., Chesson & Huntly 1997 (great paper), Mackey & Currie 2001, Miller et al. 2011). I don’t think those criticisms have been noticed and taken on board nearly as much as they should’ve been. So I’ve written a bunch of posts highlighting those criticisms and referring to the IDH as a “zombie idea”—an idea that should be dead, but isn’t. It lives on not because the people who buy into are bad scientists—they’re not—but because of the basic conservatism of the scientific process. The IDH was a good idea when it was first proposed, and it quickly took root. And once any idea takes root, it’s very hard to uproot (see also). But that doesn’t mean we shouldn’t try. Criticism of ideas is part of science.
But criticism of ideas on a blog only gets you so far, for various reasons. For instance, the majority of ecologists don’t read this blog.* So in 2013, I published an opinion piece in Trends in Ecology and Evolution that grew out of my blog posts, arguing that the IDH should be abandoned. It’s now been a bit more than two years since that paper was published, enough time for any impact it might be having to start showing up in the peer-reviewed literature. So I decided to look at how Fox (2013) is being cited. Did I slay the IDH? Or did my paper have the same effect as Clarence Darrow’s closing speech in the Scopes Trial?** Or somewhere in between?
As of mid-August, Web of Science listed 42 papers citing Fox (2013).*** That’s a lot for an ecology paper that’s only two years old! So between that, the fact that TREE is a prominent venue, and the fact that Fox (2013) was the most-downloaded TREE paper the year it was published, Fox (2013) certainly hasn’t gone unnoticed or been ignored. But just for context, Connell (1978), the classic paper that coined the term IDH, has been cited 581 times since 2013. Now, some of those citations are from papers having nothing to do with the IDH—but as you’ll see in a sec, the same is true for papers citing Fox (2013). So only a small minority of IDH-related papers published since 2013 cite Fox (2013).
It’s one thing for people to read or even cite a paper, another for the paper to affect what sort of science gets done and how that science gets reported and discussed. So I skimmed those 42 papers to see how they cited Fox (2013). Below, I summarize the results, because the big picture is what matters. I don’t want to single out particular authors. Note that one or two papers cited me in more than one way and so are counted twice.
No idea how they cited me because I couldn’t access the full text: 10 papers. Even though I’m at a large research university that subscribes to lots of journals. I think this has interesting implications, which I’ll return to.
Cited in passing: 15 papers. The most common category. These are citations for statements in passing that the IDH has “recently been criticized”, or “remains controversial”, or is the subject of “ongoing discussion”, etc. They usually occurred early in the introduction.
Cited as a primary reference on competition-colonization trade-offs: 3 papers cited me as a reference on how competition-colonization trade-offs work. Even though Fox (2013) doesn’t actually explain competition-colonization trade-offs—it just cites papers on the topic.
Cited as a primary reference on the storage effect: 1 paper that isn’t about disturbance and diversity cited me as a source for a statement about how the storage effect works. Which I can sort of see, since Fox (2013) has a textbox explaining the storage effect.
Cited because the authors totally agree with me: 4 papers cited me to say that the IDH, as traditionally defined, hasn’t worked out. Three of these went on to report research on diversity-disturbance relationships based on modern coexistence theory. The authors of these three papers are now my science crushes. 🙂 I’m especially pleased to see that one of these papers was co-authored by someone who reads and comments on Dynamic Ecology. :-)
Miscited: 2 papers miscited me. That’s annoying, but it’s not a big deal so I won’t say more. Neither is the worst I’ve ever been miscited.
Cited by mistake: 1 citation was for a statement about nutrient retention in lakes, in a paper that has nothing to do with disturbance or diversity. I assume the authors must’ve clicked the wrong “Fox 2013” in their reference management software.
Cited in some other way: 4 papers.
This little exercise changed how I think about the IDH. Here are my thoughts:
- I’m not surprised that the most common response to Fox (2013) in the literature has been to just note its existence in passing and move on. It’s very rare for any one paper, even a high-profile paper, to have much effect on anything. Especially a paper that tells people to stop doing something, but doesn’t spell out in any detail what they ought to do instead. Well, I did say people should apply modern coexistence theory to diversity-disturbance relationships. But I didn’t say how to do that. Fortunately, several groups are now doing it.
- Following on from the previous bullet: I of course can’t tell if any papers weren’t published because of Fox (2013), either because I changed people’s minds about what work to do, or because I changed reviewers’ and editors’ minds about what work to accept. But honestly, I highly doubt that Fox (2013) prevented publication of any papers that would otherwise have been published. That’s just not how science operates in my experience. (UPDATE: Let me just emphasize that I really did not expect Fox (2013), or my various blog posts, to make much difference to how the field as whole treats the IDH. These first two bullets are mostly aimed at any readers who are under the mistaken impression that a single critical paper or a few critical blog posts can have a big influence on an entire field.)
- Here’s the thing that struck me most: not only did these 42 papers mostly only cite Fox (2013) in passing, they mostly only cited the IDH in passing. Even the ones that were about diversity-disturbance relationships. Not that they didn’t mention the IDH at all–many began with a one-paragraph review of the IDH. But they hardly ever returned to the IDH later in the paper. And with very few exceptions, they weren’t framed as tests of the IDH, often focusing on system-specific biology instead. They mentioned the IDH, but went on to focus on the effects of disturbance in their own study systems. Without much or even any attempt at generalizing the results, and little or no suggestion of any implications for the IDH. Honestly, I’m not sure why the large majority of these 42 papers bothered citing the IDH, much less Fox (2013). Their citations of the IDH didn’t do any intellectual work–didn’t provide a strong motivation for doing the study, didn’t provide a testable hypothesis… Which isn’t a criticism of these papers, by the way. I’m not suggesting that they all should’ve been framed as tests of the IDH, or that they should’ve made more effort to generalize their results. I’m just describing how most of these papers cited the IDH, and Fox (2013)–as boilerplate or throat-clearing. Basically just nodding towards the IDH and what’s been said about it recently, before getting down to the specifics of the paper. Ironically, the papers that made the strongest use of their citations of the IDH, at least to my eyes, were the ones that agreed with Fox (2013). They used the lack of success of the IDH to motivate the need for an alternative theory of how disturbance affects diversity, and to motivate the need for applications or tests of that alternative theory.****
- Here’s the other thing that struck me: many of the citations of Fox (2013) were from papers in specialized journals (Geobotanica, Annals of Forest Science, Cold Regions Science and Technology…). Some of these journals are so specialized that not only hadn’t I head of them, but the big research university that employs me doesn’t even subscribe to them. Most of the remaining citations were from unselective journals like Plos One. And a couple of the exceptions were papers that weren’t even about the IDH. All this actually mirrors the citation pattern for Connell (1978). It’s a classic paper that’s massively cited—but most of its citations these days are in specialized, low-impact journals. Of those 581 citations it’s had since 2013, only 13% come from a long list of leading selective journals in general ecology/general biology/general science.***** And some fraction of that 13% are papers that aren’t actually about the IDH.
- The previous two bullets together suggest to me that maybe the IDH isn’t a zombie idea after all. Maybe the IDH is a ghost idea. An idea that, while not dead, isn’t sufficiently alive to have much effect any more. Ecologists writing for a broad audience about big, general ideas and reporting what they feel are major novel results mostly aren’t trying to test or further develop the IDH (there are exceptions). And ecologists writing for a specialized audience about disturbance-diversity relationships in particular systems mostly aren’t trying to test or further develop the IDH either. The IDH is just the big, general idea to which they give a passing nod before they turn their attention to the system-specific details of interest. So the IDH isn’t dead—but as best I can tell, it might as well be.
A while back, I used citation data to argue that the IDH remains a popular subject of active research—that it’s alive and well, even if I think it shouldn’t be. But now that I’ve seen how the IDH, and my critique of it, are cited, I’ve changed my mind on that. I think the IDH is still widely taught to undergrads, and I question whether it should be (depends on how it’s taught, I think). But for the reasons discussed above, and for other reasons, it now looks to me like a ghost idea as far as current research is concerned.
As always, looking forward to your comments. I’ve learned a lot from past comment threads on this topic.
*When I give talks on my blogging, and ask the audience who reads Dynamic Ecology, less than half the hands go up.
**”[P]recisely the same as if he’d bawled it up a rainspout in the interior of Afghanistan”, according to H. L. Mencken.
***I went with WoS because I didn’t want to have to sort through all the conference proceedings and other flotsam that Google Scholar indexes. You get the background research you pay for on this blog.
****Let me emphasize that I have no idea if that’s how the authors of these papers intended their citations of the IDH to come off. That’s just how they came off to me. That perhaps deserves a post of its own at some point: what is the introduction section of a paper for? I suspect there’s a range of views on this.
*****Science, Nature, Nature Communications, PNAS, Plos Biology, Proceedings B, Phil Trans B, Eco Letts, Ecology, Ecol Monogr, Ecol Appl, Am Nat, JAE, J Ecol, J Appl Ecol, Funct Ecol, Conserv Biol, Global Ecol Biogeogr, Ecography, Oikos, Oecologia, Ecosystems, Global Change Biol, AREES, TREE. I’m sure you could quibble with the composition of this admittedly-arbitrary list, but it wouldn’t change the results much. And before anyone asks, yes, this does seem to be a change in how Connell (1978) is cited. 40% of the 632 citations of Connell (1978) that occurred before 1990 were from the journals on my list. Even though several of those journals didn’t exist in 1978, or even before 1990.
I previously did a similar exercise with a paper critiquing a popular approach in phylogenetic community ecology. It wasn’t having much impact either.