Inspired by Mark Vellend’s recent guest post on subtle scientific biases in ecology, I got to wondering how often ecologists and evolutionary biologists still do “but see” citations. That is, you make a claim, cite some papers in support, but also write “but see” and cite papers that contradict your claim. Without thinking about it much, I figured that “but see” citations are becoming rarer. That increasingly, ecologists and evolutionary biologists are just citing papers that support their claims and not bothering to cite papers that cut against their claims. To check my hypothesis, I did a bit of searching on JSTOR.
Boy, was I wrong! Well, as best I can tell, I was extremely wrong. But maybe I’m wrong about being wrong? Anyway, for the details, read on.
I searched JSTOR’s “ecology and evolutionary biology” category, which covers 69 journals, for items (mostly journal articles, but also some book chapters) containing the phrase “but see”. I took care to search for “all content” , not just content to which I have access through my university library. To get a denominator, I did the same search for items in the EEB category containing the word “the”, because that was the only way I could figure out to get JSTOR to give me the total number of items in the EEB category. I did this for each of a number of years: 1970, 1980, 1990, 2000, 2005, and every year from 2010-2015. A few spot checks indicated that items returned by the search on “but see” did all contain that phrase.
Note that the number of items in the EEB category peaked in 2010 and exhibited a slow noisy downward trend after that. Which doesn’t seem like it can possibly be right, and makes me wonder if there’s some limitation of JSTOR’s database or search tools that invalidates this whole exercise.
It turns out that the proportion of items containing “but see” increased steadily from a very low level in 1970 up through 2010, and has been dropping slightly since them. Here are the numbers (sorry, too lazy to make a graph):
- 1970: 0.6% of EEB articles (and a few other items such as book chapters) contain “but see”
- 1980: 2.1%
- 1990: 5.7%
- 2000: 9.8%
- 2005: 11.2%
- 2010: 13.7%
- 2011: 12.7%
- 2012: 13.1%
- 2013: 12.5%
- 2014: 12.4%
- 2015: 12.2%
I’m not sure what to make of this. Changes over time in the proportion of items containing the phrase “but see” (steady increase up through 2010, followed by a slow noisy decline) mirror changes over time in the total number of items. So I’m worried that there’s some sort of artifact or JSTOR limitation here that I’ve overlooked. But taking the results at face value, my hypothesis going in was maximally wrong. By this measure, it’s more common than it used to be for authors to cite papers that disagree with the authors’ claims.
Now, part of that might be because papers these days just cite more stuff. (do they?) Maybe if you looked at the proportion of citations that were “but see” citations, rather than at the proportion of papers containing at least one instance of the phrase “but see”, you’d get a different result? I dunno, I kind of doubt that’s the whole story, but I could be wrong.
Now, it’s still not all that common for an EEB paper to include the phrase “but see”. As of 2015, only about 12% of EEB papers included that phrase. And I highly doubt that’s because only 12% of EEB papers in 2015 made any claims for which there is only mixed support. Of course, papers can discuss claims for which there is only mixed support without using the phrase “but see”. So I wouldn’t assume that only 12% of EEB papers ever discuss mixed evidence for any of their claims.
So, if you know of a reason why these results might be some sort of artifact, please tell me and I’ll update the post! But on the (risky!) assumption that these results are valid–that EEB papers really are more likely to include the prhase “but see” than used to be the case–I’m curious to hear how you interpret them.