Inspired by Mark Vellend’s recent guest post on subtle scientific biases in ecology, I got to wondering how often ecologists and evolutionary biologists still do “but see” citations. That is, you make a claim, cite some papers in support, but also write “but see” and cite papers that contradict your claim. Without thinking about it much, I figured that “but see” citations are becoming rarer. That increasingly, ecologists and evolutionary biologists are just citing papers that support their claims and not bothering to cite papers that cut against their claims. To check my hypothesis, I did a bit of searching on JSTOR.
Boy, was I wrong! Well, as best I can tell, I was extremely wrong. But maybe I’m wrong about being wrong? Anyway, for the details, read on.
I searched JSTOR’s “ecology and evolutionary biology” category, which covers 69 journals, for items (mostly journal articles, but also some book chapters) containing the phrase “but see”. I took care to search for “all content” , not just content to which I have access through my university library. To get a denominator, I did the same search for items in the EEB category containing the word “the”, because that was the only way I could figure out to get JSTOR to give me the total number of items in the EEB category. I did this for each of a number of years: 1970, 1980, 1990, 2000, 2005, and every year from 2010-2015. A few spot checks indicated that items returned by the search on “but see” did all contain that phrase.
Note that the number of items in the EEB category peaked in 2010 and exhibited a slow noisy downward trend after that. Which doesn’t seem like it can possibly be right, and makes me wonder if there’s some limitation of JSTOR’s database or search tools that invalidates this whole exercise.
It turns out that the proportion of items containing “but see” increased steadily from a very low level in 1970 up through 2010, and has been dropping slightly since them. Here are the numbers (sorry, too lazy to make a graph):
- 1970: 0.6% of EEB articles (and a few other items such as book chapters) contain “but see”
- 1980: 2.1%
- 1990: 5.7%
- 2000: 9.8%
- 2005: 11.2%
- 2010: 13.7%
- 2011: 12.7%
- 2012: 13.1%
- 2013: 12.5%
- 2014: 12.4%
- 2015: 12.2%
I’m not sure what to make of this. Changes over time in the proportion of items containing the phrase “but see” (steady increase up through 2010, followed by a slow noisy decline) mirror changes over time in the total number of items. So I’m worried that there’s some sort of artifact or JSTOR limitation here that I’ve overlooked. But taking the results at face value, my hypothesis going in was maximally wrong. By this measure, it’s more common than it used to be for authors to cite papers that disagree with the authors’ claims.
Now, part of that might be because papers these days just cite more stuff. (do they?) Maybe if you looked at the proportion of citations that were “but see” citations, rather than at the proportion of papers containing at least one instance of the phrase “but see”, you’d get a different result? I dunno, I kind of doubt that’s the whole story, but I could be wrong.
Now, it’s still not all that common for an EEB paper to include the phrase “but see”. As of 2015, only about 12% of EEB papers included that phrase. And I highly doubt that’s because only 12% of EEB papers in 2015 made any claims for which there is only mixed support. Of course, papers can discuss claims for which there is only mixed support without using the phrase “but see”. So I wouldn’t assume that only 12% of EEB papers ever discuss mixed evidence for any of their claims.
So, if you know of a reason why these results might be some sort of artifact, please tell me and I’ll update the post! But on the (risky!) assumption that these results are valid–that EEB papers really are more likely to include the prhase “but see” than used to be the case–I’m curious to hear how you interpret them.
Dynamic Ecology 
At least to me, “but see” is close to equivalent to “cf.”. Perhaps there’s a shift from using “cf.” to “but see” that explains some of the increase
Though Stephen Heard will tell you it is impossible to check this since cf. gets used incorrectly.
Yeah, I confess I don’t know exactly what “cf.” is supposed to mean! But I also feel like “cf.” has always been super-rare in EEB. Maybe I’m wrong about that? Somebody else will need to check because I’m swamped right now.
I don’t doubt that you’re wrong about EEB. Though the opposite is true in my field of physical oceanography in that “cf.” is about 10× more common than “but see” (based on a quick check from some papers I used in a recent post about <a href="LINK_URL“>word usage)
The cynical, depressing interpretation of the results in the post is that it used to be that EEB papers would discuss contrary or mixed results at length. But nowadays, they don’t bother with discussion of contrary results and just use “but see” citations instead. I have no idea if that’s right. To check, you’d have to read a whole bunch of papers very closely.
Simple but interesting analysis, thanks for sharing!
I do not use JSTOR so often, so I do not really know their inclusion criteria for journals/publications.
I only have a comment: “but see” publications could be also used the other way around. Suppose that the authors’ results support a ground-breaking discovery, that is against the currently established knowledge in the field, they would start by reviewing the “mainstream” publications (which their data are going to refute) and present “but see” publications as supporting evidence to their own work. Any opinions (or even data) on this?
Just offhand, based on my own experience, I think that’s a rare sort of “but see” citation in EEB. But I haven’t compiled data on the frequency of different kinds of “but see” citations.
A couple of potential hypotheses
1. when did we switch to mostly online peer-review, online articles, vs hard copy journals. I think after 2000, we’ve just got better at being able access articles and hence are more likely to find articles with conflicting views
2. peer review. How common was it to have 2-4 reviewers for a manuscript in the distant past, all aware of different literature? What % of “but see” citations often come from reviewers? Perhaps we’ve just gotten better at peer-review or have more reviewers. This might combine with hypothis one, if reviewers are also more aware of the literature, having more of them would create a synergistic effect.
I expected the increase from the title without reading the article. Interesting that you thought it would be declining. I’m not sure I followed why you thought researchers would be more likely these days to ignore articles with opposing views.
Hmm. Re: 1, looks like a steady, near-linear increase in use of “but see” from many decades ago up until 2010. No trend break in the early-to-mid oughts. So I don’t buy your #1.
Re: your 2, I’m pretty sure two reviewers/paper was routine in ecology by the 1970s? Certainly, it had been routine for a good while by the time I started grad school in 1995.
“I’m not sure I followed why you thought researchers would be more likely these days to ignore articles with opposing views.”
I was probably just being overly cynical and grouchy. Feeling that these days, people increasingly write papers to “sell” the reader on whatever story they want to tell.
Good point. 2nd attempt, both also likely to be garbage hypotheses but will throw them out there anyways.
1b. Increasingly, there is more attention paid to unethical sci pratices and retractions. This both explains the trend and your perhaps cynical hypothesis. We feel like like people are overly selling their papers now, but whats really happening is greater attention being paid to it. Kind of like how most people think crime is increasing, when it’s actually decreasing (due to crime occupying an increasing proportion of media reports).
2b. Are authors more likely to oversell in high impact journals [would be interesting to track “but see” citations in nature and science)? If so, I would guess that the proportion of low impact journals is increasing. In other words increased publishing in low impact journals is outpacing increased publishing in high impact publications (think this has to be true) and if folks are less likely to use but see in high impact submissions, maybe this could explain the result.
It would be interesting to know if the frequency of “but see” citations varies with journal impact factor.
“By this measure, it’s more common than it used to be for authors to cite papers that disagree with the authors’ claims.”
By _this measure_, yes. However, what I suspect you’re finding is merely linguistic change. There are other ways to cite papers that disagree with a claim. (See https://en.wikipedia.org/wiki/Citation_signal#Signals_indicating_contradiction for examples.) It could be that some ways have waxed and waned in popularity over the years. Language is not static!
You’d have to read a sample of earlier papers to see if and how they cited contradictory evidence in order to investigate further.
Yes, it would be interesting to also look at the frequency of “cf” and “contra” citations.