A few months ago I posted about my side project asking whether there’s a systemic tendency for published ecological effect size estimates to decline in magnitude over time (the so-called “decline effect“). Answering that question involved compiling every published effect size estimate I could get my hands on, from every ecological meta-analysis I could find.*
The data compilation is done now. I now have a 14 MB csv file containing over 114,000 effect size estimates from 470 ecological meta-analyses. This is close to (not quite) a census of all ecological meta-analyses and their effect sizes published from 1991 through the spring of 2020. I also have the sampling variance for every effect size estimate, and some other bits of information: the publication year of every meta-analysis, the publication year of every effect size estimate, an identifier for the original paper in which every effect size estimate was originally published, a (crude) descriptor of the response variable for each meta-analysis (e.g., “abundance”, “diversity”, “various”…), and the effect size measure used (usually Hedge’s d or g, the log-transformed response ratio, or the Fisher’s z-transformed correlation coefficient).
Also this week: ASN award winners, working from home vs. scientific fraud (?), WEB Du Bois vs. scientific fraud (?), cliches of scientific writing, crowdsourced species interactions database, and more
Inspired by Mark Vellend’s recent guest post on subtle scientific biases in ecology, I got to wondering how often ecologists and evolutionary biologists still do “but see” citations. That is, you make a claim, cite some papers in support, but also write “but see” and cite papers that contradict your claim. Without thinking about it much, I figured that “but see” citations are becoming rarer. That increasingly, ecologists and evolutionary biologists are just citing papers that support their claims and not bothering to cite papers that cut against their claims. To check my hypothesis, I did a bit of searching on JSTOR.
Boy, was I wrong! Well, as best I can tell, I was extremely wrong. But maybe I’m wrong about being wrong? Anyway, for the details, read on.
Note from Jeremy: this is a guest post from Mark Vellend.
************
As a Ph.D. student (long ago) I was interested in studying ecological consequences of human land use that might last a really long time. One of my Ph.D. advisors, Peter Marks, and colleagues had reconstructed the history of each bit of forest in Tompkins County, New York, providing a superb template for asking how biodiversity varies between forests never cleared for agriculture (primary forests) and those growing up on abandoned agricultural land (secondary forests) starting at different times.
From that point of departure, I then made a critically important decision. I am a plant ecologist, but rather than survey the entire plant community in different forests, I decided to focus on “forest herbs”: herbaceous species whose persistence seems to depend on forest cover in the sense that their main and sometimes exclusive habitat is the forest understory. As a group, forest herbs tend to be very long-lived, most lack a persistent seed bank, and a great many lack mechanisms for seed dispersal beyond a few tens of meters. (Many have seeds dispersed by ants.) As such, they are expected to be quickly eliminated when a forest is converted to agriculture, and very slow to return when the forest grows back.
Good decision, right? Well, it depends on the question. If I wanted to ask, “are there some species whose (meta)populations require many decades to recover from historical disturbance?”, this was a wise choice. But if I wanted to ask if plant biodiversity in secondary forests is lower than in primary forests, my decision created an obvious bias. I deliberately chose the subset of the flora most likely to remain underrepresented in secondary forests for a long time. Indeed this is exactly what I found, as have many studies of a very similar nature in temperate forests. In a review, we said: “For centuries after agriculture has ceased, plant communities on abandoned agricultural lands remain impoverished in herbaceous species characteristic of uncleared forests.”
The problem is that the quote above is an answer to the first question (about a particular set of species), but it is routinely read as an answer to the second question (i.e., evidence of a negative effect of human actions on biodiversity). If we take a broader view of the landscape and how human activities have likely influenced plant biodiversity, the story is rather complex and nuanced. While the diversity of forest herbs might be lower in secondary forests, the diversity of all plants might be just as high or even higher in secondary forests. We have found that the composition of different secondary forest patches tends to be more similar than among different primary forests, suggestive of biotic homogenization caused by land use. However, if we take primary forests as representing what the landscape looked like prior to European influence, then beta diversity in the current landscape is almost certainly much greater than it was previously, given greatly increased environmental heterogeneity (= biotic differentiation). The cartoon version of these and other results (below) is modified from a recent paper in a philosophy journal that I wrote on this topic (same journal issue as Jeremy’s philosophy paper!).
Despite this highly context-dependent picture of how biodiversity has changed under human influence in a commonplace landscape like this one, our results for forest herbs appear consistent with the idea of a “biodiversity crisis” – a powerful and dominant narrative in conservation biology, a field with which many (most?) ecologists feel a strong alignment. The existence of our science is often justified by how our research can help fight the crisis, and so we are quite adept at finding bad news stories. My experience is that ecology undergraduates arrive at university already with a sense of certainty that any and all changes in nature caused by human actions are for the worse, and so essentially the conclusion sections of term papers and research projects can be written before any research is done: people are screwing up nature, probably more than we thought, and we’ve got to do something about it. In my view, reality is much less straightforward, and far more interesting.
The main point of telling this story is to argue that I think we have good reason to be concerned that as ecologists, we make decisions about what to study, how to study it, what to report, and how to interpret the findings, that seem likely to emphasize some conclusions more than others. As a collective, ecologists tend to share a set of values that might create an important source of bias in our science. If we each had individual biases in different directions – representative of the broader society – we might hope that they would cancel one another out and that a fairly accurate view of reality would emerge. I assume that most readers would agree that ecologists bring a set of values to the table that is not representative of the distribution of values found in the societies in which they live. (As one example, just think about the distribution of votes among political parties wherever you live: ecologists vs. a random sample. Significant differences? For social scientists in the U.S. the difference is massive.)
The other point in telling this story is to draw attention to the article mentioned above, in a philosophy journal, that explores this topic in greater detail (not going to bother hiding it): “The behavioral economics of biodiversity conservation scientists”. I present additional examples where values entered into ecological science, and I draw on the literature from conservation and environmental science about common values shared by ecologists, from the social sciences on the relationship between values and science, and from psychology (the “behavioural economics” part) on some of the reasons we might have developed preferences for how nature ought to look (e.g., more like it did centuries ago than it does now). I hope that some of you find it thought-provoking and worthwhile and that you’ll let me know what you think.
I think that as ecologists it would do us well to think more deeply about the values we bring to our science, how it might influence what we conclude about the world, and whether there are consequences for our credibility and our hopes to influence policy. The example I laid out here is, in some senses, subtle, with results represented accurately in publications, but with an overall message that emphasizes the “people are bad for biodiversity” message to a degree that seems misaligned with what has actually happened in the landscape. Just as a small selection coefficient and many generations in a large population can lead an allele to fixation, so might small biases in one direction, among many researchers, over many years lead us to a skewed view of the world. Is it happening? And if so, is there anything to be done?
Many retracted papers continue to be cited years after they’re retracted. Some are even cited about as often after retraction as before–i.e. the retraction appears to have had no effect on how often they’re cited. See here for discussion and links to some data. Which is a pretty depressing commentary on scientists’ citation practices.
But #pruittdata is an unusual case. Approximately a year ago, very serious concerns were raised about anomalies in the data underpinning dozens of papers co-authored by Jonathan Pruitt. Those concerns were widely reported in the scientific press and even made it into some newspapers, and were widely discussed by scientists on social media. Numerous papers of Pruitt’s have now been retracted, corrected, or subjected to Expressions of Concern (EoC), and various investigations are still ongoing. How have citing authors reacted to #pruittdata? Are papers co-authored by Jonathan Pruitt still being cited? And are citing authors differentiating between papers for which Jonathan Pruitt collected data, and papers for which he did not collect data (e.g., review papers)?
“A society—a civilization, if you like—is a hard thing to hold together. If you live in an agrarian society, and you have only stone, wood, and bone for tools, and you are on the western edge of Europe, few times are harder than the dead of Winter. The days are at their shortest, the sun is far away, and the Malthusian edge is right in front of you. It’s no wonder so many religious festivals take place around the solstice. Here were a people, more than five millennia ago, able not only to pull through the Winter successfully, but able also to build something like a huge timepiece to remind themselves that they were going to make it.”
Click through and read the whole thing. I link to it every year. It seems especially apropos this year.
I am really looking forward to this year being over. I don’t feel up to doing our usual navel-gazing year in review post this year. So here’s some Christmas music I like. Come for Eartha Kitt and The Ukulele Orchestra of Great Britain, stay for The Ramones, Judy Garland, and a surprise appearance by a Nobel Prize winner. Thanks for reading and commenting this year, hope you and yours are safe and well, see you in 2021.
