Note from Jeremy: this is a guest post from Kevin Lafferty, Western Ecological Research Center, U.S. Geological Survey
I hereby challenge you to help me redesign the scientific paper through a process called “Collaborative Independent Review”. But if you’ve already comfortable writing the traditional scientific paper, you’re probably not going to like it.
If you don’t like it, blame Andy Dobson. When Andy invited me to write a chapter for the new book Unsolved Problems in Ecology (Dobson, Holt and Tilman eds., PUP – check it out), he figured I would write about how everyone should think about parasites as much as I do. But I had been reading blog posts on Dynamic Ecology about how we do business as Ecologists (which means you can blame Jeremy, Brian and Meghan too). This got me more worried about ecologists than parasites. I became convinced we could get more return on investment in Ecology through better training programs, funding distribution, synthesis, publication models, and evaluation metrics. And so I wrote a chapter on A Science Business Model for Answering Important Questions. While writing, I kept remembering a 1979 paper called Ecology: A science and a religion, where one of my heroes, Paul Dayton, predicted that ecologists’ increasing focus on conservation would begin to undermine their scientific objectivity. This led me to add a section about reproducibility, which is what Jeremy asked me to blog about. Lots has been said about reproducibility in other disciplines, but I wondered if re-visioning how we write papers and how journals publish them was the key for ecology.
My nagging worry about reproducibility in my own work goes back to a paper that an undergraduate named Kimo Morris and I published in 1996 in Ecology. Kimo’s data showed a strong effect of a brain parasite on fish behavior and subsequent predation by birds. I really wanted to believe Kimo’s data were true. Audiences loved the story, which motivated me to tell it more. And, yet I harbored an insecurity that my own infatuation with his results might have led me to be less skeptical than I was trained to be. I should have repeated the study, but found convenient excuses not to (two kids, no funding, a good swell always on the horizon). Fortunately, as time passed, others looked into this host-parasite relationship in more detail, finding results largely consistent with what Kimo had found. No retraction needed.
What follows is excerpted from my above-mentioned chapter.
We don’t know the extent to which ecological results are reproducible, but concerns about reproducibility from other disciplines suggest this is a topic ecologists should think about. Whereas economics, psychology and biomedical research study humans and a few model organisms, ecologists study biodiversity in its entirety. For this reason, ecologists expect that a single study might not be general, and it is only after amassing many studies from many researchers on many systems do ecologists consider whether support for a hypothesis is general. Ecology is, by its nature, often not reproducible, and there is a tradeoff between ecologists replicating specific studies versus gaining insight from doing similar studies in different contexts. And that might be why progress in ecology sometimes seems like a random walk more than a stable attractor.
Although the goal for Ecology might not be reproducibility, ecologists should at least strive for transparent and unbiased data interpretation. Unfortunately, complex modern statistical analyses allow multiple interpretations, leaving it up to ecologists which results to report and emphasize. Increasing ambiguity is revealed by lower R-square values and higher P-values per paper over time. The desire to report something significant can lead authors to subconsciously report significant outcomes from multiple tests without controlling for multiple comparisons (p-hacking) and ecologists report more significant findings when they gather data with a preconceived hypothesis. On the other hand, the joy in reporting an unexpected finding leads to HARKing (hypothesizing after the results are known), which is encouraged by high impact journals that require authors to emphasize novelty and importance. Furthermore, under the current biodiversity crisis, it is harder to remain neutral and dispassionate about the systems ecologists study. Ecologists’ personal concerns for the environment can emphasize catastrophes, collapses and crises that attract readership, provoking calls for more careful analyses and sober interpretations. And with each Pruitt retraction, there is renewed reason to be skeptical about ecologists in general.
If scientists spend taxpayer money to generate irreproducible results, the public’s logical response should be to either withhold funds or demand a new process that emphasizes reproducibility. Ecologists increasingly acknowledge that reproducibility is important, and there is already a move among journals for transparency and openness guidelines that could help foster reproducibility by having authors adhere to citation standards, data transparency, code archiving, materials archiving, design transparency, pre-registering hypotheses and analytical methods, and replicating past studies. Some have argued that research institutions should implement Good Institutional Practices (i.e., rules, standards, documentation, transparency, blind assessment), but ecologists don’t often follow such practices even when they would be easy to implement. For instance, blind assessment helps researchers avoid bias, and is standard in clinical trials, but is not common in Ecology. Although no journal has adopted all transparency and openness guidelines, several have their own lists. For example, Nature Magazine has an 18-point checklist for Good Institutional Practices in its instructions to authors. Independent assessment could be extended into several other publication steps with the aim to reduce bias, increase specialization, and foster critical thinking. For instance, basing publication acceptance on sound hypotheses and methods rather than the significance or findings (as per the Public Library of Science journals scope) makes it possible to publish negative results, which helps reduce the file-drawer problem. In addition, a few journals embrace reproducibility by inviting repeat studies (e.g., F1000Research). But I don’t think this is enough.
Collaborative Independent Review is one way that funders, journals and scientists could implement a more reproducible paper. The process is collaborative in the sense that four independent teams and an editor author a paper together. The first step is for a Principal Investigator (PI) to propose the questions, hypotheses, predictions, and methods (including proposed analyses). The pre-registered proposal includes an Introduction and Methods, and suggests a target budget for the methods and analyses. Proposals receive double-blind panel review based on expected return on investment. Competitive proposals are revised according to panel review and then put out for bid on by (1) a lead technician (which can be the PI), and (2) a lead analyst (not associated with the PI). The technician receives half the funds up front to implement the methods and report the data. The analyst, who maintains independence by remaining anonymous until publication, blindly tests the a priori predictions and writes and illustrates the results, including appendices describing the analyses in detail. The analyst sends draft results to the PI and technicians for review. Once the three parties agree on the results, the PI submits the Introduction, Methods and Results to an editor who sends the sections to outside referees. In response to the referee reports, the PI, technician and analyst revise the Introduction, Methods and Results. The referees then write a collaborative Discussion about the Results, at which point the funder pays the award balance to all authors (PI, technician, analyst, referees, editor). All data produced in the project become available to the public at the publication date so others can repeat the analyses.
If you’ve read this to the end, I am curious if you hate this idea as much as I expect you will. I certainly have my reservations. In particular, Collaborative Independent Review could discourage the scientific creativity that generates new ideas and hypotheses when unexpected results occur. And all that reproducibility comes at the cost of time, expense, creativity and investigator control. Would you still be motivated to put in the long hours at the expense of not controlling how your data are analyzed and interpreted?
Can we do this now? The biggest obstacle to Collaborative Independent Review is probably engaging the funding agencies. But there should be little barrier to doing collaborative independent writing. If this sounds interesting, I’m looking for fellow ecologists to give this a trial run (sans the funding). Email me and let me know if you are interested in either 1) proposing a question and providing data, 2) blindly analyzing the data, 3) reviewing the results and writing the discussion. I’ll volunteer to act as editor, and find a journal.
Any use of trade, product, or firm names in this publication is for descriptive purposes only and does not imply endorsement by the U.S. government.