Does ecology need more criticism of the literature? If so how?

This post has evolved substantially over its writing. It started from a good post over on EEB and Flow by Marc Cadotte arguing that ecology needed a more robust culture of critique to weed out bad papers, and arguing that comments/critiques to the journals that published the original papers was an important way to do this. Despite strongly agreeing with the first part, I instinctively disagreed with the later part. (And have been thinking about critique letters a lot lately in my role as Editor-in-Chief at an ecology journal just as Marc has)*. But unpacking why I don’t like critique letters has led to a lot of musings on how ecology works, how the human mind works, and my own answer to the specific question of how best to steer the field away if you see a bad paper. And just maybe along the way I stumbled on a strategy or two for killing zombie ideas!

Lets start with the premise that there are bad papers that get through peer review and published. This is undeniably true. And I am clearly on record as thinking ecology is not advancing as quickly as it could because of our inability or unwillingness to declare ideas as decisively rejected and move on (just e.g. “Is ecology idea free“, many posts on the need for prediction not to mention Jeremy’s zombie ideas posts most of which I agree with). There even was an “Ask us Anything” question on how to respond to bad papers once that we responded to.

More data-driven, there is actually good (but depressing) bibliometric research on this.  Banobi et al identified 7 “prominent” examples where an original paper was wrong (in their case all making management-relevant claims) that were decisively rebutted. The original papers continued to be cited at very high rates after the rebuttals were published. The original papers continued to be cited 17x as often as the rebuttals, and 95% of the citations to the original paper remained positive to the original paper, rarely acknowledging the rebuttal.

My favorite anecdotal example of how hard it is to kill off an idea is Robert MacArthur published the broken stick model of species abundances in 1957. In 1966 MacArthur pleaded “Let us hope these comments do not draw additional attention to what is now an obsolete approach to community ecology, which should be allowed to die a natural death.” In short he pretty vehemently trashed his own theory as being obsolete. Yet 50 years later, you can still find new papers being published using the broken stick model (and the answer still remains it doesn’t work very well).

There are a lot of places to go with this devastating problem in ecology ranging from questioning the whole culture, to pushing philosophy of science, to critiquing wish-washy statistical inference methods, to arguing for a more predictive approach (you could argue Jeremy and I do little more than natter on about this problem!). Here I want to step away from the big picture and tackle this at a very small-grain, tractable level. If you see a “bad” paper, how best to respond? 

First, I think you have to unpack what it means to be a “bad” paper. I can think of dozens of ways in which a paper can be bad (which I’ll define here as needing overturning).  A few are:

1. Outright undeniably wrong
   1. Fraud
   2. Honest mistake (usually a data error, coding error or math error)
2. Not outright wrong, but not good
   1. Of limited generality (e.g. worked in one organism or site but not repeated in most other studies done afterwards)
   2. Assumptions that turn out not to be true (but weren’t known untrue when the paper was written)
   3. Statistics are “flawed”
   4. Paper doesn’t fully achieve goals
   5. Paper overinterprets results
   6. Assumptions too simplistic
   7. Vague, hand-wavy models
   8. etc., etc.

So, what should happen if a paper is bad? I would argue pretty strongly that it depends on which category of bad. If it is in group 1 (fraud or honest mistake), then a retraction or correction should be published. Period. Highlighting the error in a critique letter and then (usually) having a rebuttal by the author does not serve science well. It also does not follow the COPE (Council of Publish Ethics) guidelines that most ecology journals subscribe to and are bound to follow. Something stronger is needed (preferably with the authors own agreement). All journals have corrections or corrigendum. They also all have retractions. There is also increasingly a category known as “statement of concern” published by the editors when the author hasn’t come along. All of these are stronger than a critique letter. In a very important way. Somebody a year or two later who cites the original paper without acknowledging a corrigendum or a retraction or a statement of concern is committing malpractice and likely will be criticized by reviewers and required to change. You can’t say the same thing if it just has a critique letter about it. So I think category 1 bad papers need something stronger than a critique letter. That said, we should never lose the distinction between a 1A paper (fraud) which deserves all the reprobation it gets and a 1B paper (honest mistake) for which authors should be praised for correcting.

Category 2 papers are much harder. And the vast majority of “bad” papers are in this category. What exactly should be the goal if there is a category 2 “bad” paper? Such papers are kind-of-sort of wrong but not black-and-white wrong. They’re in a grey zone. A person is not crazy to keep citing the original paper (but nor is a person crazy for disagreeing with it). A lot depends on what you took away from the paper and how you cite it (because some parts of category 2 papers are valid and some parts are wrong). Sousa’s classic paper on IDH (intermediate disturbance hypothesis), clearly got the basic empirical existence of the pattern (and attribution to a form of disturbance) in his system right, but his conclusions about mechanism (“delaying competitive exclusion”) wrong.  If a paper is not wrong enough to be corrected or retracted, you are inherently in the zone of opinion where people can legitimately read the situation differently. Like most of the zombie ideas Jeremy talks about, the zombie idea of the intermediate disturbance hypothesis can be decisively rejected in one specific form (mechanism of “delaying competitive exclusion”) but not in some other forms. 

So what should you do if you are bugged by a Category 2 bad paper? What is the best route for science to advance? Well, obviously you can just ignore it and let somebody else take it on. That can be a very a valid choice, but for science to advance, somebody has to take it on, and lets assume for the moment you feel called to take on paper X. What then?

I can think of three approaches**:

1. Write the editor to prove it is so wrong it belongs in category 1 and should be retracted/corrected
2. Write a “letter to the editor” style critique letter
3. Identify the root issues (i.e. 2A vs 2B vs 2C etc in the list above) and address them in a full length opinion or research paper

#1 is a very high bar and should be. To retract a paper you need absolute evidence.But go for it if you think you can do it. But by definition you can’t do that for the much more common case of a category 2 paper. So really it comes down to #2 vs #3. I’m going to argue that #3 (stand alone paper) is a way more constructive option than #2 (critique letter).

Can you name a critique letter that changed your mind on a paper? I can’t. The debates on null models and on Diamond’s checkerboard island patterns were amusing and somewhat enlightening to read but their length (stretched over years, and still debated to this day) pretty much prove the point that they never solved the debate.

I think there are good reasons why critique letters often aren’t impactful and decisive:

1. Critique letters and their rebuttals have a real tendency to turn into two trains passing in the night that never talk to each other.
2. By definition critique letters are very short and not a place to bring to bear extensive new data or ideas.
3. They must be written entirely reactive to the original paper, not  heading off into a new synthesis or direction nor usually even responding to a suite of papers.
4. They typically only critique a subset of the paper which means most people will walk away remembering that the original paper is partly right and partly wrong.
5. The bar for novelty and importance is often lower than the bar for regular papers

All of these reasons may in part explain my experience as both an editor and an author of papers with critique letters submitted about them that a fairly high fraction (75%?) of all critique letters are rather weak. Many raise rather debatable points of analysis (e.g. data is maybe not normal enough so a generalized linear model is needed which only slightly changes the conclusions, but data doesn’t fit the proposed alternative distribution very well either – and yes this is a recent example for a paper I am a coauthor on) or concerns that the discussion overstates the conclusions compared to what is justified by the data (something the reader can form their own opinion on without a critique letter). Moreover, good fraction of the critique letters submitted to a high profile journal like Science or Nature strike me as largely opportunistic (i.e. based on the belief, mistaken in my experience, that it is easier to get a critique letter into Nature or Science than an original research article). So, in short, I haven’t seen a lot of really important, decisive critique letters.

On the other hand, I think full-length stand-alone papers may not stop the original paper from getting cited (as Banobi et al showed pretty clearly) which is a cultural flaw in ecology. But I do think they can very much influence the debate and overall direction of the field. I think this is true for a number of reasons:

1. The bar on importance and novelty is higher for a stand-alone paper which I think encourages more effort and filters for truly compelling work
2. The higher word counts allow one to more carefully build a case, bring in simulations and models, etc.
3. One can make a general point, not just react to a single paper and generality is good
4. Most importantly one can frame things in a positive, constructive, field advancing way rather than a reactive negative, tearing down way.

I think that last point is vital. Advancing science, not reversing science is more likely to be appreciated, remembered and influential for lots of reasons rooted deep in human psychology. Erasing an idea is impossible. If I instruct you not to think about clowns with balloons it is not going to work very well. Moving people to a new idea works well. If I give you a vivid image of elephants on the Serengeti, you are going to forget about clowns with balloons even if you were intensely thinking about them a minute ago. And much as we might like to think scientists dwell in a more rationalistic domain, we don’t. People need a new idea to move onto, not a suggestion than an old idea should be thrown away. And this fits the full length paper better than the critique letter.

Similarly putting a new cohesive idea in the literature is much more likely to be remembered than pointing out that somebody else’s paper is partly wrong (which paper isn’t partly wrong …?). Critiquing another paper is effectively carving in bas relief – gouging away a few bits but most of the paper will remain standing in people’s minds. Although the scientific method may be framed around falsification, the human mind works differently (for good reason – we should not eliminate an estimate of risk because it failed to materialize once). So if you’re goal is to overturn a paper, you should assume up front you will only overturn a piece of the paper.

There are plenty of examples where stand alone papers had impact when critiques did not. Neutral theory received multiple criticisms of type 2B and 2F above (invalid assumptions) that had zero impact. Papers by myself, Dornelas et al, and Gilbert and Lechowicz and several other early papers that made a methodological point that we needed to compare neutral theory predictions to alternatives (e.g. lognormal or environment vs dispersal) significantly tempered the way people thought about neutral theory. And, although I don’t like it, neutral theory hung around for several years despite these rejections until the idea of niche-neutral continuum expressed in several papers came along to provide a viable alternative. Gilbert & Bennett’s paper on using variance partitioning between environment and space has been far more successful as a stand alone comprehensive analysis than it would ever have been as a 1000 word critique of a single paper using this method (as Jeremy has pointed out). I hope that the advanced analysis of Grace et al 20016 will help us move on from the simplistic productivity-diversity idea. Most of these examples serve to highlight, though, that while advancing our methodological understanding is useful and serves as an example of how a new idea can replace an idea better than a critique, these examples also illustrate it is hard to really kill something off until a clear (equally simple) alternative explanation is available.

Returning to Banobi’s work, two things struck me as I spent some time looking at their examples in detail. First the rebuttals were always poorly cited in comparison to the exciting (but wrong) claims. But the stand-alone rebuttals that put a new, different answer out there were cited about 10x as often as the critique letters that were written directly in response to the original paper (many of the critiques were only cited 10 times when the main paper was cited 1000, but many of the standalone rebuttals were cited over 100 times). Second, although the rebuttals faired poorly in the metric of citations and in the metric of killing off the original paper, there is evidence they DID have an impact on the conversation. A detailed followup was done by Branch in 2013 of one case study in the Banobi paper (Wurm et al’s 2006 paper on fisheries collapse by 2048). Banobi showed that many of the citations of the rejected claim (projected dire state of the fisheries in 2048) were either rather naive or the paper was being cited in a rather generic way (e.g. fisheries are declining so my research question is important). Specific citations of the specific 2048 claim were in fact largely prevented by the rebuttals and the more sophisticated the authors, the less likely they were to repeat the specific claim that had been rebutted. So these paper-length independent rebuttals did influence the discussion materially.And the citations of the original paper were very often to parts of the paper that hadn’t been rebutted.

Is there any place for critique letters? Sure. But when they exist, the one thing I feel very strongly about critique papers is that stringing out the dialogue through a series of correspondences separated by months or years and hundreds of pages in the journals is not helping the reader very much. So it is incumbent on editors to get the conversation all published in one place back-to-back. And I personally believe correspondence need to rise to the same editorial standards of novelty and importance as original papers (which they rarely do).

To summarize, truly wrong papers need something stronger than a critique letter (i.e. a correction or retraction). Critique letters often seem to me to be indecisive (partly because the flaws in the original paper are rarely truly fatal to the entire paper and partly due to the limits of critique letters). Really strong critiques that are going to have a chance of changing the field are worth standing in their own right as a paper. And if you really want to kill off an idea, pay attention to human psychology and: 1) give the reader (and the field) a replacement, and 2) recognize critiques only invalidate parts of papers, not whole papers, in people’s minds.  A stand alone paper that presents a positive, coherent alternative is much more effective based on psychological principals.

What do you think? Does ecology need more rejections of published literature? How should this best be done? Do you read critique letters? Do you find stand-alone articles to be more decisive rebuttals?

---

* although they are in no way responsible for all the opinions in this blog post, I worked closely with the Deputy EiCs at GEB, Maria Dornelas and Richard Field, in developing our journal’s approach and my thinking benefited greatly from their thoughtfulness on the topic.

**or go blog about it. I am not touching blogs here, but I think blogs can actually be a rather effective tool for starting a conversation about the merits of a paper or idea in a format that is lightweight but rapid and with a permanent record.