Should old or superseded papers ever be retracted?

In a recent linkfest, I linked to a story about a 2014 Nature paper on human genetics that subsequent work showed to be incorrect. My understanding is that subsequent work used different, better statistical methods than the 2014 paper, showing that the 2014 paper’s statistical analysis doesn’t actually support the paper’s scientific conclusions. The 2014 paper has now been retracted, at the request of all but one of its authors. The holdout author agrees the paper is incorrect, but argues that not all incorrect papers should be retracted. As I understand it, the holdout author argues that papers should only be retracted if they’re flawed for some reason, not because they’ve been superseded by subsequent work that’s based on improved methods and/or better data.

I don’t want to debate whether this specific paper should’ve been retracted or not; I don’t know enough about the case to have an opinion. But the broad issue is interesting and worth discussing, I think. Should papers be retracted if they’re undermined by subsequent work, even though we had good reason to think them solid at the time they were published? There’s clearly disagreement about this issue, even among collaborators! And anecdotally, I have the sense that views on this issue are shifting, perhaps because of a generational divide. I feel like more senior scientists believe–even hope!–that all of today’s work will be superseded eventually, that that’s just scientific progress. On that view, it seems pointless at best to go back and retract all superseded papers. Rather, it’s the job of every professional scientist to know the relevant literature, and so know (say) that nobody should use the now-superseded method proposed by Smith & Jones (1985). Against that, one could argue that scientific thinking has too much inertia, that science’s vaunted self-correction processes are just too slow. Maybe science would actually progress faster if we were quicker to scrub the scientific record clean of any and all superseded papers.

One could also imagine other views intermediate between those two extremes. For instance, one might take the view that, once a paper is too old, there’s no longer any point to retracting it. A bit like how various crimes are subject to a statute of limitations in many jurisdictions. Or, one might take the view that, if the authors of a now-superseded paper want to retract it, they should be able to do so. After all, fiction authors sometimes repudiate their own work, even if it was widely acclaimed at the time it was published. Why shouldn’t scientific authors have that option?* And I’m sure there are many other possible views I haven’t sketched.

So here’s a short poll! Tell us: Should old or superseded papers ever be retracted?

*Not a rhetorical question! There might be good reasons why scientific authors–or fiction authors!–shouldn’t have that option, at least not in all circumstances. For instance, the linked article notes that most of Franz Kafka’s work only exists today because Kafka’s editor refused Kafka’s request to destroy it. There’s surely a case to be made that Kafka’s editor was right to refuse Kafka’s request. As a (hypothetical) scientific example, in the fictionalized biopic Creation, Charles Darwin offers his wife Emma the chance to burn the manuscript of the Origin of Species. Emma doesn’t burn it, which was surely the right call. So, are there circumstances in which a scientific journal ought to refuse an author’s request to retract a paper? I feel like there are, though I’m not sure I’d be able to list them all if you asked me too. There may be connections here to debates over whether there is a “right to be forgotten.”

32 thoughts on “Should old or superseded papers ever be retracted?

  1. Although I think it’s an interesting topic for discussion, if ever there was a general policy about retracting ‘superseded’ papers I think that it would be unworkable in practise. Not only would it lead to a huge amount of wasted time better spent on actually doing science, but what about papers in which the conclusions were wrong but the underlying data are sound? Are those data to be lost forever? My view would be to let the literature speak for itself unless there are sound reasons for retraction, e.g. fakery.

    Just before I read your post I looked at the ToCs for the latest PNAS, and this jumped out at me:

    Letters:
    No evidence of SARS-CoV-2 reverse transcription and integration as the origin of chimeric transcripts in patient tissues (Parry et al.)

    Response to Parry et al.: Strong evidence for genomic integration of SARS-CoV-2 sequences and expression in patient tissues (Zhang et al.)

    So here’s two groups of scientists arguing over the same data and methods and coming to opposite conclusions. Which we know happens all the time. Now fast forward 10 years and perhaps it will be clear which of these groups is correct. But who is going to adjudicate? Which scientists have the time to do that in their own fields?

    As far as I can tell from Google Scholar there was only one Smith and Jones paper published in 1985: https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/JC090iC01p00869

    Let’s hope that they don’t take it personally 🙂

  2. What is the age distribution of paper citations in books providing general coverage of a field, written by very experienced authors?

    Here is an example from software engineering:
    http://shape-of-code.coding-guidelines.com/2020/01/12/the-dark-age-of-software-engineering-research-some-evidence/

    A lot more papers are published in successive years, so some adjustment is needed, plus methods get superceeded (batch processing used to be a big thing in the 60s and 702, and is now coming back via cloud computing).

    The number of papers returned by search engines starts to significantly drop off doing back through the 1990s. Not being an academic I don’t have access to gated material.

    Here are citation counts for a book written during the rise of search engines:
    http://shape-of-code.coding-guidelines.com/2015/10/05/citation-patterns-in-my-two-books/

  3. An addendum/errata which states why the old paper was flawed/erroneous in the light of new evidence/analyses might be the best – especially if it can be directly attached to the old paper somehow (should be possible in the era of digital publication). Possibly, there might be data in the old paper that can be useful even if the analysis is flawed.

    • This is a good argument for making data sets citable (e.g. GenBank, Dryad) independent of the paper that originally analyzed them.

      • Yes, it is – especially when the data sets can be searched for independently from the paper. But the paper itself gives much more information about the study than what is usually included in meta data for the data set. Given that some (digital-only) publishers remove the PDF when the paper is retracted, that information risks being lost.

    • Personally I think much too much attention is paid to retractions and not enough to corrigenda. Unless outright fraud is involved, I think generally science is better served by correcting/updating erroneous literature with a clear explanation of what went wrong and how it changes the results than just a binary retraction with no explanation.

  4. This is a really interesting question in general, and I wonder how much of it depends on a particular field (or even a particular field at a particular point in time). In mathematics, for example, there are lots of different levels of rigour one can use, depending on basic axioms and things (and some are simply incomparable, such as taking inconsistent axioms as a starting point). There are examples of ‘results’ which work for most reasonable things, but which were later shown to be wrong by pathological examples (e.g. continuous functions which are nowhere differentiable essentially ‘disproved’ quite a lot of 19th century arguments). My impression is that very few papers in mathematics are retracted, even when this happens, and it is only when there are technical errors which change the spirit of the result, rather than just technical problems in the precise statement of assumptions.

    As an example, both Benoit Mandelbrot and Louis de Branges have published several ‘incorrect’ results (see also Andrew Wiles’ first proof of Fermat’s Last Theorem, though I don’t think this was published). But these papers led others to correct their proofs, often by slightly restating the results or modifying the methods. I think retractions do happen, but in many cases instead people will use it as an opportunity to either publish a correction or a counterexample. It is not always clear either – de Branges has several versions of a claimed proof of the Riemann Hypothesis, so despite having done quite a lot of great work/being very respected, people now choose to ignore a lot of his work. From Wikipedia: “de Branges’ apparent estrangement is not an isolated case, but he is probably the most renowned professional to have a current unverified claim.” NB: This is all exacerbated a bit by mathematicians sometimes eschewing journal publication for treating arXiv/preprints as the real claim.

    Also related: This old post on the sociological aspects of the claimed proof of the ABC conjecture, which does now appear as published in a journal https://www.galoisrepresentations.com/2017/12/17/the-abc-conjecture-has-still-not-been-proved/ .

    Sorry for the long post – I just find the whole issue interesting!

    • That’s a great example. I’d be curious to hear whether any mathematicians think their field ought to start retracting (or formally correcting) old proofs that turn out to have mistakes or limitations that weren’t recognized at the time. I’m guessing that hardly any mathematicians would think that’s a good idea, but obviously I don’t know.

    • For an analysis of how mathematical challenges work in the long view, I cannot recommend highly enough Imre Lakatos’s “Proofs and Refutations”.

  5. I wish I knew more about the genetics paper to make an informed opinion on the matter. But it seems unreasonable to retract a paper for using outdated or even wrong statistics. Take Jared Diamond’s classic work on species co-occurrences of New Guinean birds. Connor and Simberloff (1979) convincingly showed how Diamond’s lack of null models made the inferences from this work incorrect. But nobody would argue that Diamond’s original paper should be retracted. It’s an important part of the literature on this topic. If conclusions from an old paper become outdated in light of new evidence, then we would simply expect that paper to be cited less frequently over time.

  6. Oddly enough, aside from clear scientific fraud, there should be a reverse rule for the statute of limitations. So the older a paper is, the more likely we should retract it. And that’s because new information can always itself be easily turned over. If someone did something via method 1, then method 2 comes out and shows a different result which invalidates method 1 entirely, who is to say that method 3 will not come out and show that method 2 is also “wrong” or incomplete and validate aspects of method 1? In that case, do we take back the retraction? We don’t necessarily know the impact of the work until decades or centuries later. For example, we know phrenology is clearly wrong, and as such there is no need to have it as citable literature anymore expect to say “This is a clearly wrong and pseudoscientific method.”

    • Phrenology is a good example. Should every phrenology paper be formally retracted? I’m inclined to say why bother. It’s not as if anyone believes in, or even cares about, phrenology any more, except out of historical interest.

      I agree with you that, the older a paper is, the more likely it is to have been superseded by subsequent work. But the older a paper is, the more likely it is the paper’s already been forgotten. And the more likely it is that everyone already knows the paper’s been superseded. So what is formal retraction of such papers supposed to accomplish? Is it a purely symbolic gesture? Is it a way for journals to give a “costly honest signal” that they really, really care a lot about preserving the integrity of the scientific record? Thereby hopefully nudging others to care as well, and so propagating good professional norms? Something else?

  7. It’ll be hard to maintain a good grip on the history of science if we delete papers. I second the idea above, that at the request of authors, a note be attached to the existing electronic files stating their changed view of their publication. But the original paper must be left in place so that everyone can understand the evolution of that particular arm of science. For papers later discovered to be outright frauds, just put a big red X across each page with an explanatory note, but don’t remove the paper. Let it serve as a warning to others, and as a statistic for historians.

    • Best practice for retracted papers is indeed to mark them as retracted but not delete them from the website on which they’re hosted. As far as I know, most major scientific publishers follow that best practice. And as far as I know, no one who advocates retraction of old or superseded papers is advocating for those papers to be erased from the internet.

  8. https://retractionwatch.com/2016/04/27/biologist-critiques-own-paper-journal-retracts-it-against-her-wishes/

    ” … that constant fears of retraction could make academics reluctant to admit past mistakes” I think this is an important point.

    Also, the general purpose of science is to correct mistakes and move forward. In physics, retracting Galileo, Newton, etc. would essentially wipe out much of the history of science. Nothing could explain how we got to the moon!

  9. I suppose that this whole question of retraction is a consequence of that there is so much information out there, so that we are unable to read and evaluate all papers. Except for the fraudulent papers, with made up data, a well written method section allows me to evaluate whether the conclusions are valid. If not, I just throw the paper in the waste bin. If I were to call for retractions of all problematic papers then I would not do much else. Part of the problem is the often poor scholarship when including references by acknowledging outdated and even erronneous studies, giving them a much longer life than deserved. I understand that one of your favourites is the intermediate disturbance hypothesis, but it continuous to live no matter how much arguments that are posed to its problem. I have a similar issue with the resource concentration hypothesis, that is so often wrongly cited that I wonder if anyone actually read the original paper. So, calling for retractions of truly fraudulent papers is absolutely necessary but the outdated papers we can just let die by themselves (or reanalyse the data if it is available).

  10. In reviews of clinical trials a paper is considered “old” if it is over five years old. And when performing a review, if more than two years has elapsed between analysis and searching the literature the findings are considered ‘out of date’. These ‘statutes of limitation’ are horribly short. And to my mind, completely unrealistic.

    This position of a 5-year rolling collection of evidence (held by a lot of government research institutes and evidence-based guideline developers) assumes that the clinical trial evidence base is constantly growing … but this is not at all the case in many, many areas of medicine. There are plenty of Cochrane reviews (the gold standard for systematic reviews of clinical trials) that are more than 5 years old and still represent the best evidence available – simply because no more has been done!

    I work in a controversial medical field (I shan’t mention it for fear of igniting comment armageddon) and there is a review paper from the 1990s that is so awful it deserves to be retracted. But it has yet to be superseded by any other review, specifically on that topic, even though there is more evidence available – I have asked people’s opinions about starting retraction procedures and most people said ‘it’s so old, why bother’? To me, it is not the age of the paper that matters, but its impact – the 90s review has been cited dozens of times and has been used over and over to mislead and misinform. The unfortunate impact of the error-filled data it contains far outweighs its age, to me.

    Perhaps impact should be considered alongside age: i.e. what would the impact be if the ‘old’ paper were left alone, and what would the impact be if it were retracted? If nothing would change, why waste time retracting it? So, I agree with the comments above that statutes of limitation would be field- and topic-specific, and almost impossible to apply. There will be more exceptions than rules! But that does not mean we shouldn’t consider age…just consider it wisely.

    Thanks for highlighting this issue though, as it is one that plagues much of the literature I have to work with in my job. It makes me feel a little less alone 😉

    • Wait ’til you see how much variation in opinion there is among the poll respondents as to what the “statute of limitations” on retractions ought to be…

      I agree with you that it’s hard to imagine a good, “one size fits all” length for a statute of limitations. But personally, I think some of our respondents are suggesting some pretty bad lengths!

  11. Everyone knows that evolution depends critically on the mechanism by which traits are inherited from parent to offspring. Surely any work on evolution that gets that mechanism completely wrong should be retracted, right? Removed from the scientific canon. No longer cited or citeable.

    The program of logical positivism was to make sure that scientific knowledge is true. It failed. This idea of “retracting” papers later shown to have flaws sounds an awful lot like trying to go back to that. There is absolutely no reason to retract publications that are wrong. It’s the job of Science (the big scientific community of which we are all a part) to critically evaluate and argue about the literature, but to cherish works about which we have to say, nice try but it didn’t work.

    • “Everyone knows that evolution depends critically on the mechanism by which traits are inherited from parent to offspring. Surely any work on evolution that gets that mechanism completely wrong should be retracted, right? Removed from the scientific canon. No longer cited or citeable.”

      Heh. I thought of using that example in the post. But I didn’t want to skew the post too much towards my own views and turn the poll into a push poll. Was also thinking of using Newton’s Principia as an example.

      “The program of logical positivism was to make sure that scientific knowledge is true.”

      Not sure I’d summarize the program of logical positivism quite that way. But it’s been a loooong time since I read A. J. Ayer back in college so my memory might be faulty…

      “It’s the job of Science (the big scientific community of which we are all a part) to critically evaluate and argue about the literature, but to cherish works about which we have to say, nice try but it didn’t work.”

      You’ll be pleased to know that your view seems to be widely held, at least among the people who read this blog…

      • The retraction model may work for fraud or OBIVOUS math/stat errors, but it is not part of the theoretical literature. Scientific concepts are developed through fits and starts, and early papers always contain goofs, hunches, assms that cant yet be defended, unrealistic assms, misunderstood assms, incorrect assms ( that later turn out to not matter) and so forth. That’s how new theory gets invented. And how theory advances.
        It is also worth noting that parties to scientific disputes always defend their position, and (almost) never admit defeat. so who gets to decide what is wrong?
        Like Hal, I vote for the ‘scientific community’ and the usual process of individuals deciding what is worth paying attention to.

  12. Pingback: Poll results: here’s what (some) ecologists think about retracting old and superseded papers | Dynamic Ecology

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.