Another high profile case of post-publication review is in the news, to do with a Nature paper demonstrating an easy method to create pluripotent stem cells. The paper’s dramatic claims immediately received heavy scrutiny from a lot of people, and look like they might be walked back or even retracted. See here for an overview of the post-publication discussion from PubPeer (a website dedicated to post-publication review).
Advocates of post-publication review like to trumpet such cases, understandably and rightly so. Thanks to new online tools, science can now self-correct instantly! I’m all in favor of finding new, effective ways for science to self-correct (see also this). In the linked piece above, the folks at PubPeer suggest that post-publication review is “here to stay”.
To which I’d respond, that’s fine, but let’s be more precise. Post-publication review of very high-profile papers is here to stay. Post-publication review of most other papers not only isn’t here to stay, it doesn’t exist.
Think of the cases of post-publication review everyone talks about. Arsenic life, stem cells, papers on dinosaur growth rates and network analyses discussed in this old post…Such papers are the scientific 1%. Actually, probably 0.01%, or some other really small number. It’s mostly only really high-profile papers, published in Science, Nature, and a few other top journals, that receive close post-publication scrutiny from lots of readers. The vast majority of papers don’t receive any post-publication “review” at all, because relatively few people read them and the people who do read them mostly just read the abstract or skim.*
Which isn’t necessarily a criticism of post-publication review, actually! Indeed, one could argue that some sort of “hybrid” system is close to ideal. Because in a hybrid system, every paper gets careful pre-publication review from 2-3 people, and then the small fraction of papers that lots of people really care about gets post-publication review from lots of people.
But it’s important to recognize the essential role that pre-publication review plays in this hybrid system: it ensures every paper gets scrutinized by somebody. You need some mechanism to ensure that, because left to their own devices most people just pay attention to the same small fraction of stuff everyone else pays attention to.** That’s why distributions of “attention concentration” for everything from citations, to downloads of scientific papers, to website traffic, to YouTube video views, to book sales, to movie box office grosses, are highly skewed.
In a recent post, I suggested that post-publication review will be most effective when, like pre-publication review, it’s based on agreed norms and practices everyone buys into. One aspect of pre-publication review is that it forces redistribution of attention. It’s a practice that ensures at least some close attention is paid to every single paper–namely, the attention of the pre-publication reviewers. I don’t see any sign of any norm or practice that would force redistribution of attention in post-publication review. So if you want post-publication review can serve as a complete replacement for the current functions of pre-publication review (and I recognize that that may be a big “if” for some), then I think you need to suggest some mechanism to redistribute the attention of post-publication reviewers.***
In summary, advocates of post-publication review are rightly impressed with high-profile cases in which it works. It’s very rare for the scientific record to rapidly self-correct, so we should be glad when it happens. But in its current form, post-publication review is only for “elite” papers–it doesn’t scale to the whole scientific literature. It’s actually pre-publication review, or some hybrid of pre- and post-publication review, that works at scale.
p.s. Please don’t push back against this post by noting that pre-publication reviewers sometimes do a careless job, or sometimes just miss things. That’s true, but irrelevant to the point of the post.
*That’s why the large majority of papers attract no post-publication comment at all, on any of the many commenting systems that have been tried–PubPeer, PubMed Central, journal-based commenting systems, blogs, etc. At least as far as I know–if you have data to the contrary, I’d love to hear about it. I know some folks chalk this up to the technical design of previous commenting systems, and believe that if only we get the design right, most every paper will attract comments. I doubt it, but I’ll happily change my mind if the data demand it.
**In the current hybrid system, pre-publication review also contributes to a second function: concentrating the attention of post-publication “reviewers” on a small fraction of papers. Post-publication review mostly falls on papers published in high-profile journals, and pre-publication review goes a long way towards determining what gets published in high-profile journals. Of course, if pre-publication review ceased to exist, post-publication review would still concentrate on a small subset of papers to the exclusion of most others. It’s just that the concentration would occur via some other mechanism(s), which probably would be just as stochastic and subjective as current attention-concentrating mechanisms. See this old post.
****For instance, you could instead argue that some current functions of pre-publication review don’t need replacing. For instance, you could argue that science would work as well, or better, if everyone just published everything without review, and then readers just chose to read, comment on, and use those preprints however they saw fit. Which is a much larger and different discussion, I think. I hesitate to dive into that discussion too much, as that hypothetical world is just too different from the one I’m used to for me to really wrap my mind around it. For what it’s worth, there certainly are disciplines (economics, physics) in which unreviewed preprints are a much more important part of professional discourse than is currently the case in ecology. But even in those disciplines, peer-reviewed journals do still play various important roles: validating the outcomes of pre-publication discourse, scrutinizing and improving papers that might not otherwise be closely scrutinized at all, and other functions. For instance, see this editorial by a physicist who’s a strong supporter of the ArXiv preprint server, or this recent post from Andrew Gelman. So I think the examples of economics and physics certainly are suggestive, and indeed have argued that ecology could learn something from how economists communicate. But I don’t think that economics and physics provide already-existing examples of disciplines where post-publication review has replaced pre-publication review.