One function of pre-publication review is redestribution of attention: it ensures that every paper gets closely read by at least a couple of experts, and none gets read by many (no paper ever gets hundreds or thousands of pre-publication reviews). Post-publication, attention (by any metric you care to name) is very highly concentrated: a small fraction of papers (the “scientific 1%“) attracts a large fraction of scientists’ collective scrutiny. Which is one reason I why I’ve long been a skeptic that post-publication review, in the form of uninvited reviews or other comments, can ever replace (as opposed to supplement) pre-publication review. Under post-publication “review”, the vast majority of papers don’t get reviewed in any meaningful way (e.g.).
As evidence for this view, I and others have noted that many journals have long had online systems allowing comments on their papers, but those systems are little used and the vast majority of papers attract no comments. Some advocates of post-publication review recognized that, but chalked it up to the clunky design of journal commenting systems, and/or the fact that no single system allowed people to comment on papers from many journals.
The advent of PubMed Commons a couple of years ago put our views to the test. PubMed Commons allows commenters to comment on any paper indexed by PubMed–which is pretty much every biomedical paper and then some. And it was proposed and designed by very sharp, prominent people who think that the key obstacles to a culture of post-publication review are technical.
Turns out I was right. I searched PubMed Commons for all articles that have comments. (conveniently, PubMed Commons provides a link on their homepage to conduct this very search!) The search returns 3082 articles. By contrast, PubMed indexed 1,186,208 articles in 2014 alone.*
I can imagine various responses to this, if you think post-publication review can and should replace pre-publication review (and yes, there are those who think that). Probably some of my imagined possible responses are ones that nobody would actually make, but I’ll just list them anyway.
- You could argue that the problem is that there are now too many centralized hubs where people can comment on published papers, causing many people to just not bother commenting because they’re not sure of the best place to do it. I highly doubt it. If you surveyed scientists and asked them why they don’t do post-publication review, they mostly wouldn’t say “because I’m not sure of the best place to do it.”
- You could point out that the number of commented articles on PubMed Commons is growing exponentially, which it is. To which I’d respond that at this rate it’d take many years at best for the majority of PubMed articles to be commented, depending on the growth rate of the entire PubMed database and whether the growth rate in the number of commented articles ever slows down. But we’ll see, I guess.
- You could claim that the vast majority of published papers don’t receive serious pre-publication peer review either. That claim suffers from having no evidence to support it.**
- You could acknowledge the issue and try to fix it by urging everybody to please publicly share all their private discussions about published papers (all your journal club meetings, email exchanges, Facebook exchanges, post-seminar questions, private face-to-face conversations…) To which, good luck with that, you’ll need it. Plus, what makes you think that most papers get thoroughly discussed in private? People’s journal club meetings, emails, etc. etc. are mostly about the same high-profile papers that get the bulk of the attention by other metrics.
- You could argue that the issue isn’t important enough to be worth worrying about. I disagree, of course, but beyond that I’m not sure how to respond because that argument implies a totally different vision of what peer review is for.
- You could grant the point and use it to argue for some post-publication system of invited reviews.
- Or you could take what I think is the reasonable view that post-publication review can be a useful supplement to pre-publication review, but mostly for high profile papers.***
But quite possibly, there are other responses I haven’t thought of. Which is what the comments are for. Looking forward to them, as always.
p.s. I freely admit that this post focuses on one narrow issue with post-publication review. Please don’t read it as a blanket condemnation of any and all post-publication review, because it’s not.**** I’m just writing about one issue that I think is important enough to be worth discussing, that isn’t discussed as much as it should be*****, and on which I could easily obtain a bit of data.
p.p.s. Before anyone asks, I also tried to figure out how many papers have PubPeer comments. I couldn’t figure out how to extract those data, but if anyone knows, please tell me. Given that there are recent comments over there from PubPeer users, remarking on how most articles never get commented on PubPeer, I’m guessing that looking at PubPeer data would reinforce my argument. But happy to be proven wrong.
*So maybe we should be talking about post-publication review being for the scientific 0.01%. Note that the fraction of papers that get cited at least once, while infamously low, is over three orders of magnitude higher. So at least judging from PubMed Commons data, post-publication review attention is much more concentrated than post-publication citations.
**No, the fact that a tiny but growing proportion of papers get retracted does not show that the vast majority of papers receive only cursory pre-publication review.
***Especially if we develop agreed norms and practices to govern it. But I freely admit I have no idea how to do that.
****I’m not opposed to any and all post-publication review. Quoting myself: “[A]dvocates of post-publication review are rightly impressed with high-profile cases in which it works. It’s very rare for the scientific record to rapidly self-correct, so we should be glad when it happens.” I’m just not a utopian. I don’t like focusing exclusively on the upsides of any new thing while hoping that the downsides will somehow get addressed. Sorry. I’m aware that can make me a downer sometimes.
*****Well, as far as I know it’s not discussed all that much. But I don’t often lurk in the sorts of forums where those discussions would be mostly likely to happen. So maybe I’m unaware of tons of discussion of the fact that most papers get no post-publication review. FWIW, I did a bit of googling, and didn’t turn up much. Just a few posts (e.g.,this, this, and this) raising the issue, often along with other issues, without much in the way of plausible solutions from advocates of post-publication review. I did find the suggestion that somebody (it’s not clear who) should increase scientists’ incentives to do post-publication review. Overlooking that there’s already one incentive to do it, but it’s not a very nice one.