Pre- vs. post-publication review and procedural justice

Wanted to elaborate some ideas from my recent post on “culture clashes” surrounding post-publication peer review. In that post, I suggested the post-publication review isn’t likely to work well unless we develop widely-accepted norms and procedures for how it should be done, just as we currently have for pre-publication review. I just thought of an analogy that I think clarifies and bolsters that argument. It’s an analogy between pre-publication review and procedural justice.

Procedural justice is the idea that justice reflects the fairness of the procedure used to resolve disputes. In A Theory of Justice, philosopher John Rawls distinguished three flavors of procedural justice. Perfect procedural justice consists of an independent criterion for what constitutes a just outcome, plus a procedure that guarantees that the just outcome will be achieved. Obviously, perfect procedural justice is more of a philosophical ideal than a real world possibility. Imperfect procedural justice is like perfect procedural justice, except that the procedure doesn’t guarantee that the just outcome will be achieved. Criminal trials are an example–there are rules and procedures to ensure that both sides get a fair hearing, but there’s no guarantee that the guilty will always be convicted and the innocent always cleared. Finally, pure procedural justice is when there’s no independent criterion for a just outcome; any outcome of a fair procedure is a just outcome by definition. For instance, in some sports a coin flip is used to decide which team gets the ball first. As another example, any outcome of a democratic election conducted under appropriate, fair procedures is a just outcome.

One reason to care about fairness of procedures is that fair procedures often will be more effective than unfair ones at producing just outcomes. For instance, “show trials” with unfair procedures stacked against defendants will mistakenly convict innocent people much more often than fair trials. But if what we ultimately care about is achieving just outcomes, then shouldn’t we prefer whatever procedure produces just outcomes most frequently, even if that procedure is unfair in some sense? Indeed, why should we even care about procedures at all, if what we ultimately care about is achieving just outcomes? Think of how criminal trials often are decried as “miscarriages of justice” in cases where the guilty go free or the innocent are punished, even though fair procedures were followed. That’s why books and movies often portray as heroes individuals who use ad hoc and illegal methods to achieve just outcomes.

Of course, there are other reasons for caring about procedural justice besides its effectiveness at achieving the right outcomes as judged by independent criteria. For instance, in order for a legal system to function effectively, it needs the cooperation of the general public. And that cooperation isn’t likely to be forthcoming if the legal system is widely viewed as unfair (e.g., Tyler 2003). That’s also one of the arguments for democratic elections over other means of transferring governmental power. Because democratic elections are seen as fair, the losing side is more likely to accept the outcome than if power is transferred by, say, a military coup. Put another way, while unfair or ad hoc procedures might seem like they’d lead to better outcomes than fair procedures in particular cases, in the long run they would actually lead to worse outcomes because of the chaos that would result from people refusing to accept them as legitimate. Put yet another way, unfair or ad hoc procedures are in some sense often parasitic on fair, agreed procedures. The only context in which occasional use of unfair or ad hoc procedures can improve the justice of outcomes (as judged by independent criteria) is a context in which those unfair or ad hoc procedures are used in rare, extraordinary circumstances. That way they don’t undermine everyone’s confidence in the fairness of the usual, agreed procedures.

In various ways, pre-publication review is analogous to a system of imperfect procedural justice, and advocates of post-publication review are analogous to critics of imperfect procedural justice. There are independent criteria as to what constitutes a publishable scientific paper, such as technical soundness. And pre-publication review is an imperfect system of procedures for fairly and accurately identifying papers that meet those criteria. In their insistence that what ultimately matters is getting the science right and correcting the record in individual cases by whatever means necessary, the stronger advocates for post-publication review are analogous to those who care little for procedural niceties in other contexts. Instead focusing on achieving the right outcomes by whatever means necessary on an ad hoc, case by case basis. And just as with other examples of imperfect procedural justice, like the criminal justice system, one can argue that any effective peer review system needs to not only achieve the right outcomes at a high rate, but be seen to comprise fair procedures. Otherwise people won’t voluntarily participate in or cooperate with the system, causing the system to break down and cease to function effectively. And finally, one can argue that ad hoc post-publication review is effective only in rare, unusual cases, as a supplement to the existing pre-publication review system (e.g., fraud detection; close and rapid scrutiny of especially important or high-profile papers).

I also wonder if pre-publication review is in some ways analogous to a system of pure procedural justice. This is a new thought I’ve just had, unlike everything above, which is just elaborating that recent post of mine. So bear with me while I think out loud… The idea is basically that we need some way of adjudicating disagreements as to what science is worth publishing (and such disagreements are common; see this old post). We also need some way of coming to final accept or reject decisions, some way of declaring the review process closed so that everyone concerned can move on. Sort of like democratic elections. You can’t expect everyone to come to an agreement about who the best candidate is, no matter how long a political campaign is allowed to go on. There will always be differences of opinion, both because some differences of opinion are reasonable, and because some people will never change their minds under any circumstances. But despite those irresolvable differences, we need a procedure for cutting off debate and picking a winner, because we need to have a government and everyone needs to get on with their lives. What we need, and all we can ask for, is a fair procedure for picking the winner. If the procedure is fair, that makes the outcome fair, whatever the outcome may be.

I think there’s something to this argument, both in the context of democratic elections and in the analogous context of peer review. Which is why I disagree with folks like Andrew Gelman who’d prefer to move to a review system in which publication is never the last word, even provisionally. Who would prefer scientific communication to consist of totally open-ended discussion, with traditional papers either ceasing to exist or else being viewed very differently. Don’t get me wrong, like Andrew I hate it when somebody defends a paper from post-publication criticism by saying “It was peer reviewed”, as if that means the paper must be right. Analogously, I hate it when an elected government defends its policies by saying “We won the election”, as if that means its policies must be wise. But I suspect that those who defend their work against post-publication criticism by saying “It was peer reviewed” generally don’t mean “it must be right.” Instead, what they mean is something like what elected politicians mean when they say “elections have consequences“. The phrase “elections have consequences” is shorthand for the fact that fair elections are a form of pure procedural justice, not imperfect procedural justice. The fairness of the procedure is what makes the outcome (and its consequences) legitimate. So if you don’t like the outcome, well, you’re free to complain, but ultimately there’s nothing you can legitimately do about it besides live with it and try to win the next election.* Analogously, one can argue that science needs fair peer review mechanisms that, for at least some purposes, cut off debate and discussion and allow all concerned to move on. Not all debate and discussion, of course, any more than elections cut off all political debate and discussion. But political discussion and debate aims to change the outcome of the next election, rather than undo the outcome of the previous one. Otherwise politics and government cease to function effectively. Analogously, one can argue that scientific discussion and debate should be directed primarily towards determining what papers get published in future, rather than re-reviewing papers that were published in the past. Because otherwise science will cease to function effectively.

Interestingly, the analogy to pure procedural justice partially undermines attacks on pre-publication peer review based on its inability to predict the subsequent “impact” of papers (however “impact” is measured). That kind of attack only has bite insofar as you think of pre-publication peer review as an imperfect procedure for attaining some independently-defined goal like “identify high impact papers”. It has no bite insofar as you think of pre-publication peer review as analogous to pure procedural justice.

I think the analogies of peer review to imperfect procedural justice and pure procedural justice both have something to them. Which may be what makes debate about peer review reform challenging, and leads to people sometimes talking past one another. Because you need to evaluate pre-publication peer review on totally different and even contradictory grounds depending on whether you see it as a system of imperfect procedural justice or a system of pure procedural justice.

This has been another edition of Jeremy’s Half-Baked Analogies. 🙂 Looking forward to your comments.**

*Except perhaps in extraordinary circumstances such as when the elected government tries to change the rules so as to enshrine itself in office permanently. And one can of course argue that there’s a scientific analogue of that sort of thing–papers that have especially egregious flaws, maybe.

**I predict that the proposed analogy to pure procedural justice will convince few of you. 🙂