Here is a Science news article on the recent retraction of high-profile papers on COVID-19 treatment. The retractions are for dodgy data that came from dodgy doctor Sepan Desai (see here for further background). I was struck by this comment on the situation in the Science news article from former NEJM EiC Jerome Kassirer:
Desai, Mehra, and Patel had never before published together, and that should have been a red flag to any journal, says Jerome Kassirer, editor-in-chief of NEJM during the 1990s. Co-authors of high-profile papers normally share subject area expertise or have clear professional ties, he says, calling the collaboration of the apparently disparate individuals “completely bizarre.”
The trend in ecology and evolution is towards double-blind peer review. But Kassirer’s comments suggest that there are rare circumstances in which it might be informative for reviewers to know the authors’ identities. Or that there are rare circumstances in which editors should take author identities into account in their decision-making, even if the author identities aren’t revealed to the reviewers.
What do others think of this? I’ve only just started to think about it myself, so here are my tentative thoughts:
- UPDATE: judging by the first few comments, I definitely need to clarify something: I don’t think that editors or reviewers should regard it as suspicious if the authors on a paper have never co-authored before or lack any professional ties. This post is about the broader question in the post title. I disagree completely with Jerome Kassirer’s specific answer to that broader question, quoted above. I think that, if that Desai paper was going to be subjected to greater-than-usual scrutiny during peer review, it should’ve been because Desai, and Surgisphere, appear to be dodgy. Not because Desai hadn’t previously co-authored with the other authors on the paper. Below, I think out loud about how I myself would answer that broader question in the post title. /end update
- The Desai case is unusual in that it’s very extreme. Desai and his company Surgisphere look dodgy even to casual web browsing; see the second link above for details. Arguably, one shouldn’t try to make policy based on unusual and extreme cases–not even policy designed to prevent other unusual, extreme cases. So maybe the Desai case shouldn’t inform any journal’s policies regarding double-blind review.
- On the other hand, it’s not hard to think of other real or hypothetical scenarios in which journal decision-making ought to be informed by information about the author’s background and expertise. For instance, I think a journal should subject a submission from somebody with more than, say, 20 retractions to more than the usual scrutiny. Rather like how banks will refuse to loan to people with long track records of dodgy business dealings, at least without more than the usual amount of guarantees and paperwork. But on the third hand, the other real and hypothetical scenarios that occur to me also are pretty rare and unusual. So maybe they too shouldn’t lead us to try to write exceptions into journal double-blind review policies.
- In the context of grant review, there seems to be fairly widespread agreement that reviewers should know the identities of the applicants, so that they can take into account whether the applicants have the expertise, equipment, and other resources needed to carry out the proposed research. For instance, here in Canada, 1/3 of the score of an NSERC Discovery Grant proposal is based on the applicant’s track record of research productivity over the previous 6 years, on the grounds that researchers who run productive research programs are good bets to remain productive in future. NSERC Discovery Grant reviewers also tend to assume that applicants with relevant experience and expertise can carry out the research they’ve proposed. Applicants proposing to do something they’ve never done before tend to get treated with more skepticism by reviewers, and are expected to provide more evidence that they can do whatever it is they’re proposing to do. I personally think it’s reasonable for Discovery Grant reviewers to take that attitude, though I can also appreciate complaints that it makes reviewers too risk-averse. Heck, I myself have in the past proposed to do things that Discovery Grant reviewers were a bit skeptical of, because I hadn’t done anything similar before. And then I went and tried to do those things and they didn’t work out well! I ran into technical obstacles that someone with more experience than me probably would’ve been able to either avoid or overcome. So at least in my own admittedly-anecdotal case, the reviewers were right to be a little skeptical of my ability to do certain things I’d never done before, solely on the grounds that I’d never done them before! But of course, a grant proposal is a different beast than a paper, because a paper reports work that’s already been done.
- Dealing sensibly with rare, unusual cases in which the usual rules don’t apply, or likely would lead to bad outcomes if they were applied, seems like a job for good professional judgment, exercised with care and discretion. Because it’s impossible to anticipate in advance all the rare, unusual cases one might encounter.