What, if any, are the circumstances in which the authors’ background or expertise should inform journal publication decisions? (UPDATED)

Here is a Science news article on the recent retraction of high-profile papers on COVID-19 treatment. The retractions are for dodgy data that came from dodgy doctor Sepan Desai (see here for further background). I was struck by this comment on the situation in the Science news article from former NEJM EiC Jerome Kassirer:

Desai, Mehra, and Patel had never before published together, and that should have been a red flag to any journal, says Jerome Kassirer, editor-in-chief of NEJM during the 1990s. Co-authors of high-profile papers normally share subject area expertise or have clear professional ties, he says, calling the collaboration of the apparently disparate individuals “completely bizarre.”

The trend in ecology and evolution is towards double-blind peer review. But Kassirer’s comments suggest that there are rare circumstances in which it might be informative for reviewers to know the authors’ identities. Or that there are rare circumstances in which editors should take author identities into account in their decision-making, even if the author identities aren’t revealed to the reviewers.

What do others think of this? I’ve only just started to think about it myself, so here are my tentative thoughts:

  • UPDATE: judging by the first few comments, I definitely need to clarify something: I don’t think that editors or reviewers should regard it as suspicious if the authors on a paper have never co-authored before or lack any professional ties. This post is about the broader question in the post title. I disagree completely with Jerome Kassirer’s specific answer to that broader question, quoted above. I think that, if that Desai paper was going to be subjected to greater-than-usual scrutiny during peer review, it should’ve been because Desai, and Surgisphere, appear to be dodgy. Not because Desai hadn’t previously co-authored with the other authors on the paper. Below, I think out loud about how I myself would answer that broader question in the post title. /end update
  • The Desai case is unusual in that it’s very extreme. Desai and his company Surgisphere look dodgy even to casual web browsing; see the second link above for details. Arguably, one shouldn’t try to make policy based on unusual and extreme cases–not even policy designed to prevent other unusual, extreme cases. So maybe the Desai case shouldn’t inform any journal’s policies regarding double-blind review.
  • On the other hand, it’s not hard to think of other real or hypothetical scenarios in which journal decision-making ought to be informed by information about the author’s background and expertise. For instance, I think a journal should subject a submission from somebody with more than, say, 20 retractions to more than the usual scrutiny. Rather like how banks will refuse to loan to people with long track records of dodgy business dealings, at least without more than the usual amount of guarantees and paperwork. But on the third hand, the other real and hypothetical scenarios that occur to me also are pretty rare and unusual. So maybe they too shouldn’t lead us to try to write exceptions into journal double-blind review policies.
  • In the context of grant review, there seems to be fairly widespread agreement that reviewers should know the identities of the applicants, so that they can take into account whether the applicants have the expertise, equipment, and other resources needed to carry out the proposed research. For instance, here in Canada, 1/3 of the score of an NSERC Discovery Grant proposal is based on the applicant’s track record of research productivity over the previous 6 years, on the grounds that researchers who run productive research programs are good bets to remain productive in future. NSERC Discovery Grant reviewers also tend to assume that applicants with relevant experience and expertise can carry out the research they’ve proposed. Applicants proposing to do something they’ve never done before tend to get treated with more skepticism by reviewers, and are expected to provide more evidence that they can do whatever it is they’re proposing to do. I personally think it’s reasonable for Discovery Grant reviewers to take that attitude, though I can also appreciate complaints that it makes reviewers too risk-averse. Heck, I myself have in the past proposed to do things that Discovery Grant reviewers were a bit skeptical of, because I hadn’t done anything similar before. And then I went and tried to do those things and they didn’t work out well! I ran into technical obstacles that someone with more experience than me probably would’ve been able to either avoid or overcome. So at least in my own admittedly-anecdotal case, the reviewers were right to be a little skeptical of my ability to do certain things I’d never done before, solely on the grounds that I’d never done them before! But of course, a grant proposal is a different beast than a paper, because a paper reports work that’s already been done.
  • Dealing sensibly with rare, unusual cases in which the usual rules don’t apply, or likely would lead to bad outcomes if they were applied, seems like a job for good professional judgment, exercised with care and discretion. Because it’s impossible to anticipate in advance all the rare, unusual cases one might encounter.

19 thoughts on “What, if any, are the circumstances in which the authors’ background or expertise should inform journal publication decisions? (UPDATED)

  1. Jeremy, I don’t think the point about NSERC (and other) grants is really relevant. We consider the applicant in a grant because we are seeking to predict future success with research, as you point out. When a manuscript is in hand, we aren’t trying to make that prediction.

    I’m also really surprised by this from Kassirer: “Desai, Mehra, and Patel had never before published together, and that should have been a red flag to any journal”. That’s bizarre. New collaborations are a routine thing and a good thing. In fact, the only way you can have any collaborations at all is to have new collaborations. I have no idea what he could possibly be thinking, calling that a “red flag”!

    I do take your point about an author with 20 retractions. That seems like a case where we’d pay extra attention…

    • My very first thought aligns perfectly with Steve’s second point. The idea of treating a paper from authors who have never worked together before with a “red flag” approach truly is a bizarre idea, and perhaps even a dangerous one. I mean, if one moves positions, you’d expect one to start collaborating with a whole new group of people. I don’t see why that should be a red flag. Even if one doesn’t move and strikes up a new collaboration, I really can’t see why this should be viewed as suspicious. Perhaps if the collaborations are considered outstandingly odd, but even still, “odd” cross-disciplinary collaborations happen.

      All this to really say that I agree with Steve – I find Kassirer’s insinuation that “new” collaborations should be treated with skepticism very puzzling, and, for me, it’s an unwelcome idea.

      • “All this to really say that I agree with Steve – I find Kassirer’s insinuation that “new” collaborations should be treated with skepticism very puzzling, and, for me, it’s an unwelcome idea.”

        Agreed. As I said in the post, I do think there are dodgy things about Desai that and editor or reviewer would want to take into account during the editorial decision-making process. But “Desai collaborated with other people who’ve never co-authored together before” isn’t one of them.

    • “eremy, I don’t think the point about NSERC (and other) grants is really relevant. We consider the applicant in a grant because we are seeking to predict future success with research, as you point out. When a manuscript is in hand, we aren’t trying to make that prediction. ”

      I agree; just thinking out loud in the post.

      “I’m also really surprised by this from Kassirer: “Desai, Mehra, and Patel had never before published together, and that should have been a red flag to any journal”. ”

      Yeah, I thought that was strange too. “Would I have expected these authors to collaborate on research on this topic?” just doesn’t seem like a question I’d ever want to ask as a reviewer or editor. I mean, yes, it’s weird that established academic researchers would be co-authoring with somebody like Desai. But it’s weird because Desai’s background and company look dodgy. So once you as an editor recognize that Desai seems dodgy, I don’t think “there are other, established authors on this paper” adds any additional reason to subject the paper to greater-than-usual scrutiny.

  2. The argument about authors never having published together before doesn’t make much sense to me; often good research comes from people forming new collaborations. From my admittedly low experience, it’s good to have long-term research partners but it’s also productive to try new collaborations. I don’t really see why this would be a red flag; but perhaps it’s something subject-specific.

    On the other hand, I think that knowing author identities may be relevant not only for evaluating grants but also for evaluating papers – assuming the reviewers are honest and don’t let personal issues get in the way of a fair evalluation. People working in the same field and in the same region are likely to have some familiarity with one another’s research, including possible misconduct. If author names are known to the reviewer and the reviewer knows about misconduct from the author, he or she may signal it to the editor, who may in turn request a more detailed assessment of the manuscript.

    This, however, brings another question: should a manuscript be evaluated solely based on what’s written in it, or should other information which reviewers might have also be taken into account? What are your thoughts on this?

    • I can see the rationale in your second point, but I’m not sure I agree with it. High profile cases of misconduct almost always seem to happen post-peer review. The probability of an editor selecting a reviewer that has inside knowledge of an author’s malpractice is, I would think, very low. To have that kind of researcher-specific information would mean you’d have to work closely with that researchers, and in most cases means you wouldn’t review their paper due to a conflict of interest. I think that negative effects of knowing author names (e.g. https://www.natureindex.com/news-blog/dont-let-famous-names-scientists-influence-peer-review-journal) outweigh the (likely low) potential of a reviewer catching misconduct based on author identity.

    • “The argument about authors never having published together before doesn’t make much sense to me”

      Agree, I thought that was weird.

      “This, however, brings another question: should a manuscript be evaluated solely based on what’s written in it, or should other information which reviewers might have also be taken into account? What are your thoughts on this?”

      Well, there’s a sense in which reviewers already do that, right? For instance, it’s normal for a review to say something like “The methods section doesn’t mention if the authors did X. As someone who works in this system myself, I know how important it is to do X, for reason Y. So the authors need to say if they did X.”

  3. I think that a good number of important papers in ecology, especially synthesis papers, are published by groups of authors who had never worked together before. So I don’t think Kassirer’s point is correct, at least in ecology. And I am not sure whether it would be a good practice that journals gathered information about the authors to make decisions about their papers; I think decisions should be made based on the science being reported in the manuscript.

    Regarding grant evaluations, knowing who the applicant is seems necessary to assess the feasibility of the work proposed. It does seem a different situation from that of journal papers.

  4. This may be a bit off-topic. I don’t think I agree with how NSERC handles discovery grant. I agree that scientists should have sufficient experience but having worked in a particular area shouldn’t be a strict requirement for a “discovery grant”, especially if the proponent has shown a level of competence and productivity elsewhere. Of course, a novice to a field may run into avoidable technical difficulties as Jeremy points out. But many did just fine. I’m not even sure that “people may have difficulties” is a valid argument. But I’m sure they can bring about a breakthrough via cross-system thought processes. Check out backgrounds of those scientists that gave us the more enduring theories, you will find some commonalities. My experience is that scientists within a field can develop a collective tunnel vision, which can spill over to what journals accept to publish.

  5. The handling editor, EiC, or in some instances, even the journal’s administrative staff can check retraction lists (or do potentially other dodginess checks) without revealing author identity to the reviewers. Is there a scenario where they can’t? And if so how likely is it compared to the scenario where reviewers are biased for or against particular people.

    • I think this is an important point. It is the distinction between double blind (authors & reviewers blinded) and triple-blind (editors also blinded). Although triple-blind is common in the social sciences, it is almost unheard of in the sciences where double-blind is much more common (although I think single blind is still even more common). To me double blind gets the best of both worlds. The people with the most weight on outcome (reviewers are blinded) but others in a position of scrutiny are unblinded for exactly these kinds of scenarios (and also to avoid sending papers for review to colleagues down the hall from the author).

  6. When I clicked in to this post, I was actually expecting it to open up the can of worms around “separating the man from the science” and personal misconduct and appalling viewpoints, rather than discussing the question of professional malpractice. That former discussion is on-going in some spaces in science and is long over-due in others. In these cases, knowing author identity would be extremely important. The central question might be: Who do we agree will become responsible for the positive peer pressure necessary to correct (or even ultimately black list) scientists who behave inexcusably (say to those over whom they have seniority or direct supervisory powers)? Knowing that (academic) professional evaluations are largely based on awarded grants and publications (and to some extent citations), we have viable mechanisms for constraining and even “divesting” from toxic individuals, regardless of how “good” their science may be. Publications could ban them for a period of time or permanently. We could seek alternative citations to replace those from toxic individuals, or develop methods for qualifying citations of such work. And, we’ve seen major societies starting to move in this direction with the move to expel folks from fellowships, rescind funding, etc. This all seems particularly important in light of current events. And, fundamentally, there is the need to acknowledge that one person’s toxic behavior may compromise or prevent so many other people’s good science that that one toxic individual’s science cannot be held up as a defense. I’m sure readers and the Dynamic Ecology co-publishers are quite aware of any number of cases where this discussion and actual steps to address these issues are warranted. Since the blog post title provoked a renewed mulling-over of this for me, I figured I’d bring it up here. 🙂

    • I can see why some readers might’ve anticipated that this post would speak directly to current events in ecology & evolution. It’s actually a coincidence that this post came out shortly after the CSEE rescinded Denon Start’s awards. I wrote the post and put it in the queue in the middle of last week

      “Who do we agree will become responsible for the positive peer pressure necessary to correct (or even ultimately black list) scientists who behave inexcusably (say to those over whom they have seniority or direct supervisory powers)?”

      I feel like the obvious answer to that question is “their employers, funding agencies, and scientific societies”. Did you have others in mind?

      Re: an individual’s scientific achievements not somehow compensating for their bad behavior towards others, I *completely* agree. I would only add that my own reasons for saying that have *nothing* to do with whether those others would have gone on to do good science themselves. My own view is that bullying, sexual harassment, and other toxic behaviors aren’t any worse if they’re directed towards aspiring scientists than towards anyone else. And when reflecting on the terrible consequences of bullying and harassment, I don’t think there’s any reason to try to weigh the bully’s or harasser’s professional previous and future achievements (scientific or otherwise) vs. the previous and future professional achievements of the victims (scientific or otherwise). Rather, everybody’s life and aspirations have equal and *intrinsic* value. One absolutely can, and should, mourn for the scientific careers that are derailed by bullying and harassment, without trying to somehow weigh up the scientific achievements of the bully or harasser against what the victims would have achieved scientifically. Bullying and harassment of aspiring scientists is not bad because of the possible net loss to future scientific knowledge when victims of bullying are driven out of science by a bully who’s a scientist. Rather, bullying and harassment are bad because nobody, scientist or non-scientist, should have their life derailed by another person against their will. The way to push back against an attempt to defend a bully by pointing to the bully’s scientific achievements is not to say “But what about the science the bully’s victims would have done?” That defense, well-meaning though it undoubtedly is, is wrong-headed because it implicitly accepts that scientific achievement potentially *could* compensate for bullying or harassment, at least partially. Rather, I think the right way to push back against any attempt to defend a bully by pointing to the bully’s scientific achievements is simply to say “bullying is bullying, and it’s wrong; that the bully also has scientific achievements is irrelevant”.

      Re: whether to cite individuals who have behaved badly, I have an old post on that: https://dynamicecology.wordpress.com/2018/11/07/when-should-scientists-cite-the-work-of-sexual-harassers/

      • “Rather, bullying and harassment are bad because nobody, scientist or non-scientist, should have their life derailed by another person against their will. […] bullying is bullying, and it’s wrong; that the bully also has scientific achievements is irrelevant.”
        –100% with you on this, Jeremy.

        Re “Who do we agree will become responsible for the positive peer pressure necessary to correct (or even ultimately black list) scientists who behave inexcusably (say to those over whom they have seniority or direct supervisory powers)?

        ‘I feel like the obvious answer to that question is “their employers, funding agencies, and scientific societies”. Did you have others in mind?'”

        –Yes. 🙂 There are also things we ourselves can do, individually and collectively as peers/colleagues, departments, reviewers, editors, etc. I envision squeezing from the top *and* the bottom, based on the insights I’ve read about this from folks working actively in these arenas.

        You’ve raised one of those in the post you linked to about citations, both to exclude and to more fully include certain authors. This is one I’m particularly interested in for a couple of reasons:

        1. The single- vs. co-authored nuance is important, for reasons discussed in the comments of that post. Agreed – it is much more straightforward when the material is single-authored. I wonder – is it possible/has it ever happened that individuals have been retracted from an author list, while leaving the publication on-record?

        2. There’s a scholar of creativity in science (especially in historical contexts) who has abhorrent views about several things, and I can no longer justify (to myself) citing his work most of the time. However, when writing grant proposals to fund ambitious, transdisciplinary work (particularly training programs for students and early career researchers), some of his work is the only extant work that we can point to, and it makes a powerful, foundational case for integrating arts and sciences in science training and research. I wish someone would do some research in this area (it’s not really my area of specialty, hence the wishing), so we could point to someone else. But, for now, I feel stuck in your area of ‘compelled citation’ discussed in your blog post.

        3. I’ve more recently become aware of the potential to deliberately diversify citations as a way to a) expand the “canon” and b) contribute to boosting people’s career impacts. I’m also seeing it, increasingly, as a way of expanding discourse to include more publicly available materials relevant to my work (not just citing peer-reviewed work).

        Beyond citations, I’ve also been learning that there is work we can do individually and collectively to stigmatize reprehensible behavior through the way we organize and manage labs and research projects, how we train and mentor people, how we react to unacceptable comments in hallways, meetings, etc., and even how we set and implement department policies and/or assign/respond to peer-review requests. There’s so much more that I’m missing in this list, but it’s a start. And, mostly, the point is to say that we can push at this from the bottom, too. 🙂

      • Re: retractions of some authors from the author list: no, that’s not a thing, to my knowledge. And I don’t think it should become a thing. If the paper can’t be relied on and so needs to be retracted, then it needs to be retracted (and should no longer be cited, etc.) Ensuring that authors who were not at fault for the retraction don’t suffer professional consequences (e.g., as when one author fabricated the data, unbeknownst to the co-authors) is a separate issue.

  7. At least some types of grant applications in Astronomy are now double blind, and it doesn´t seem to matter much. It may be somewhat format related; in my experience, applications typically have to be technical enough that you can tell if someone knows what they´re doing – I believe the NSERCs are not so much?

    Otherwise, publishing with people you don’t know or haven´t published with before is pretty common for us. I doubt I could name half my co-authors without looking them up, nor pick them out of a lineup.

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.