Writing in Science, Lambert et al. report that they were unable to recreate the results of Scheele et al., attributing declines in many amphibian species to chytrid fungus, based on a synthesis of various lines of evidence. Scheele et al. reply. I’m interested in this exchange itself. But I’m more interested in a broader issue it raises, regarding how to structure comment-reply exchanges in the scientific literature.
Ok, regarding the exchange itself: It’s striking because it’s not to do with the interpretation of the data (as is usual for most comment-reply exchanges), but with what the data actually are and whether the analyses are reproducible. Now, I am far from an expert on amphibian conservation; I’m merely a curious bystander here. So while I certainly have my own opinion on who gets the better of the exchange, it’s probably not worth much.*
(UPDATE: Scheele et al. have updated the data file and associated references on Science’s website. The original data file is no longer available, which seems like bad practice to me.)
I’m more interested in this exchange because raises a broader issue. Reading the exchange, it seemed to me that Scheele et al.’s reply often talked past Lambert et al.’s comment rather than directly addressing it. Or at least, I found it hard to match up Lambert et al.’s criticisms with Scheele et al.’s replies to those criticisms. And there’s one point at which Scheele et al. reply to a criticism that Lambert et al. not only didn’t make, but explicitly disavowed.** I think it’s bad when authors reply to a comment on their work by talking past the comment. In my admittedly-anecdotal experience, that happens fairly often in comment-reply exchanges. Authors address the questions they think were raised, or the questions they wish had been raised, rather than the questions that were actually raised. It’s easy to see how this can happen. As an author, it’s often annoying when somebody comments on your paper. It’s easy for that annoyance to cause you to misunderstand the comment. But whatever the reason for authors talking past commenters, it confuses scientific debates when it happens.
So, how should comment-reply exchanges in journals be structured? The current approach at many journals is that authors are entitled to say more or less whatever they want in reply to a comment on their work. Comments and replies do get peer-reviewed, but at the end of the day most editors prefer to publish comments they deem meritorious, let the authors reply however they want, and let readers make up their own minds. There’s certainly a lot to be said for that approach. It’s loosely analogous to how, at a criminal trial, the defendant is entitled to defend themselves however they want (within various legal bounds). The defendant isn’t obliged to respond point-by-point to every claim made by the prosecution.
But there’s an alternative possibility, to which I confess I’m partial: structure comment-reply exchanges like peer reviews and replies to reviews. That is, the comment should ideally be structured as a series of numbered points, and the reply has to respond point-by-point to the comment. With the reply to each point being preceded by an extended quotation of the point. Just like how, when you reply to peer reviews, you are expected to quote each point in the review before replying to it, you can’t skip replying to any points (on pain of seriously annoying a good editor), and you can’t reply to any points that the reviewers didn’t actually make (again, on pain of seriously annoying a good editor). Quoting each point before replying prevents authors from ignoring or talking past comments. It also helps prevent authors from accidentally misreading comments on their work. If you want readers make up their own minds, well, shouldn’t you put them in the best possible position to do so, by providing them with a clear series of points and counterpoints?
So I think commenters should be obliged to structure their comments as a series of numbered points. And I think any authors who don’t reply to every point by quoting it and then providing an on-point reply should have their reply rejected. In that case, the comment would be published along with a note from the editor reading “The authors were offered the opportunity to reply point-by-point to this comment. They chose not to do so.” Again, I’m assuming here that any comments that don’t merit a reply will be rejected by the editor, without being forwarded to the authors. Much like how a good editor will do everything possible to ensure that authors receive only thoughtful, professional peer reviews.
Like I said, I can see good arguments for both these possibilities. There’s a bit of a tension between allowing authors to defend their work however they see fit, and ensuring that comment-reply exchanges are maximally informative for readers. How do you think this tension ought to be resolved?
*FWIW (not much), I find Lambert et al.’s comment more convincing than Scheele et al.’s reply. Indeed, it seems to me that Scheele et al. more or less grant Lambert et al.’s criticisms; Scheele et al. just see those criticisms as features rather than bugs. But again, I’m not an expert here!
**Lambert et al. go out of their way to emphasize that they’re not criticizing the use of expert knowledge in conservation, writing “We are not critiquing the importance of expert opinion, but failing to clearly report how and when expert opinion is used impedes conservation efforts.” In reply to which, Scheele et al. write, “Lambert et al. treat expert knowledge…as unreliable (at best) and suspicious (at worst)”.