A couple of recent incidents have gotten me thinking about how norms of scientific criticism are changing. With one side effect being nasty fights.
Here’s the first incident (ht Retraction Watch). Computational biologist and blogger Lior Pachter has a series of posts strongly criticizing the network research of A.-L. Barabási, Manolis Kellis, and others. This is very prominent work, reported in a long string of papers in Nature and other top-tier journals. In particular, Pachter digs into one recent paper by Kellis and co-authors in great detail and argues that it’s “nonsense”. And he goes further and accuses Kellis and co-authors of misconduct: being intentionally-vague about their methods in order to get the paper into a leading journal and then trying to cover their tracks by “correcting” the paper post-publication. Pachter has taken to his blog because his correspondence with the authors left him unsatisfied and journals rejected the comments he submitted. A reply from Kellis is here, and a reply to the reply is here.
The second incident is the recent dust-up over some prominent papers on dinosaur growth rates. Nathan Myhrvold, a millionaire dinosaur hobbyist, has strongly criticized a series of papers by paleobiologist Gregory Erickson and colleagues for numerous omissions of methodological details that make it impossible for others to fully reproduce the data analyses. And also for various mistakes (including two different figures in which the fitted curve shown in the figure doesn’t come close to matching the equation provided in the figure legend). Myhrvold has described his concerns publicly and in great detail in a peer reviewed paper, and apparently has had private correspondence with Erickson. But he also took his criticisms to the New York Times, and in an interview with the Times said the errors in Erickson’s work were “consistent with scientific misconduct”. And while some of the co-authors of the criticized work have said that Myhrvold has identified some important issues, Erickson’s public response has been limited to noting that his papers were peer-reviewed.
I’m not qualified to judge the first dispute, and while I do feel I’m qualified to judge the second I don’t really want to get into that.* I just found both incidents to be striking signs of the times. Whatever the rights or wrongs of these two particular incidents, this kind of incident is going to become more common, I think. Because like Bob Dylan sang, the times they are a changin’. What follows are my admittedly anecdotal impressions and probably-rash generalizations. I’m tossing them out there in the hopes that others will chime in with their own impressions.
It seems like more and more people are increasingly demanding about the level of detail in which published work should be documented. Are increasingly less likely to see any mistake or omitted detail as minor. Instead taking the view that all mistakes and omissions are serious by definition, or at least that they should all be corrected and their seriousness left to the reader to judge. Are increasingly disinclined to trust authors, editors, or the pre-publication peer review process, preferring to at least have the in-principle option of checking everything themselves.** Are increasingly disinclined to care about voicing post-publication criticisms through the “proper” channel (i.e. by submitting formal comments for peer review). Instead taking the view that once something’s published, it’s fair game. And that peer review of comments is at best too slow, and at worst functions as a way to suppress legitimate criticism. So that it’s better to just publish criticisms of peer-reviewed papers in whatever venue seems most convenient and visible. And are increasingly comfortable with strongly-worded language, and disinclined to care about tone. Indeed, often viewing worries about tone as a distraction from discussion of substantive issues.***
I’m actually not sure if more and more people feel this way, or if thanks to the internet people who feel this way are merely more visible and vocal than they used to be. Could be some of both. In any case, I do think times are changing, and that they’re going to keep doing so. So if you don’t like the way things are changing, well, for better or worse I think you’re going to end up on the wrong side of history in the long run.
We’re living through a culture clash, I think. People who feel more or less as described above certainly can give reasons for thinking that science would be better off if everyone felt the same. And conversely, people who don’t feel that way can give reasons for thinking that science will be better off if everyone felt as they do. And I actually think everyone’s reasons (including mine!) mostly are post hoc rationalizations. For instance, people who are used to scientific communication working a certain way would like it to continue working in that familiar way. And people who are used to other things working a certain way (e.g., they’re used to the informality and openness of social media) would like scientific communication to work in that familiar way. Put it this way: do you often see people arguing that scientific communication should change in a way that they personally would hate? And culture clashes are a recipe for arguments.
But having said that, just because people’s proffered reasons are post hoc rationalizations doesn’t mean they’re merely post hoc rationalizations. It’s by giving reasons that we can (sometimes) get others to understand where we’re coming from and appreciate our own point of view. Even if we don’t ultimately hold that point of view for rational reasons. And through trying to rationalize your point of view and thereby justify it to others with different preferences, you’ll often come to realize that you are rationalizing. Which is a useful thing to recognize. So if indeed there is a culture clash here, I don’t think that means there’s no hope for mutual understanding or compromise.
On balance, I think the trends I’ve described are good for science, but my feelings are somewhat mixed. Which isn’t surprising, since I’m something of a ‘tweener when it comes to novel ways of communicating and criticizing science. So here are my post hoc rationalizations for my own views.
I think the bar always gets raised rather than lowered in science. Our standards for everything go up over time, and that’s good. Right now, the bar is getting raised on the detail and completeness with which authors are expected to report their methods. I think that can only be good for science.
I also think it’s good for science if we all get more comfortable with mutual criticism and vigorous debate. As Andrew Gelman recently remarked, the only way for the vaunted self-correcting nature of science to work is if we actually allow corrections. Various lines of evidence show that science self-corrects more slowly than it could or should (e.g., this quite interesting paper I just found [UPDATE: link fixed]). And I don’t like the view that any post-publication criticism of peer reviewed work necessarily is nitpicking, and in any case is never worth publishing because any seriously flawed work will be ignored and so needn’t be publicly criticized. I think that view (which is common though far from universal) is based on false premises, and holds back scientific progress. And I don’t think we need to worry about most authors having to deal with a barrage post-publication criticisms. The vast majority of papers aren’t read often enough or carefully enough to attract any post-publication criticism. (Which still leaves significant concerns about the minority of authors who do have to deal with such criticisms; see below)
And I think it’s a good thing on balance if people have less ability these days to dismiss criticisms on the basis of the critic’s credentials or the venue in which the criticisms were published, or any other basis besides the substantive content of the criticism itself. Science is supposed to be all about logic and evidence. Logic and evidence don’t care what credentials you have, or the venue in which they’re published, or etc.
On the other hand, I do think we need some professional norms or other about post-publication review. Because otherwise post-publication review isn’t going to work well. Pre-publication review works because everybody agrees to participate voluntarily. It’s part of what you’re signing up for when you choose to do science. But while at some level everyone’s aware that once they publish something others are free to read and react to it as they see fit, lots of people don’t feel like they signed up for post-publication “review”, at least not in all its forms. I mean, do you feel like being accused of serious mistakes and possible misconduct in the New York Times just “comes with the territory”?
Pre-publication review also works because it’s a system with formal and informal rules, not a free-for-all. For instance, it’s private, which saves people from the embarrassment of having their mistakes pointed out publicly, thereby encouraging them to participate. And there’s an editor involved, which helps ensure (though of course doesn’t guarantee) that referees and authors address each other professionally, and obliges authors to respond point-by-point to referees rather than just ignoring them or whatever. The fact that it’s a system with rules is a big part of what encourages people to participate voluntarily. People know what they’re signing up for.
Which is why I don’t think you can just tell people to “have a thick skin” or “don’t take it personally” when it comes to post-publication review. As this very good piece about another recent post-publication review dust-up points out, post-publication review these days seems to slip quite easily into personal criticism of the author. Which isn’t something you should be expected to “have a thick skin” about. Nobody should be expected to “have a thick skin” about implications or accusations that they’re incompetent or unethical. And further, sufficiently-strong criticism of someone’s work is awfully hard to distinguish from an attack on their competence or integrity, whether or not you preface it by saying “Don’t take what I’m about to say personally” (again, see the very good piece I just linked to).**** Further still, just as authors sometimes make mistakes, sometimes because they’re incompetent or unethical, so do critics. It seems like advocates for “anything goes” in post-publication review presume that critics are always right. They’re not. So what’s needed isn’t for everyone to have a thick skin. A thick skin is what you need to survive perceived or actual attacks. What’s needed are new norms and practices that everyone buys into.***** So that, as far as reasonably possible, people don’t get attacked and don’t feel they’ve been attacked, and so don’t need to have thick skins.****** For instance, in the post linked above, Andrew Gelman offers some suggestions on changes to how journals publish criticisms. In the absence of widely-accepted norms and practices, effectively critiquing the published literature is a really tricky thing to do.******* This isn’t (just) about sparing people’s feelings or protecting their reputations–it’s about creating a post-publication review system that actually works. That actually does effectively correct the scientific record, and that is seen by all to do so.
I don’t have any answers here–I don’t know what the new norms and practices will be, or should be. And I don’t know how to create a set of norms and practices that everyone would buy into and happily participate in. Attempts to codify rules of good commenting behavior on the internet are infamous failures. So we may have no alternative but just to let things sort themselves out through some sort of quasi-evolutionary process. But until we have wide agreement on new norms and practices, expect more dust-ups like the two described above.
*For the record, I’ve read Myhrvold’s paper carefully, and I feel like I’m competent to judge it because his concerns all relate to data processing and statistical analysis. Assuming that he’s truthfully described what he did (and he does seem to have described it in sufficient detail for anyone to reproduce it), then I think he has indeed identified some mistakes that really ought to be corrected. And while the mistakes may not alter the broadest qualitative conclusion (“dinosaurs grew fast”), the quantitative changes seem big enough to me to be more than trivial. As for the various methodological omissions he’s identified, I suspect that this is an example of changing reporting standards. The work was originally reported in enough detail to satisfy the referees that it was done competently. But like an increasing number of readers, Myhrvold wants to see methods reported in sufficient detail so that others can exactly reproduce all of the results without any need to contact the authors.
**E. D. Deming supposedly said “In God we trust; all others must bring data.” That’s never been more true and it’s going to get truer.
***Although there’s an irony, noted by a commenter on Pachter’s blog, that people who feel comfortable expressing even minor criticisms in strong language sometimes also complain about authors overhyping their results. Leaving one wondering why it’s not ok to hype results, but is ok to hype criticisms of those results.
****I’m probably one of the guilty parties here. Probably one mistake I’ve made at times in the past has been phrasing criticisms of people’s ideas so forcefully that it sounds like I question the competence of anyone who disagrees with me. Even though I’m usually careful to say that I’m not questioning anyone’s competence.
*****Some would disagree. For instance, the comments on that piece I just linked to include the view that not only will scientific debates get personal sometimes, but that they should get personal. I don’t buy that at all. But the fact that that view is even out there is another sign of the times.
******A bit of a thick skin is always going to be needed, no matter what our norms and practices are. Some people find pre-publication review quite hard to take. Which is a feeling they have to learn to get over, since pre-publication review is standard practice.
*******”Effectively” is a key word here. If you want your post-publication critique to actually convince people rather than falling on mostly-deaf ears, you can’t just voice it however you want and then write off anyone who doesn’t like your tone as a wimp. If you want to be convincing, you have to write for the audience you have, not the audience you wish you had. See this old exchange of comments.