Post-publication review: the culture clash continues

tl;dr: Making scientific debate faster can be a good thing, but only in combination with other good things. But the combination of “speed plus other good things” may not be a stable combination, because changes in technology and norms of scientific practice that promote speed also tend to inhibit those other good things.

I have various old posts noting that we’re living through a culture change–and thus a culture clash–when it comes to post-publication “review” (see here, here, and here). The latest example is online discussion of a high-profile PNAS paper by Case & Deaton, looking at variation in human mortality rates across different countries, and across different ethnic groups in the US. Andrew Gelman drove a lot of this discussion (here’s a recent post in his series).

But here I’m more interested in the associated discussion about norms of scientific communication. Andrew got upset with the paper’s authors, who complained in passing that the online discussion of their paper was moving too fast to be an effective scientific discussion.

I appreciate where Andrew’s coming from. But I think he went a bit too far on this one; Case & Deaton have a point. Noah Smith of Noahpinion articulates that point pretty well, as do many of Andrew’s commenters (who seem to disagree with him more on this one than they usually do). But I think Jeff Leek of Simply Statistics articulates it best. It’s a great summary of the culture clash, and includes thoughtful advice on what to do if there’s a fast-moving online discussion about your latest paper. So I’m going to outsource this to him, with a few comments of my own at the end:

It is much, much easier to critique a paper than to design an experiment, collect data, figure out what question to ask, ask it quantitatively, analyze the data, and write it up. This doesn’t mean the critique won’t be good/right it just means it will happen much much faster than it took you to publish the paper because it is easier to do…

The first thing to keep in mind is that the internet wants you to “fight back” and wants to declare a “winner”. Reading about amicable disagreements doesn’t build audience. That is why there is reality TV. So there will be pressure for you to score points, be clever, be fast, and refute every point or be declared the loser. I have found from my own experience that is what I feel like doing too. I think that resisting this urge is both (a) very very hard and (b) the right thing to do. I find the best solution is to be proud of your work, but be humble, because no paper is perfect and that’s ok. If you do the best you can , sensible people will acknowledge that…

I think this route can be the most scientific and productive if executed well. But this will be hard because people will treat that like “you didn’t have a good answer so you didn’t respond immediately”. The internet wants a quick winner/loser and that is terrible for science.

I agree with all of this, and I think Andrew Gelman would too.* As I read him, Andrew wants quick back and forth discussion. But he wants open ended discussion not necessarily leading to a quick resolution, except perhaps with regard to purely technical errors. And he doesn’t want to declare individuals as “winners” or “losers”, taking the sensible view that everyone is fallible.

Trouble is, I’m not sure Andrew, Jeff Leek, and I can have what want. Because the norms of post-publication review suggested by Jeff Leek and Andrew (and me) aren’t the only ones in circulation. And I’m not sure which norms, if any, are going to get widely established in the long run. I say that in part because people adopting more aggressive norms tend to dominate the conversation, which has the long-run effect of crowding out those who’d prefer to adopt less aggressive norms.

Note also that, if you don’t give people the quick response they want in an online discussion, some of them won’t stop at presuming that you don’t have a good response. They’ll also accuse you of having outdated, immoral norms of professional behavior that hold back science as a whole. And since science as a whole is much bigger than any individual scientist, in their minds holding back science as a whole gives them license to attack you personally as well as critiquing your work. Anecdotally, this seems like an increasingly common move in online debates about appropriate professional practice in science: characterizing practices you disagree with, and the people who adopt those practices, as immoral or unethical.

It’s for these reasons that I much prefer online debates about broad issues and widespread practices (e.g., statistical machismo) than debates about individual papers. Not only are broad issues and widespread practices usually more important than individual papers, but discussion about them tends to be more impersonal and slower moving. No one feels singled out by criticism of a widespread practice, and no one feels rushed to respond instantly. Not that that’s a panacea, obviously–I’m sure you can think of online debates about widespread practices that have become heated and unproductive.

Bottom line: I can certainly understand authors who’d prefer to discuss their papers in a “slower” way, via a process governed by agreed norms and procedures and overseen by an editor to enforce those norms.

It’s not all bad news on the normative front, of course. The developing norm to share one’s data and code encourages post-publication debate to be data-driven. That’s a good thing. But it’s not clear to me that data-drivenness crowds out other, less desirable norms of post-publication debate. For instance, data and code sharing didn’t prevent both of the post-publication debates I discussed here from becoming very heated and personal. Indeed, to the extent that data and code are readily shared, that only increases the speed with which “the crowd” can try to pick apart published work, and the speed with which authors are expected to respond.

I struggle with this because in part because I think that more and faster-moving discussion, criticism, and debate in science would be a good thing (e.g., this post and links therein). But not if it’s governed by the wrong norms, or no norms at all. Because then the debate, criticism, and disagreement won’t serve the purposes they should serve in a healthy scientific communication ecosystem.

*As an aside, it’s worth noting that elsewhere Andrew has said he strongly prefers blogs to Twitter. Blogs are of course slow compared to Twitter. So apparently, everyone thinks there’s an optimal speed at which scientific debate should be conducted, and it’s whatever speed they personally happen to prefer. 🙂 By the way, I say that as someone who much prefers blogs to Twitter as a medium of discussion and debate, for the same reason Andrew does. But I try to remember that my reasons for preferring blogs to Twitter are other people’s reasons for preferring peer-reviewed papers to blogs.

10 thoughts on “Post-publication review: the culture clash continues

  1. Jeremy: You write,

    Andrew has said he strongly prefers blogs to Twitter. Blogs are of course slow compared to Twitter. So apparently, everyone thinks there’s an optimal speed at which scientific debate should be conducted, and it’s whatever speed they personally happen to prefer. 🙂 By the way, I say that as someone who much prefers blogs to Twitter as a medium of discussion and debate, for the same reason Andrew does. But I try to remember that my reasons for preferring blogs to Twitter are other people’s reasons for preferring peer-reviewed papers to blogs.

    No. I prefer blogs to twitter not because blogs are slower but because blogs are not limited by space and, as a result, blog discussions are more open-ended. When you have only 140 characters, it’s all about getting off a good quip. When you have unlimited space, the quick quip is no longer king.

    Do I prefer blogs to peer-reviewed papers? Not quite. I think both blogs and papers have their useful roles. What I was objecting to was the claim by Anne Case that blogging is “not the way science really gets done.” That’s a ridiculous claim, and I felt that the work I was publishing on my blog was just as much “science” as the work she had published in PPNAS (and, for that matter, both of our work were much more “science”-like than various notoriously bad PPNAS artifacts such as the himmicanes and hurricanes paper).

    What do I like about blogs compared to journal articles? First, blog space is unlimited, journal space is limited, especially in high-profile high-publicity journals such as Science, Nature, and PPNAS. Second, in a blog it’s ok to express uncertainty, in journals there’s the norm of certainty. On my blog, I was able to openly discuss various ideas of age adjustment, whereas in their journal article, Case and Deaton had nothing to say but that their numbers “are not age-adjusted within the 10-y 45-54 age group.” That’s all! I don’t blame Case and Deaton for being so terse; they were following the requirements of the journal, which is to provide minimal explanation and minimal exploration. Journals typically value the appearance of certainty and penalize you if you express to much openness to alternative possibilities. Similarly for the comparison of trends to men and women. In their paper, Case and Deaton supply the terse and misleading sentence, “Patterns are similar for men and women when analyzed separately.” It was only in a later online magazine interview that Case explained that the patterns were not actually so similar for men and women, but they had a way of understanding these differences. I don’t know that they did any quantitative analysis, though. Again, on the blog I was able to compare men and women in many ways; in the journal, sorry, no space. On the blog i was also able to post graphs showing trends for each single year of age. In the journal: sorry, no space. In an interview with the New York Times, Deaton did talk about trends in single years of age, but he got things wrong! He wrote, “If we want to be more precise about the age range involved, we could say that for all single years of age from 47 to 52, mortality rates are increasing.” But that statement isn’t even correct! He got this by simply looking at changes from 1999 to 2013. If you look at the time series, that’s not a good summary at all. So, over and over again, we’re seeing journal article, or journal-article-followed-by-press-interviews, as discouraging data exploration and discouraging the expression of uncertainty.

    So my reasons for preferring blogs to Twitter are not other people’s reasons for preferring peer-reviewed papers to blogs. Yes, twitter is faster than blog, and blog is faster than journal. But speed does not seem to me to be the key dimension here.

    And let me just emphasize one thing here: Throughout this discussion I have continually, over and over again, recognized the value of Case and Deaton’s work and expressed uncertainty about my calculations. It is they who have continued to dismiss the contributions of various outsiders (not just me) who presented alternative analyses. I find this very frustrating.

    It’s easy to tell a story in which scientific journals are so civilized and online discussion is all about point scoring. But what I’ve seen here is the opposite. The norms of peer reviewed journals such as PPNAS encourage presenting work with a facade of certainty. It is the online critics such as myself who have continued to display a sprit of openness.

    • Thanks for taking the time to comment Andrew.

      “No. I prefer blogs to twitter not because blogs are slower but because blogs are not limited by space and, as a result, blog discussions are more open-ended. ”

      Thanks for the clarification. In practice I think speed and substantiveness tend to go hand in hand. It takes some time and thought to put the space afforded by a blog to use.

      “Do I prefer blogs to peer-reviewed papers? Not quite. I think both blogs and papers have their useful roles. ”

      I agree.

      “What I was objecting to was the claim by Anne Case that blogging is “not the way science really gets done.” ”

      We’ll probably have to agree to disagree here. Like some of your commenters, I think you’re reading Anne Case uncharitably (defensibly, but uncharitably) in reading this passing remark as you have. Speaking as a scholarly blogger, I don’t find her remark the least bit upsetting and didn’t feel insulted. I read it as an understandable broad remark about scholarly communication in general, coming from someone who’s probably been working flat-out dealing with a huge splash of publicity and a bazillion simultaneous demands for comment from all sorts of sources, and who like many scientists probably doesn’t read blogs. I don’t read it as a comment specifically dismissing the value of the analyses you did.

      I also think it’s understandable but a little uncharitable of you to expect someone to judge blogging as a form by the best examples of it, such as your blog. Personally, I can totally see where someone who doesn’t pay much attention to blogs (as many scientists of course don’t, which I think is fair enough) might tend to lump them in with other social media (and PubPeer comments, and etc.). Or might just tend to be wary of blogs out of lack of familiarity. And I wouldn’t expect anyone to totally revise their previous opinion of blogs based on a few interactions with a few bloggers about one paper.

      “PPNAS”

      I know that your joke about PPNAS isn’t meant as a blanket statement that all or even most work published in PNAS is shoddy or overhyped or superficial or whatever. At least I hope it’s not. But I wonder about if that’s how jokes about PPNAS or “the tabloids” come off to at least some scientists who publish good careful work in those journals. Much as Anne Case’s blanket remark about how “science isn’t done” comes off to people like you, who do careful work and publish it on blogs. Yes, the himmicanes paper was ridiculous, and it reflects badly on PNAS to have published it. Just as it reflects badly on blogs as a form when somebody uses them to, say, publish personal attacks on the competence and professional integrity of others. Attacks that would be very unlikely to get past an editor at PNAS. Put another way, I think you’re overgeneralizing about “online critics”. I think you’re (presumably unintentionally) focusing narrowly on those critics who conduct themselves well. and who create light rather than heat. I think that’s just as understandable–and just as incorrect–as when someone dismisses all scholarly blogging out of hand because they know of cases in which online critics have conducted themselves poorly and created heat rather than light.

      Blogs and journals are both just tools. Like any tools, they can be used well or badly. And like any tools, they don’t just get used badly in random or idiosyncratic ways. Rather, they have “failure modes”–the ways in which they get used badly tend to run to type. The virtues and vices of any tool tend to be two sides of the same coin, so that it’s very difficult to have the virtues without the vices. I don’t like it when somebody focuses on the vices of blogs, but I don’t think you can address their concerns by just focusing on the virtues, or on the vices of other means of scholarly communication. I think gaining wide acceptance of blogs as a means of serious scholarly communication is going to require more than (i) leading by example and doing good scholarly blogging, (ii) highlighting cases in which scholarly bloggers do a good job, and (iii) highlighting cases in which journals do a bad job. I think that three-pronged argument has obvious holes. Though I confess I’m not sure how easy it will be to plug those holes. If it were easy to have the virtues of blogs without the vices, we’d all have done it by now. So I dunno. Maybe it would help to also highlight cases in which scholarly bloggers do a bad job, as a way of establishing and enforcing norms of good scholarly blogging?

      One thing I am sure of is that getting upset and berating Anne Case and others who say similar things isn’t going to change any minds. It won’t cause anyone to think more highly of blogs than they currently do, and will just confirm some people’s fears. I too disagree with her passing remark about how science is done. But her remark has understandable sources–I can see why an intelligent person would say what she did. I think it’s better to address the source of that sort of remark in a sympathetic way, if only as a matter of pure tactics. But then, I’m assuming here that one of your goals is to convince people like Anne Case to take scholarly blogs seriously; perhaps it’s not among your goals.

      But in saying all this, I think I’m just saying things that Noah Smith and various commenters on your blog have already said (as your comment reiterates things you’ve said before). So we may just have to agree to disagree.

  2. Thanks for sharing all this. You state “It’s for these reasons that I much prefer online debates about broad issues and widespread practices (e.g., statistical machismo) than debates about individual papers.” I will admit to learning more from Gelman’s specific examples than from general debates. For example, concerning multi-model inference and AIC posts here, I would have benefited from more detailed critique of specific papers illustrating best practice. Unfortunately, it’s hard to do this without insulting or at least embarrassing an author; we are all human (and we all very in brain wiring and some of our brains are wired to be less sensitive to feelings in the way we choose our language, or it takes much more mental work to edit in this sensitivity). I suppose one could use as examples simulated data but then apply actual good or bad practices lifted from a paper (without citing the paper).

    • “For example, concerning multi-model inference and AIC posts here, I would have benefited from more detailed critique of specific papers”

      Yes, good point. It’s hard to talk about general principles without also talking about specific examples.

    • See Cade 2015, Model averaging and muddled multimodel inferences, in Ecology 96: 2370-2382 for a general critique followed up with specific examples. None of the authors critiqued (Burnham and Anderson, and Rice et al) are very happy with me. But I did try to engage them all in constructive critique as I was writing my paper.

  3. The journal BRAIN AND BEHAVIORAL SCIENCES has had post publication review ( called peer commentary) for almost 40 yrs, and the authors of a target article respond to the reviews. There are, of course, limitations on the reviews, enforced by the editors ( they are peers after-all), but a wide range of scholars are invited to comment. The journal publishes some very influential articles.

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s