Friday links: when you stop believing your own results, and more

Also this week: sympathy cards for scientists, Pixar movies vs. your next talk, journal of critiques?, reviewing the literature on double-blind review, and more

From Jeremy:

At Methods Blog, Bob O’Hara reviews the evidence on the effects of double-blind review. Comes to broadly the same conclusions as another review I’ve linked to in the past, by Hilda Bastian. The main take homes, from both reviews combined:

• There aren’t many studies, and they’re often are low-powered or have design issues
• Author blinding is seen through for anywhere from ~25% to ~75% of papers
• Double blinding doesn’t reduce gender and other biases, and may well create or exacerbate them. The reason is that author blinding gets seen through non-randomly. For instance, Bob O’Hara notes that if reviewers who are blinded to author identity tend to be slightly more negative (and there is some evidence for this), and blinding tends to get seen through for papers by well-known authors, then double-blinding may favor well-known authors. Who tend to be male, white, based in certain countries, based at certain types of institutions, etc.

Just to comment a bit: I’m in favor of Am Nat’s experiment with double-blind review, and look forward to seeing the results. More broadly, journals should do all they can to prevent bias in the review process. It’s think it’s precisely because the issue is so important that it’s important that journals take measures that work, and that don’t have perverse effects. The very real possibility of perverse effects is an argument against widespread implementation of double-blind review on the grounds that it wouldn’t hurt. Quoting from Hilda Bastian:

Author and peer reviewer anonymity haven’t been shown to have an overall benefit, and they may cause harm. Part of the potential for harm is if journals act as though it’s a sufficiently effective mechanism to prevent bias.

Which of course doesn’t mean that we should just throw up our hands. I take heart from the fact that it is possible to greatly reduce and even eliminate gender bias throughout the peer review process without going to double blinding, as the examples of Functional Ecology and the New Zealand Journal of Ecology illustrate. I’d like to see more journals go through the same exercise of looking very hard at their own data for evidence of bias throughout the peer review process, and then publishing the results. I think that doing that has value not just in identifying and quantifying any problems, but also as a public demonstration of taking the issue of bias seriously. Finally, I say all this as someone who hasn’t ever been held back by any biases that might exist in the peer review system. Which might well cause me to overweight certain arguments against double-blind review, and underweight arguments for it. Looking forward to comments, as always.

Do we need a new ecology journal dedicated solely to critiquing the literature? Given that I’m famous (infamous?) for banging on about zombie ideas, the need for more short-selling of scientific ideas, the value of contrarians, etc., it might surprise you to learn that I have mixed feelings about the idea of a journal of critiques. Mostly for reasons Brian has discussed. But also because I think it’s rare for individual papers to be important enough to need public critiquing, and those rare papers can’t be reliably identified when they’re first published. You can’t tell in advance what will go on to become influential enough to need critiquing. I continue to think ecologists collectively are too slow to discard zombie ideas and too quick to jump on bandwagons. But I’m not sure a journal of critiques would make any difference. If it did, it would be a long-term difference, by changing the culture of the field in some small way. (ht @ucfagls)

Following on from the previous link: Data Colada with three concrete suggestions for how to publicly criticize others’ scientific work constructively.

How to structure your talk like a Pixar movie. Excellent advice.

From Meg:

Dana Carney, one of the authors of the “power pose” paper, now says she doesn’t believe power pose effects are real. This document where she describes the evolution of her thoughts on power poses is remarkable.

Sympathy cards for scientists

These ecology & evolution activities (some of which relate to ongoing citizen science projects) for middle school students seem really neat! (ht: Holly Menninger)