Friday links: when you stop believing your own results, and more

Also this week: sympathy cards for scientists, Pixar movies vs. your next talk, journal of critiques?, reviewing the literature on double-blind review, and more

From Jeremy:

At Methods Blog, Bob O’Hara reviews the evidence on the effects of double-blind review. Comes to broadly the same conclusions as another review I’ve linked to in the past, by Hilda Bastian. The main take homes, from both reviews combined:

  • There aren’t many studies, and they’re often are low-powered or have design issues
  • Author blinding is seen through for anywhere from ~25% to ~75% of papers
  • Double blinding doesn’t reduce gender and other biases, and may well create or exacerbate them. The reason is that author blinding gets seen through non-randomly. For instance, Bob O’Hara notes that if reviewers who are blinded to author identity tend to be slightly more negative (and there is some evidence for this), and blinding tends to get seen through for papers by well-known authors, then double-blinding may favor well-known authors. Who tend to be male, white, based in certain countries, based at certain types of institutions, etc.

Just to comment a bit: I’m in favor of Am Nat’s experiment with double-blind review, and look forward to seeing the results. More broadly, journals should do all they can to prevent bias in the review process. It’s think it’s precisely because the issue is so important that it’s important that journals take measures that work, and that don’t have perverse effects. The very real possibility of perverse effects is an argument against widespread implementation of double-blind review on the grounds that it wouldn’t hurt. Quoting from Hilda Bastian:

Author and peer reviewer anonymity haven’t been shown to have an overall benefit, and they may cause harm. Part of the potential for harm is if journals act as though it’s a sufficiently effective mechanism to prevent bias.

Which of course doesn’t mean that we should just throw up our hands. I take heart from the fact that it is possible to greatly reduce and even eliminate gender bias throughout the peer review process without going to double blinding, as the examples of Functional Ecology and the New Zealand Journal of Ecology illustrate. I’d like to see more journals go through the same exercise of looking very hard at their own data for evidence of bias throughout the peer review process, and then publishing the results. I think that doing that has value not just in identifying and quantifying any problems, but also as a public demonstration of taking the issue of bias seriously. Finally, I say all this as someone who hasn’t ever been held back by any biases that might exist in the peer review system. Which might well cause me to overweight certain arguments against double-blind review, and underweight arguments for it. Looking forward to comments, as always.

Do we need a new ecology journal dedicated solely to critiquing the literature? Given that I’m famous (infamous?) for banging on about zombie ideas, the need for more short-selling of scientific ideas, the value of contrarians, etc., it might surprise you to learn that I have mixed feelings about the idea of a journal of critiques. Mostly for reasons Brian has discussed. But also because I think it’s rare for individual papers to be important enough to need public critiquing, and those rare papers can’t be reliably identified when they’re first published. You can’t tell in advance what will go on to become influential enough to need critiquing. I continue to think ecologists collectively are too slow to discard zombie ideas and too quick to jump on bandwagons. But I’m not sure a journal of critiques would make any difference. If it did, it would be a long-term difference, by changing the culture of the field in some small way. (ht @ucfagls)

Following on from the previous link: Data Colada with three concrete suggestions for how to publicly criticize others’ scientific work constructively.

How to structure your talk like a Pixar movie. Excellent advice.

From Meg:

Dana Carney, one of the authors of the “power pose” paper, now says she doesn’t believe power pose effects are real. This document where she describes the evolution of her thoughts on power poses is remarkable.

Sympathy cards for scientists

These ecology & evolution activities (some of which relate to ongoing citizen science projects) for middle school students seem really neat! (ht: Holly Menninger)

4 thoughts on “Friday links: when you stop believing your own results, and more

  1. That Carney document is indeed remarkable. She also added blurbs on her cv indicating that she no longer believes the papers in question.

    Part of what’s remarkable about that document is that she explicitly argues against the “power pose” effect being context dependent or otherwise hard to study and saying “more research is needed”. Which are the usual things people seem to say when some high-profile result fails to replicate, but they want to keep on keepin’ on. Instead, she comes right out and says there’s nothing there to find, continuing to work on this is a waste of time, nobody should even be teaching or citing work on this topic.

    I’m very curious what her co-authors on that power pose work think of her stance.

    Carney’s been widely and rightly praised for taking a brave stance. A small illustration of why it’s brave:

    Ok, just one anonymous internet commenter. But that individual surely isn’t the only person who thinks in that appalling way.

  2. Thank you for the presentation link! My feeling is that it (the advice) is really solid and I will certainly give the approach a try for my next talk.

    RE: critiquing the lit… No, maybe not a journal. And maybe focusing on individual papers isn’t too productive. However, discussing zombie or controversial theory/methods sure is productive but is not done nearly enough in my experience. Which is one reason why I stay tuned into this blog!

    From my perspective, increasing the amount of discussion and focus on these issues is a good thing. How to do so in a focused and constructive way is a question. DE is one way for sure. What about a group of talks at meetings? Or department seminars?

    • The idea of a “zombie ideas” symposium at the ESA meeting is one that hadn’t occurred to me. I’ll have to mull that over. I think if you did it right it could be very interesting, though it would inevitably annoy or even upset a few people.

  3. Pingback: Redacted ecology faculty search | Dynamic Ecology

Leave a Comment

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.