Also this week: the boy who cried wolf vs. type I errors, pre-registered replication vs. stereotype threat, update on double-blind reviewing at Am Nat, myths of scientific software, scientific texts vs. Google Ngrams, and more.
From Meg:
How do you use the time before class begins? I enjoyed this piece with three ideas for things to do during that time. It might be hard to scale some of these up to a 300 person lecture, but it might be worth trying. I also like that he talks about how, early in his career, he worried about being able to fill an entire class period, but now he feels like there isn’t enough time to cover what he wants. That has been true for me, too.
Type I error vs. type II error, as explained by the boy who cried wolf. This should help students remember the difference!
From Jeremy:
I remain fascinated with the ongoing rapid shifts in practice in psychology, towards pre-registered replication. Social psychology results on “priming” appear to be having a particularly rough time of it. Effects supported by dozens of statistically-significant results across bunches of papers vanish entirely when you try to find them with a high-powered, pre-registered replication. Which is perhaps not surprising in retrospect, since in at least some cases funnel plots of previously-published results just scream publication bias + p-hacking. I note this in part because experimental studies of “stereotype threat” are one example of priming. Meg’s done a great job in the past reviewing the existing literature on stereotype threat. As best I can tell (since my knowledge here is limited to what I read on blogs), pre-registered replications, more powerful studies, and formal meta-analyses on stereotype threat are just starting to come in (e.g., Müller & Rothermund 2014, Gibson et al. 2014, Moen & Roeder 2014, Ganley et al. 2013 [discussed here from a priming-skeptic point of view], Flore & Wicherts 2015). Those new studies are a mixed bag, but overall they don’t look great for the existing literature on stereotype threat. It’s early days though, too early to say whether the conclusions of the existing literature on stereotype threat need substantial revising. And I’m no expert, happy to be pointed towards relevant studies I missed in my skim. Finally, hopefully it doesn’t need saying, but my comments here are concerned with one rather narrow issue. We should still be alert to subtle biases, should still teach our students that they can improve, should still be mindful of sending unintended messages, etc. Indeed, those things still sound like good ideas to me independent of the experimental literature on stereotype threat.
An update on Am Nat’s experiment giving the authors the option of double blind review. 16% of authors opt out of blinding, mostly because they think their identity is too obvious to bother trying to hide it. Gratifyingly (and slightly surprisingly, to me), there’s been no uptick in people declining to review. Reviewers do often see through the blinding, at least to the level of guessing the lead author’s lab–but occasionally they only think they do (i.e. they report incorrect guesses of author identity to the editor). Too early to say yet if the new policy is making any difference to which papers get accepted, though one handling editor thinks that famous people are now getting tougher reviews. (ht Trish Morse, Am Nat Managing Editor, via the comments).
The increasing prevalence of scientific texts in the Google Books database makes it tricky to use Google Ngrams to capture broader cultural shifts in word use. But it makes it easier to write silly, nerdy Ngrams-based blog posts. 🙂 (ht Neuroskeptic)
Jabberwocky Ecology lives! Morgan Ernest weighs in on me vs. Charley Krebs on microcosms, turning the discussion towards related issues.
This is a few months old but I missed it at the time: the myths of bioinformatics software. Applies to scientific software more broadly. The myths include “somebody will build on your code”, “if you choose the right license more people will use and build on your program”, and “you used the right programming language for the task”.
In a recent linkfest I noted a new R package, statcheck, that spots statistical reporting mistakes in psychology papers. Here’s the creator’s story of how the package came to be. Unfortunately it won’t work with most papers from other fields, unless they report statistical results the same way psychology papers do.
The Browser has posted its nominees for the best online writing of the year. You can vote for your favorites. There’s a piece from Andrew Gelman on the list, and a piece on teaching Bayes’ Theorem with lego that I’ve linked to in the past. It’s nice to have one’s taste validated. 🙂 There are other popular and semi-popular science pieces on the list. I liked this piece on why the world will only get weirder (though I disagree with the libertarian/anarchist direction the author takes it at the end), and this one on why you often can’t apply game theory to real life. I didn’t look at many of the non-science pieces, but this one on “the Copenhagen interpretation of ethics” is thought provoking. (ht Andrew Gelman)
xkcd on why biology is more than just gene sequencing. 🙂
And finally, Trudeau considers re-muzzling Canadian scientists after 3 hour conversation about rare seaweed. 🙂 (ht Not Exactly Rocket Science)
Dynamic Ecology 
Hi Jeremy,
About the software link – there is a paper about that. I don’t know if you have seen this:
http://biorxiv.org/content/early/2015/11/16/031930
“16% of authors opt out of blinding, mostly because they think their identity is too obvious to bother trying to hide it. ”
– Interesting interpretation. My interpretation is that people opting out want to make their identity know to reviewers.
“Interesting interpretation”
My understanding is that that’s not “interpretation” on the part of Trish–it’s what authors who opt out of blinding have been telling her.
I suppose those authors could be fibbing, or have subconscious motivations they’re unaware of. But I’m inclined to take author self-reports of their reasons for opting out at face value.
Of course reading other people’s mind is challenging, but “they would recognize me anyway, so I opt out” is less credible “I want them to recognize me, so I opt out”. For the author, there are potential advantages for the second, none for the first. Clearly in the “why have you decided to opt out?” the second reason is not as nice as the first one. Minor point.
With respect, I think you’re far too quick to attribute ulterior motives here, and giving far too much weight to the incentives that you think senior ecologists face. I know a lot of famous senior ecologists. I can imagine many of them saying “Meh, they’re gonna know it’s me anyway, so I’m just going to leave my name on it like I’ve always done.” I can’t imagine any of them saying “I want my name on there because I want the reviewers to know it’s me and defer to my fame.”
Can you elaborate on your reasons for your suspicions here? Perhaps I’m misreading you (in which case my apologies), but it sounds to me like, in the absence of other information, your default assumption is to be suspicious of your colleagues’ motives and honesty. Which if so, strikes me as a default stance that would make it quite difficult to function as a scientist. How can you function as a scientist if your assumption is that anyone with an opportunity and incentive to be dishonest (which is most scientists) might well have been dishonest? I don’t mean to keep pressing you on a minor point, but I am genuinely curious.
Jeremy, maybe because as you stated, “one handling editor thinks that famous people are now getting tougher reviews.”
The assumption that some people chose (whether consciously or subconsciously) the option that slightly improves their chances of a favorable review seems reasonable. And if pressed for an answer on why they chose that option, is it really that hard to believe someone wouldn’t want to admit to an editor that “I want my name on there because I want the reviewers to know it’s me and defer to my fame.”?
Jeremy, it is pretty much what Benjamin said (thanks). I wrote quite a long comment anyway.
I see no reason for people to say “they’d recognize me anyway”. What does it mean? It is neither a prank nor an anonymous gift to your spouse. It is not that as a scientist you are trying to hide behind the double-blind review like it is something shameful (better coming clear before they find out who I am!). I am pretty confident (in terms of hypothesis, as I wrote before reading people’s mind is challenging) that those who waived were neither junior scientist nor women (largely), but more senior scientists. For instance, I am one of the very few people (3, 4?) working on a species and the only one doing modeling. I would not even think about waiving the double-blind because they would recognize me anyway.
You wrote: “How can you function as a scientist if your assumption is that anyone with an opportunity and incentive to be dishonest (which is most scientists) might well have been dishonest? ” — I am not offended by this question and I think I functioned pretty well so far as a scientist. First, it is not everyone, but (in this case and tops) 16%. I think a large part of those 16% waived to have better chances of favorable review. Not something I think is dishonest, science is human activity and publications are the currency. Like the vast majority of people trying to put somewhere a connection with climate change in the paper when clearly there is zero connection. But let’s say those 10% are dishonest. It seems to be an average proportion if you think, as you should in my opinion, that science is something human. One day you find out that some scientists have falsified data, another that a scientist at UCB has been some kind of predator for years and years and apparently the thing went unnoticed. All scientists like you and me. To me, that some people try to take advantage from their fame and they do not disclose the real motives seems a minor, although quite obvious point. I had collaborators “asking” me to forget about some data, others who had no idea where the data were coming from but let’s try to model things anyway, as a reviewer I had multiple times fellow scientists refuse to share data and code (which by the way, I think it happens more often than not because scientists are afraid of having made mistakes), all sort of weird interactions with fellow scientists (a group I highly respect, also because I am part of the group). In this specific case, the alternative hypothesis (they would find out who I am anyway) seems pretty absurd.
As I brief follow up, since I do not want to go off topic, p-hacking, using researchers degrees of freedom with a certain ease, the spike of p-values in the neighbor of 0.05 all lead to a simple question: are some scientists with long careers and plenty of training very naive or they are trying to “game” the system? And I am pretty confident that just a few of them would say: you are right, I tested 100 hypotheses and I published a paper on the one with a p-value <= 0.05 without making any reference to the other 99 in the paper. Clearly there are incentives for publishing interesting, "significant" results, you wrote many times about this (and so Gelman, for example). Certainly I am not saying that every scientist does that (not even the majority), but that the above events are not the 1 in 1000 kinda thing seems quite evident.
Hi . I was going comment when I first read the above comments but shied away mostly because I could see both sides of the debate made sense in some ways. Its is certainly interesting. I was reminded of them when watching a plenary at the Australian ESA conference (unfortunately not recorded but the storify is here https://storify.com/EcolSocAus/equityinecology and further in might shed some light) and that talk really convinced me that unconscious bias means that there shouldn’t be an opt out option that is really going to mostly benefit senior white males; even if unintentionally. I have been thinking about it quite a bit lately and have to agree with most of Simone’s points (and Ben’s) in the additional comments below. Though i’d say there are probably angles I am not even aware of that could be brought into the debate on both sides.