Yesterday, I polled y’all on robustness checks–different ways of doing a statistical analysis that lead to the same broad conclusion, thereby indicating that the conclusion is robust. I asked whether, as an author, your papers usually include robustness checks, and if so, where (in the main text, in an appendix, etc.). And I asked, as a reviewer or reader, if you usually want authors to include robustness checks, and if so, where.
Here are the poll results so far! They’re pretty interesting, and surprising to Brian (I wasn’t that surprised). So you should totally read on.
As of this writing, we have 88 responses to the question about what you usually do as an author, and 78 responses to the question about what you usually prefer as a reader or reviewer. Presumably, a few respondents overlooked the second question. Thanks everyone who responded! As usual, this isn’t a random sample of ecologists, or even of our readers. But it’s a bigger and likely more diverse sample than, say, your lab group or journal club. So it seems worth talking about.
Here’s a graph of the percentage of respondents choosing each of the options. Blue bars are respondents indicating what they usually do as authors, orange bars are respondents indicating what they usually prefer as reviewers or readers:
Here are the take-home points:
- Ecologists like robustness checks. This is the result that surprised Brian. Only a small percentage of respondents to either question said they usually do or prefer no robustness checks. But I wonder how many authors only do them because reviewers ask for them, or because they think reviewers will ask for them, because…
- Ecologists like robustness checks more when it’s other people doing them. Doing and reporting robustness checks is work. Ecologists are human; we like work best when it happens to other people. 🙂 Over 60% of respondents usually prefer robustness checks to be reported in an appendix when they’re acting as readers or reviewers, vs. only 49% when they’re acting as authors. In other words, there are some ecologists who want others to report their robustness checks in an appendix, but who don’t usually do so themselves. What do those authors do instead? Well, they mostly either do the robustness checks but tell the reader that the results are “not shown”, or else they don’t do the robustness checks at all. Those are the two options that were chosen by more authors than readers/reviewers. Conversely, putting robustness checks in the main text–the option that involves the most work–was the preferred option of more readers/reviewers than authors.
These results remind me of how everyone simultaneously complains that peer review takes so long, and also complains about being asked to do peer reviews quickly. Everybody wants to go to heaven, but nobody wants to die, as the saying goes.
But we shouldn’t be too cynical about this. One way to ensure that everybody does something that few people want to do is to create and enforce a norm that everyone will do the thing. We all hold each other to higher standards than we would hold ourselves to if we were all left to our own devices. That’s mostly a good thing.