Also this week: the replication crisis vs. significance testing, work vs. thoughts of work, Peanuts vs. Superman (but not that Superman), and more.Continue reading
The National Science Foundation’s Office of the Inspector General (NSF OIG) investigates misuse of NSF funds by individuals and organizations that receive awards from or conduct business with NSF. That includes, but isn’t limited to, researchers who commit scientific misconduct in the course of NSF-funded research. The OIG reports on its activities to Congress every 6 months. The reports are public, you can read them online. As part of my ongoing efforts to educate myself about scientific misconduct, I skimmed the five most recent OIG reports. Here’s what I learned.Continue reading
When a scientist is credibly accused of serious scientific misconduct, one or more institutions–often but not always the accused’s employer–launches a formal investigation. How long do those investigations typically take? Here’s some data I compiled.Continue reading
Also this week: yes of course Jonathan Pruitt is still collecting retractions like they’re going out of style, women vs. natural philosophy, Coursera IPO, in praise of arbitrary journal formatting requirements, people-first labs vs. results-first labs, oceanography vs. math, Excel will never die, and more. Lots of good stuff this week!Continue reading
Yesterday, I polled y’all on robustness checks–different ways of doing a statistical analysis that lead to the same broad conclusion, thereby indicating that the conclusion is robust. I asked whether, as an author, your papers usually include robustness checks, and if so, where (in the main text, in an appendix, etc.). And I asked, as a reviewer or reader, if you usually want authors to include robustness checks, and if so, where.
Here are the poll results so far! They’re pretty interesting, and surprising to Brian (I wasn’t that surprised). So you should totally read on.Continue reading
There often are different ways of doing a statistical analysis, all of them defensible. Doing the analysis with vs. without an outlier. Doing a general linear model on transformed data vs. doing a generalized linear model on untransformed data. Addressing collinearity by dropping a collinear predictor, vs. doing some sort of formal model selection, vs. doing a PCA on the predictor variables and using the PCA axes as new predictor variables. Deciding how many terms your statistical model should include. Etc. etc.
Sometimes, different statistical choices lead to different results. This sensitivity of results to different statistical choices is known as “researcher degrees of freedom“. Researcher degrees of freedom can make it hard to choose the “right” statistical analysis, and can lead to arguments among researchers as to what the “right” analysis is.
But what about the case in which different defensible statistical choices all lead to the same results? That is, cases in which the scientific conclusion is robust to different statistical choices? It’s tempting to say that robustness makes your statistical choices easy. Just choose whatever analysis you want, because your choice doesn’t matter.
But here’s the problem: reviewers and readers won’t necessarily believe that your results are robust if you only show them one analysis. They’ll ask “Did you correct for [thing]?” “Are you sure the results aren’t driven by [small subset of data]?” “Wouldn’t it be more rigorous to do [alternative analysis]?” That’s why, in some fields (economics is one), it’s routine for papers to include “robustness checks”, also known as “alternative specifications”. You do the analysis in a bunch of different ways, and show that they all lead to the same conclusion. Robustness checks aren’t routine in ecology. Should they be?
Maybe not. After all, one could take the view that the whole point of a scientific paper is to tell one story. A scientific paper shouldn’t be a Choose Your Own Adventure. It’s the authors’ responsibility, and privilege, to argue for their scientific conclusion however they think best. Plus, the authors can’t possibly anticipate all the alternative ways in which readers might’ve wanted them to analyze the data. So if, as a reviewer or reader, you wonder how a different story would’ve turned out, well, that’s your problem. Go download the data (which, these days, the authors are probably required to share on a public repository), and conduct your own robustness checks.
And if robustness checks should be routine, where do they belong? In economics, they go in the main text of the paper. Which some economists complain about, because it makes the paper more difficult and boring to read. Alternatively, one could put robustness checks into an online appendix. But we all know that nobody reads online appendices–often even the reviewers don’t.* A third option is to not write up the robustness checks, but instead share code that will allow any curious readers to run the robustness checks if they want to. A fourth option is just to ask readers to trust you. Your paper can describe the alternative analyses you ran, and then say “These alternative analyses (not shown) led to the same conclusions as the main analysis, indicating that the results are robust.”
Which option do you usually take as an author? And which one do you usually prefer as a reader? Take the poll!
*Heck, once in a while you can put robustness checks in the main text of the paper and some readers will still overlook them.
Meghan has a fun old post asking what is, or will be, your “old school science cred“. The scientific thing you’ve done, or will do, that will one day cause future grad students to look at you and think “Jeez, you’re old.”
Here’s a way to get “old school science cred” that didn’t make it into that post: cite a “personal communication” from someone. Citing “personal communication” was a thing back in the day–but not so much any more.
Here’s a table of the number of papers in Ecology that include at least one instance of the phrase “personal communication”, in 5-year chunks (sorry, too lazy to make a graph):
1981-1985: 403 papers in Ecology cite at least one “personal communication”
Notice that the number of Ecology papers containing the phrase “personal communication” has dropped by well over 50% since the early ’80s, even though the number of papers published by Ecology has increased a lot since the early ’80s. So the percentage of Ecology papers citing “personal communication” has cratered, just within the professional careers of active senior ecological researchers.
I highly doubt this trend is specific to Ecology, I’m sure you’d get similar results if you looked at Am Nat or JAE or Oikos or whatever.
I can think of a few not-mutually-exclusive hypotheses to explain this. Ecology journals publish less natural historical work than they used to (though there’s been a bit of a rebound in recent years). I feel like natural historical papers are particularly likely to cite personal communications for natural historical information that is stored in the heads of natural historians rather than written down in a citable source. Maybe there’s also just a growing expectation on the part of reviewers, editors, and graduate advisors that authors will cite written sources whenever possible. Perhaps because written sources are thought to be easier for others to check for themselves. And maybe there’s a feedback loop–people stop citing “personal communication” because they don’t see anyone else doing it.
I don’t know that this is all that important in itself. I don’t think the decline in “personal communication citations” is either a problem to be solved or a victory to be celebrated. Times change, and one little symptom of changing times is that ecologists mostly don’t cite “personal communication” anymore.
Also this week: all the reasons Jeremy’s now looking forward to the rest of this year less than he was a week ago, forward vs. reverse causal inference, and more.Continue reading
Was corresponding with a colleague recently about what makes for good exploratory research in ecology and evolution. The jumping-off point for the conversation was Brian’s old post in praise of exploratory statistics. In which Brian argues convincingly that we should value exploratory studies as exploratory studies, rather than trying to dress them up as hypothesis-testing studies. But presumably, not all exploratory studies are created equal. Presumably, not all exploratory studies are even worth publishing (just as not all hypothesis-testing studies are worth publishing).* So what makes an exploratory study good?
I’m not sure! But here are a few tentative thoughts:Continue reading
I recently had the pleasure of hosting a remote seminar by Ambika Kamath in my department’s EEB seminar series. It was an unusual talk: Ambika read it word-for-word from a script. But unusual in a good way. It was a very good script, and Ambika read it very well, which made it a very good talk.
The usual advice to seminar speakers is “don’t read your talk”. I’ve given that advice myself to graduate students in years past. But having seen Ambika read her talk, I’ve changed my mind. I think it’s fine to read your talk, if that works for you.
It’s worth saying a bit more about what “works for you” means. It doesn’t just mean “I prefer to read my talks.” Because if you read your talk badly–in a monotone, for instance, or if you keep losing your place–then I don’t think reading your talk actually works. I’ve heard that the late Bill Hamilton used to read his talks–with his back to the audience and mumbling. That’s a bad way to give a talk. It’s bad even if other ways of giving talks would’ve worked even worse for him. If you need to give talks on your work, but no way of giving talks works for you, well, try your best to improve. Try new things until you find a way of giving talks that works for you.
I think this is a specific illustration of a broader point.Continue reading