Also this week: the latest on Plan S, Gresham’s Law vs. simulated pandas, fractals vs. David Foster Wallace, and more.
Marginal Revolution points us to the latest on researcher degrees of freedom (unreviewed preprint): give 70 different teams of neuroscientists the same fMRI data and tell them to test the same scientific hypotheses, and they give you back widely-varying answers because they make different choices at all stages of data processing and analysis. (though not so varying that there’s no commonality among the answers.) A previous study of whether soccer referees are more likely to give red cards to dark-skinned-toned players also found that different analysts reached widely-varying answers, though not so varying as to lack any commonalities. Both studies also did some interesting supplementary analyses to try to explain the among-researcher variation.
I’m increasingly curious what you’d find if you did a similar exercise in ecology. We know from survey data that various questionable research practices, some of which fall under the heading of “researcher degrees of freedom”, are common in ecology and evolution (Fraser et al. 2018). But we don’t know exactly how big their consequences typically are for statistical analyses. One way to find out is to give a bunch of ecologists the same dataset, ask them to address some clearly-stated scientific claim, and see how much among-ecologist variation there is in the answers. Note as well that “researcher degrees of freedom” aren’t always bad. After all, we don’t want everyone doing mindless cookbook statistics. The point of quantifying the effects of “researcher degrees of freedom” in ecology is to learn more about how ecologists exercise their professional statistical judgment, not to criticize any and all exercises of statistical judgment. Judgment calls can never be eliminated from statistics.
I’m curious enough about this that I’m seriously considering doing it! But if I’m going to do it, I want to do it well, and so I think it’d be best to do it collaboratively. So, who’s with me? If you’re interested in participating in this, drop me a line (firstname.lastname@example.org), or leave a note in the comments.
If we’re going to do this, one of the first things we’d have to do would be to identify/compile a good dataset, and a good scientific question to be addressed with that dataset. The question would need to be an interesting/important scientific question, I think, because that makes the exercise worthwhile scientifically, as well as as a study of ecologists’ statistical decision-making. And the dataset would need to be a reasonably large, rich dataset, I think. In part because you want a dataset that’s capable of addressing the scientific question of interest reasonably well. And in part because, if you just give people a very simple dataset with, say, an X and Y variable and ask for the best estimate of the slope of Y on X, you’re not leaving much scope for researcher degrees of freedom.
It couldn’t be a dataset that’s already been published and analyzed, I don’t think. Not even one that’s already been analyzed in different ways by opposing camps of researchers. Researchers’ analytical choices would be influenced by having read others’ analyses.
It would be interesting to come up with several datasets and associated questions. Some of the questions and datasets should concern issues on which there’s considerable pre-existing controversy in the ecological literature, and others should concern issues on which there’s little or no controversy. Then you could ask: do “researcher degrees of freedom” tend to loom especially large in analyses of controversial scientific questions? Asking that question is one way this study could go beyond just showing once again that, yeah, “researcher degrees of freedom” is a thing. But if you were going to do this, you’d have to make sure you had a way to measure the variance among researchers’ answers that was comparable across questions and datasets.
I also think it would be interesting to first poll researchers on what answers they expect to obtain. Then look for a correlation between the answers they expected to obtain and the answers they actually obtained.
I could imagine that this exercise might be a very interesting add-on to the work of one or more ongoing or planned working groups.
And if you wanted to go really meta, you could let different researchers independently analyze the results of this exercise!*
Whether or not you’re interested in participating in this, feel free to share any thoughts you have about this project in the comments.
UPDATE: In the comments, Shan Kothari points out that I’m not the first ecologist to have this idea (I’m not surprised; thanks to Shan for passing on this info). So the first thing I’ll do is get in touch with the folks who already had this idea.
UPDATE #2: Updating again to confirm that Hannah Fraser, Tim Parker, and their colleagues are still moving forward with this project. They had the idea first and they’re already well down the road, so it wouldn’t make much sense for me to organize a parallel effort, I don’t think. Glad to hear somebody’s already doing this, looking forward to seeing what comes out of it.
UPDATE #3: Scroll down to the comment from Hannah Fraser for instructions on how to sign up as an analyst for this project.
It’s increasingly easy for university students to share class assignments online, for instance via StuDocu. That’s nothing new–students have long shared assignments with one another. But it’s much easier to do now that assignments, and any answer keys to those assignments, often are distributed to students in electronic form. As an instructor, what do you do about this, if anything?
I can imagine various possibilities:
Also this week: big new dataset on gender bias in scientific publishing and peer review, Ohio vs. evolution, ecology bloggers wanted, comedy wildlife photography, and more.
Well, there’s at least one field that works this way: economics.
I’ve written in the past about my outsider’s fascination with how the economics faculty job market works. I find it interesting to learn about how different fields operate; it helps me look at my own field’s practices with fresh eyes. So I already knew that the economics faculty job market operates very differently than the ecology faculty job market in various ways (click that last link for background). But I only just learned about one more: in economics, it’s common practice for economics faculty and departments to talk up their favorite faculty job seekers on social media.
Also this week: math vs. Wall Street, beexploitation, and more.
Ecologists study lots of different things, and those things vary a lot in all sorts of ways. No two sites/times/organisms/populations/communities/ecosystems/species/landscapes/food webs/whatevers are exactly alike. Should ecologists seek generalities about the diverse, heterogeneous stuff they study? If so, what kinds of generalities should they seek and how should they seek them? Those are questions on which prominent ecologists have long disagreed. But the views of a few prominent ecologists might well be unrepresentative of the views of ecologists collectively, for the same reasons that the loudest voices in any discussion might not be representative of the views of everyone in the room. So to get a bit of data, I polled y’all on these questions. Here are the results!
As with all our polls, this isn’t a random or representative sample of all ecologists, or even of all readers of this blog. But it’s a big and diverse enough sample of ecologists to be worth talking about. I found the results very interesting, and a couple of them surprised me. They were so interesting I even went to the trouble of doing a PCA and making a pretty graph, as if I was Meghan or something. 🙂 So you should totally read on.
Also this week: it’s the end of California as we know it, PCA vs. Elvis, #MathGals, physics doom loop, remembering Napoleon Chagnon, and more.
One very natural response to my poll from earlier this week on ecologists’ views about generality is, wait, don’t we all seek generality in our own work? After all, every theoretical and empirical paper every ecologist writes has a passage discussing whether/how the results generalize to other circumstances–other sites or times, other species, other models making different assumptions, etc. So isn’t a poll about ecologists’ attitudes towards generality just going to reveal a boringly-high level of agreement? Like asking people whether ice cream is good?
I actually do agree–and the poll results so far confirm–that almost all ecologists care about generality in some sense or other, and pursue generality in their own work using some approach or other. But I also think–and the poll results so far confirm–that there’s substantial variation among ecologists in what specific kinds of generality they seek in their own work, and value most in the work of others. I did the poll to learn more about that variation.
I’m not claiming any special prescience here. I doubt it will surprise anyone familiar with the history of ecology to learn that ecologists disagree a fair bit about exactly what forms of “generality” to seek, and how to seek them. For instance, here are a bunch of quotes about generality from prominent ecologists past and present. Try to find any generalities about them–besides the fact that each disagrees with most of the others!
“To do science is to search for repeated patterns, not simply to accumulate facts.” – Robert MacArthur (1972)
“Unlike population genetics, ecology has no known underlying regularities in its basic processes…” – Leigh Van Valen and Frank Pitelka (1974)
“The very most important thing to me, being a scientist, is to seek out unification.” – John Harte (2014)
“I think of ecology as a library of well-developed case studies.” – Tony Ives (2014)
“General ecological patterns emerge most clearly from this glorious diversity when systems are not too complicated…and at very large scales, when a kind of statistical order emerges from the scrum. The middle ground is a mess.” – John Lawton (1999)
“Community ecology is often perceived as a mess, given the seemingly vast number of processes that can underlie the many patterns of interest, and the apparent uniqueness of each study system. However, at the most general level, patterns in the composition and diversity of species–the subject matter of community ecology–are influenced by only four classes of process: selection, drift, speciation, and dispersal.” – Mark Vellend (2010)
“[T]here are several very general law-like propositions that provide the theoretical basis for most population dynamics models…Some of these foundational principles, like the law of exponential growth, are logically very similar to certain laws of physics” – Peter Turchin (2003)
“[W]e don’t need no stinkin’ laws” – Bob O’Hara (2005)
“These [previous] studies have provided more and better data on a wide range of ecological phenomena. There has not, however, been comparable conceptual progress in organizing and synthesizing existing information, producing mathematical models that are both realistic and general, and developing a body of ecological theory that can account for both the infinite variety and the universal features of organism-environment relationships.” – Jim Brown (1997)
“Our future advances will not be concerned with universal laws, but instead with universal approaches to tackling particular problems.” – Peter Kareiva (1997)
“The multiplicity of models is imposed by the contradictory demands of a complex, heterogeneous nature and a mind that can only cope with a few variables at a time; by the contradictory desiderata of generality, realism, and precision; by the need to understand and also to control; even by the opposing esthetic standards which emphasize the stark simplicity and power of a general theorem as against the richness and diversity of living nature. These conflicts are irreconcilable. Therefore, the alternative approaches even of contending schools are part of a larger mixed strategy.” – Richard Levins (1966)
It is of course possible that these haphazardly-chosen quotes are unrepresentative of the range of views among ecologists more broadly. Maybe it’s only ecologists who write opinion pieces about “generality” who disagree about “generality”! That’s why I took the poll, to find out. 🙂 Look for the results soon.