Poll results: what are the biggest problems with the conduct of ecological research?

Recently I polled y’all on which of the many purported problems with the conduct of ecological research are actually problems. For each of 24 purported problems in ecological research, respondents were asked if it was a serious problem, moderate problem, no/minor problem, or opposite of a problem.

Here are the results! They’re very interesting! You should totally read on!

Respondent attributes

• 115 respondents (thanks to everyone who responded!). As always with our polls, that’s not a random sample of any well-defined population. It’s not even a random sample of our readers. But it’s a large and diverse enough sample of ecologists to be worth talking about.
• 17% grad students, 25% postdocs, 43% faculty, 13% non-academic professional ecologists, 2% other. Grad students are rarer in this poll than in most of our other polls. Presumably because many grad students haven’t yet formed opinions on field-wide problems in ecological research.
• 37% fundamental researchers, 13% applied researchers, 50% both.
• The vast majority of respondents use multiple approaches in their own research. 77% collect observational field data, 64% compile/synthesize data collected by others, 52% do sophisticated statistics, 51% do computational modeling, 51% do field experiments, 39% do mathematical modeling/theory (aside: which surely means theorists are overrepresented in this poll compared to ecology as a whole, right?), 37% do lab studies/microcosms/mesocosms.

Results and commentary

Here’s the first result: Most ecologists think ecological research has at least a couple of serious problems. The median respondent identified 4 problems from my list of 24 as serious problems (mean 3.9, max 17). Only 19/115 respondents didn’t identify any of the problems on my list as serious problems.

To enable further analysis and graphical display, I converted the responses to numbers: 2 for “serious problem”, 1 for “moderate problem”, 0 for “no/minor problem”, -1 for “opposite of a problem”. Yes, that conversion is arbitrary, but it’s fine for purposes of a blog post. The results don’t change all that much if you analyze the data some other way.

Here’s a plot of the mean and variance of responses for each purported problem. Each point gives results for one purported problem, a bunch of the extreme ones are labeled:

This plot is super-interesting!

• Even the most serious problems aren’t all that serious in the eyes of the respondents. No purported problem on the list was considered to be a serious problem by more than 41% of respondents. There was only one purported problem for which “serious problem” was the modal response. And no problem had a mean score higher than 1.2, which is just a bit higher than “moderate problem” on my admittedly-arbitrary numerical scale.
• There’s a lot of disagreement about the seriousness of most purported problems in ecological research. For all but three purported problems on the list, every possible level of seriousness was chosen by at least one respondent, and for the other three every level except “opposite of a problem” was chosen by some respondents. Even for the purported problems on which there’s the greatest level of agreement as to their seriousness, there’s still a fair bit of disagreement in an absolute sense. For instance, respondents mostly agreed that underpowered studies are a moderate problem in ecological research: that was the modal answer, by far. But even there, 21% of respondents think underpowered studies are a serious problem, vs. 23% who think they’re no/minor problem. Some of this disagreement might reflect heterogeneity across subfields of ecology. Respondents were asked to base their responses on the areas of ecological research with which they were sufficiently familiar to have an opinion. And some of it reflects variation among ecologists using different approaches; see below for more on that.
• No matter what it is, at least 5% of ecologists think it’s a serious problem. Every purported problem on the list was considered a serious problem by at least 5% of respondents. Now I’m kicking myself for not including some non-problems on the list as controls. Maybe 5% of ecologists think cute puppies are a serious problem. 🙂
• According to respondents, the most serious problems in ecological research are HARKing/p-hacking/cherry-picking data/garden of forking paths and vague/unclear/non-operational terms and concepts. To which, yeah, I think I buy that. At least, I buy it as much as I’d buy any other picks, and more than I’d buy some other picks. I do wonder a bit if this poll oversampled ecologists who worry about HARKing/p-hacking/etc. Our readership overlaps substantially with readership of Andrew Gelman’s blog, and he’s always harping about those problems. I disagree with the crowd regarding the next two most-serious problems, though. I personally wouldn’t say that overvaluing novelty is the third most-serious problem on the list, though I’m also very unsure how serious a problem it is. And as I discussed in an old post, I don’t think undervalued/insufficient natural history is really one problem. I think it’s better thought of as a bunch of separate purported problems, most of which I personally don’t think are problems at all. YMMV, of course. That’s what makes this whole post topic interesting to talk about: it’s not just a matter of purely subjective personal opinion, but yet there’s scope for reasonable disagreement.
• According to respondents, the least-serious purported problem in ecological research is “overvaluing meta-analysis compared to collecting one’s own data”. This was the purported problem for which votes for “opposite of a problem” outnumbered votes for “serious problem” by the widest margin, and it wasn’t close. It’s also the purported problem that got by far the most votes for “opposite of a problem”: 29%! I voted for “no/minor problem” myself. I can’t see how anyone could think over-valuing meta-analysis is a serious problem. But I also find it hard to see how meta-analysis could be considered undervalued in a post-NCEAS world where data-sharing is mandatory and meta-analyses get widely cited. Hot take: the main effect of Lindenmayer & Likens’ high-profile but groundless complaint about “data parasites” was not to convince anyone–hardly anyone agrees with it–but rather was to make meta-analysts feel threatened and undervalued. Even though they’re not undervalued, except by a minority so small it’s not worth worrying about. The moral of this little just-so story is not that meta-analysts are over-sensitive. But just because somebody famous says something in a prominent venue doesn’t mean many others–or even a few others!–silently agree. I went through a similar experience myself back in the mid-90s, worrying that lots of ecologists didn’t like microcosm research just because a few prominent people didn’t like it. We’ve talked before about other ideas in ecology and academia that are controversial–or rather, that appear controversial–just because of a few people voicing minority opinions, whether in prominent venues or on social media. The point here is not that the minority opinion on topic X is necessarily wrong–or right! The point is just that you can’t tell what the majority opinion is by noting what a few vocal people think.
• According to respondents, the most controversial purported problem in ecological research is overvaluing theory compared to data. This was one of only three problems on the list for which the average opinion was intermediate (in between “no/minor problem” and “moderate problem”), the variance of opinion was high, and there was appreciable support for both “serious problem” (19%) and “opposite of a problem” (13%). There was also considerable disagreement about the purported problems “technology-driven, hammer-in-search-of-a-nail research” and “overvaluing generality relative to case studies”. But for both of those the mean opinion was fairly close to “no/minor problem” so I don’t consider those purported problems quite as controversial. (aside: personally, I don’t think generality and case studies are alternatives) There was also a high variance of opinion about over-valuing meta-analysis. But that was variance between the large majority who consider it no problem or the opposite of a problem, and a small minority who consider it a serious problem.
• Our readers don’t necessarily agree with us. So, back to that point about prominent people voicing minority opinions… 🙂 You might think that readers of this blog tend to agree with Meghan, Brian, and I. And at some level I’m sure that’s true. But it’s not as true as you might think, as this poll illustrates. Personally, I think mistakenly inferring causation from correlation is one of the most serious problems in ecology, and I’ve said so in multiple posts (example)–but respondents mostly disagree with me. As noted above, I don’t think natural history is undervalued in ecological research, but respondents mostly disagree with me. I’m (in)famous for banging on about zombie ideas (though I actually think they’re only a moderate problem in ecology as a whole). But respondents don’t think they’re a particularly serious problem. I was surprised how many respondents thought that inefficient theory was a moderate or serious problem, because I don’t think it is. Nor do all that many respondents agree with Brian and I that weak “pseudo-hypotheses” are a serious problem in ecological research. Respondents do mostly agree with Brian that statistical machismo and lack of good predictions are at least moderate problems in ecological research. And respondents do mostly agree with me that over-valuing lab/microcosm/mesocosm studies over field studies is not a problem. So remember: just because Brian, Meghan, or I write something doesn’t mean most–or even many!–ecologists agree.

Career stage doesn’t really predict ecologists’ opinions about purported problems in ecological research, with the caveat that sample sizes are small. More senior ecologists are less likely to think “irrelevance of ecological research to conservation/global change” is a serious problem (mean answers of 0.64, 0.4, and 0.02 for grad students, postdocs, and faculty, respectively). But given the number of respondents and the number of purported problems surveyed, you might well expect such a fairly modest association between mean opinion and career stage to occur somewhere in the dataset just by chance.

There were only a few purported problems on which the views of non-academic professional ecologists might be out of line with those of academic ecologists (again, remember the caveats re: sample size and multiple comparisons here). One was bandwagons. Academic ecologists (grad students+postdocs+faculty) think that bandwagons are a moderately serious problem on average (mean opinion 0.91), whereas non-academic professional ecologists don’t think they’re much of a problem (mean opinion 0.36). At the risk of HARKing :-), I bet that’s a real difference, not a blip. I bet the research foci of government ecologists, NGO ecologists, environmental consultants, etc. are less likely to reflect bandwagon-jumping, explaining why they tend to see bandwagons as less of a problem. Non-academic ecologists also seem to think HARKing/p-hacking/etc. are less serious problems on average than academic ecologists do (mean opinions of 0.8 vs. 1.3, respectively). But that might just be a blip.

There were several purported problems on which views seem to vary substantially between fundamental vs. applied researchers (again, sample size and multiple comparisons caveats apply):

• The clearest-cut case, which confirms my priors and therefore is obviously real :-), concerns whether “overvaluing theory over data” is a problem. On average, applied ecologists think it’s a moderate problem (mean 0.92), fundamental ecologists think it isn’t much of a problem (mean 0.28), and ecologists who do both fundamental and applied work are in the middle (mean 0.45). This result is so expected it’s boring, but it’s reassuring to see because it suggests the poll is picking up some real effects.
• Applied ecologists collectively see research that’s irrelevant to conservation/global change as close to a moderately serious problem (mean 0.79). Other ecologists do not (mean 0.21). Again, no surprise there.
• Applied ecologists collectively see overvaluing generality over case studies to be something approaching a moderate problem (mean 0.63). Fundamental ecologists think it’s a non-problem (mean -0.09), and ecologists who do both are intermediate (mean 0.36). Again, no surprise there.
• Applied ecologists collectively see an overemphasis on novelty as a more serious problem in ecological research (mean 1.46) than do fundamental researchers (mean 1.06) or those who do both (mean 1.13). Yawn.
• But here’s something I didn’t expect: applied ecologists are less concerned about lack of good predictions (mean 0.5) than are fundamental ecologists (mean 1) or ecologists who do both (mean 0.79). Does that surprise you? It surprised me, given that one of the reasons everybody always gives for why ecology needs more/better predictions is “to inform management decisions“.

The research approaches ecologists use also predict their opinions about purported problems in ecological research, in depressingly predictable ways. On average, ecologists tend to think their own approaches are fine and it’s other people’s approaches that are the problem. 😦 For instance, those who don’t do mathematical modeling/theory tend to think that theory is overvalued relative to data, that generality is overvalued relative to case studies, that natural history is insufficiently valued, and that too much ecological research is irrelevant to conservation/global change. Those who do mathematical modeling/theory don’t tend to think those things. Those who don’t use advanced statistical methods tend to think statistical machismo is a fairly serious problem. Those who use advanced statistical methods think it’s no problem or a minor problem. Those who compile/synthesize data collected by others tend not to see overvaluing of meta-analysis and overvaluing of generality as problems. Those who don’t compile/synthesize data collected by others are more likely to do so, though the differences aren’t large. On the other hand, I was surprised that people who do lab/microcosm/mesocosm work, and those who don’t, agree there’s no problem with overvaluing lab/microcosm/mesocosm work in ecological research. And opinion on whether undervaluing of small scale field experiments is a problem isn’t divided by whether or not the respondent does small scale field experiments.

Concluding thoughts

I’m still mulling over what, if anything, these poll results imply for the whole genre of “critiques of ecology”. Because make no mistake, it is a genre. Ecology is somewhat infamous among both ecologists and non-ecologists for being a messy field wracked by disagreements about basic matters, extending even to the very definition of the field itself. But one thing that strikes me about this poll is that there’s no purported problem with ecological research that’s thought to be a serious problem by a majority of respondents, and only a few that are even thought to be moderate problems on average. Rather, almost everybody thinks there are some serious problems with ecological research–but nobody can agree on what those serious problems are. Is that a bad sign for the field? (“We can’t even agree on what the serious problems with ecological research are, never mind how to solve them!”) Or a good sign? (“Nobody can agree on what the serious problems in ecological research are, because it doesn’t have any serious problems. Every ecologist just has their own random bee in their bonnet.”)

I continue to be a bit depressed that ecologists’ opinions on the serious problems with ecological research are correlated so predictably with their own research approaches. Ok, not everybody thinks their own research approaches are undervalued and everyone else’s are overvalued. But enough people do that it bums me out a bit. It’s a recipe for a self-sustaining feedback loop of mutual incomprehension and rivalry between ecologists who use different approaches. I suspect it’s fueled in part by competition for scarce grant funding and space in leading journals. Everybody needs to go read this old post of Brian’s and this old post of Meghan’s, and quit thinking that everybody who doesn’t do the sort of ecology you do is Doing It Wrong and hogging all the money and publication slots.

What do you think? Looking forward to your comments, as always.