An ongoing theme to some of my posts has been the notion of statistical machismo. As noted recently, statistical machismo is not really about using (or not using) complex statistics. It is about using more complex statistics for bad reasons (e.g. to impress people) or forcing other people to use more complex reasons again for bad reasons or out of the ill-conceived notion that there is always one correct, best way to do statistics. The discussions on the last posts raised interesting questions about whether statistical machismo is really a problem or if it occurs just as often in the other direction (forcing people to use simple statistics). So of course that called for a poll . I am going to report on the results here.
You can examine the results here. Long story short, statistical machismo is not made up. People feel pushed to use more complicated statistics inappropriately much more often than they feel pushed to use simpler statistics. And people believe nearly every flavor of reputation accrues to those who use more complicated statistics even though people think more complex statistics don’t really change the science much.
There were 405 respondents to the poll (now closed). I watched it closely through the day to see if there were big swings (a sign of somebody trying to get a group to skew the results) and didn’t see it. So I think the results are a good sample of readers of Dynamic Ecology who like to answer polls. How well that represents your world of interest may vary. We had about 40% saying basic research, 20% saying applied, and 40% saying both. We had about 60% male, 35% female, 5% did not say or other. And there were about 1/4 for each career stage (graduate, postdoc, early or established permanent jobs) and even two undergraduates. We did not get a wide diversity of fields: 85% said ecology, 10% said evolution, and 5% spanned other fields.
Our poll takers skewed towards what I have to assume is stats savvy (only 4% said they had less stats knowledge than a typical field ecologist and 38% said they were typical, leaving 58% saying they knew more stats than a typical field ecologist). Only 10% said they skipped the methods sections of stats-heavy paper and over half said they fully understood such sections. Assuming accurate self-reporting, this is definitely a stats-savvy group in comparison to e.g. the typical graduate committees I sit on. With one exception related to career stage noted below, none of the demographic factors seemed to really influence perceptions of statistical machismo that I could find. So I don’t bother to present crosstabs on these, but I did look for several prior hypotheses I had about them. And I think results have to be interpreted in the context of this being a more-stats-friendly group than the average ecologist.
65% of respondents reported being forced to add more complex statistics for bad reasons, with over 30% saying it happens sometimes or frequently. While the reverse (forced to use simpler statistics) does occur, it is definitely the less common scenario. Only 42% had ever seen it happen and only 17% said it happens sometimes or frequently. Similarly, about 50% reported changing statistics to be more complex before the paper was even sent out for review out of fear of what the reviewers would say, while only 24% reported simplifying out of fear of reviewers. So very broad brush – people felt inappropriately pushed to complexify their stats about twice as often as to simplify. The only meaningful cross-tab I identified is that established researchers were more likely to report that they are forced to complexify their statistics inappropriately than postdocs & less established career researchers who in turn reported it more than graduate students.
As far as reputation, people clearly thought having complicated statistics improved the odds of a paper getting into a high profile journal, and especially using complex statistics enhanced the professional reputation of people using them. But people were much more neutral with only slight leans towards complex statistics improving the odds of a paper being impactful or improving the scientific quality of the paper. If people’s perceptions are accurate, it does appear there is room for gamesmanship in statistical machismo – complex stats improve paper positioning and individual reputation much more than they actually improve the impact of a paper or the quality of its science. In a more direct question, 27% said more complex statistics never or rarely changes the scientific conclusions, 13% said it usually or always changes the scientific conclusion and 60% said it sometimes changes the conclusion.
And as for which areas reviewers felt were pushed inappropriately:
The data largely speaks for itself (the x-axis is proportion of people responding to this question – N=317 – who checked this box so 41% said AIC had been inappropriately pushed on them), but a few observations of mine on this:
- I’m no fan of how AIC is used in ecology today. Nor am I fan of overly complex mixed models. But I perceive these as techniques that are rapidly increasing in usage in ecology and among the favorite techniques of ecologists. So I was surprised to see them at the top of techniques that were pushed too much.
- I was also surprised to see GLM and complex multivariate methods (e.g. RDA & CCA) so high up – it seems to me the use cases where they are or are not needed are fairly clear and distinct. But apparently lots of people still disagree.
- Over 30% of people think Bayesian is pushed inappropriately while 15% think frequentist is pushed inappropriately.
- The most fascinating to me was the relative ranks of phylogenetic regression, spatial regression (both pretty high on the culprit list) vs. times series tools (very low). Time series tools is a broad category, but it includes using GLS when you regress Y vs time (e.g. abundance vs. year) to see if there is a trend. These three issues all have the identical problem (overestimated degrees of freedom due to non-independence of data) and identical solution (using GLS with a metric for distance between points). And in fact using GLS on timeseries is way easier than using GLS on phylogenetic data (because you know how far apart different dates are but you have to generate a phylogeny to know how far apart species in a phylogeny are). Yet nobody worries about having to use GLS on timeseries data (I am not sure I’ve ever heard anybody being forced to do it). Yet it is apparently – for ecologists – desperately important to do phylogenetic regression, and apparently people are kind of fed up with having this pushed all the time (and to a slightly lesser degree spatial regression).
- Detection probabilities was medium high up on the list, but the majority of respondents do basic research and may not run into them. The proportion of people who perceive them as pushed inappropriately was highest among people who do both basic and applied, and then next highest in applied and lowest in basic.
So, I’m not sure a poll will settle all debate (in fact I’m sure it won’t). But there does seem to be some solid empirical evidence for the notion of statistical machismo pushing people to use more complex statistics than necessary. Two last thoughts. The respondents of this poll seem more reasonable and measured (e.g. choosing sometimes as the most common answer for whether more complex methods change the scientific answer) than reviewers who seem to push much more firmly (overconfidently). Or maybe people just remember the few extreme reviewers? And there seems to be a pretty wide spread belief among this group of fairly statistically sophisticated poll respondents that reviewers regularly and erroneously push statistical techniques (some complex, some just trendy). We would all do well to be a little more humble and remember that almost never is there only one right approach to statistical analysis. In particular, it seems that trendy might be almost as important a reason behind statistical machismo as complex.
What do you think? Did the poll surprise you? Am I overinterpreting it as evidence of statistical machismo? What would it take to get people to be a little more humble and less overconfident that there is only one right way to do statistics?