Today’s weird question: what’s the typical effect size of an ecological study? Like, of anything? Using any experimental or observational method?
Keep reading for the answer!
As part of a side project, I am compiling effect size data from many different ecological meta-analyses. I don’t care at all about the topic or the methods. All I care about is that it was a meta-analysis about something ecological, and that the authors published a table of all the effect sizes included in their meta-analysis.*
So far I have 49 meta-analyses, on topics ranging from the effect of plant-soil feedback on plant diversity (Crawford et al. 2019) to the effects of sedimentation on various features of marine organisms (Magris & Ban 2019) to phenotypic selection on flowering phenology (Munguia-Rosas et al. 2011), and more. These 49 meta-analyses were published in a bunch of different journals, from Nature to Ecology Letters to Ecosphere and more. Most of them used one of two measures of effect size: Hedge’s d, or the log-transformed response ratio. Hedge’s d (which is actually the same thing as Cohen’s d, right?) is the difference between two means, such as a treatment mean and a control mean, divided by the pooled standard deviation. The log(response ratio) is the log of the ratio of two means, usually a treatment mean divided by a control mean. The sign of the effect obviously depends on the (sometimes arbitrary) decision about which mean to designate as the “treatment” mean, and on the effect of the treatment, if any. But for both effect size measures, values near 0 indicate small effects in terms of absolute magnitude, and values far from zero indicate large effects. A typical ecological meta-analysis will include dozens to hundreds of measures of effect size, because it will include many papers, many of which will report multiple effect sizes (e.g., because the same study was repeated on multiple study species, or in multiple habitats, or etc.).
Obviously there are complicated issues of hierarchical data structure and non-independence here. But we don’t need to worry about them to get a first-pass answer to our question.
Here are histograms of all values of Hedge’s d, and all log-transformed response ratios, from all meta-analyses I’ve looked at so far. Note that the x-axis on each panel is scaled so as to exclude a few extreme outliers:
The first thing to notice about both those histograms is the peak at zero. The modal effect size of ecological studies is no effect. However, the second thing to notice in both histograms is the wide x-axis scale. The modal value is not typical; both of those histograms include a lot of big effects! Cohen (1988) famously suggested a rough rule of thumb that an absolute value of d=0.2 was a “small” effect, d=0.5 was a “medium” effect, and d=0.8 was a “large” effect. That rule of thumb is from psychology, and there are various issues with its interpretation (see this old post for discussion of how to define “small” effects). But FWIW, 80% of the d values in this dataset are >0.2 in absolute magnitude, 60% are >0.5, and the median is 0.67. Turning to the log response ratios, 54% of them have absolute values >0.182, indicating that one mean is at least 20% larger than the other. 21% of the log response ratios are >0.69 in absolute value, indicating that one mean is at least twice as large as the other.
In sum, effect size distributions in ecology are leptokurtic. They have a sharp peak at zero, but very heavy tails.** Which means that the typical effect size of published ecological studies is moderate to large in absolute magnitude, for one standard definition of “moderate to large”.
Looking forward to discussion of what to make of these results. Is it heartening to know that ecological studies often find moderate to large effects? Or is that fact just meaningless, because it’s stripped of so much ecological context? In particular, one could argue that effect sizes in experiments reflect the size of the manipulation. We probably shouldn’t be surprised that experimenters typically “kick the study system hard enough to hear it yell.”
*If you think this sounds like question-free data-dredging, well, you’re right, though I prefer the term “exploratory”. 🙂 Like I said, it’s only a side project.
**Note that effect sizes reported in meta-analyses may slightly overstate the kurtosis for all ecological studies, because meta-analyses of experiments with several treatments (e.g., several different temperatures) often only include the effect size for the most extreme pair of treatments (e.g., lowest vs. highest temperature).