For the details, read on! 🙂
One of the most basic things an ecologist might want to know about effect X is its sign. Whether the effect is the difference between treatment and control means, or a correlation between two variables, or whatever, we’d like to know: is it a positive effect, or a negative effect?
Now, ecological effect sizes can vary in sign, plus we live in world with sampling error, so there may not be any such thing as “the” sign of effect X. We could ask about the sign of the mean effect size, of course. But feedback I’ve been getting from y’all suggests that many of you don’t think that mean effect sizes are of much ecological interest. At least not if we’re thinking of those means as estimates of some notional “true” population mean. Rather, in a heterogeneous world in which the “true” mean likely varies a lot among studies, folks seem to be more interested in the overall shape and spread of the observed distribution of effect sizes. To which, fair enough. One of the most basic measures of the shape of any distribution is just: what fraction of the distribution is positive, and what fraction is negative. If effect X is usually or always positive (or usually or always negative), that might be all we need to know for some important purposes. For instance, many ecological theories only predict the sign of effect X, not its magnitude, so knowing that effect X is consistently positive (or consistently negative) might well be sufficient for testing theories of effect X. Conversely, if effect X commonly takes on values of either sign, then that seems important to know as well. For instance, even if effect X is significantly >0 on average, if it often takes on values <0, one might hesitate to recommend a management intervention premised on effect X being >0 in the particular system being managed.
Using my fairly comprehensive database of 476 ecological meta-analyses, I calculated the fraction of effect sizes from each meta-analysis that were positive.* Here’s the histogram of the results:
As you can see, the fraction of effect sizes that are positive varies widely. Which actually surprises me a little. Before I made these graphs, I actually thought that histogram would have a clear peak near 50% positive effect sizes. It’s more spread out than I thought it would be. But still, meta-analyses that only include positive effects, or only include negative effects, are rare.
Further, the meta-analyses that include only positive effects, or only negative effects, are mostly small meta-analyses that don’t include many effect sizes. Which strongly suggests that, if they had bigger sample sizes, they’d include effect sizes of both signs. Big meta-analyses in ecology–those with >200ish effect sizes–hardly ever have <20% or >80% positive effect sizes:
Here’s a zoomed-in version of that last graph, only showing meta-analyses with <500 effect sizes:
One might of course wonder if there are moderator variables that would explain variation in sign among effect sizes within any given meta-analysis. To which, sure, maybe, in some cases. But just based on my anecdotal impressions from skimming lots of meta-analyses, it doesn’t seem like sign consistency of effect sizes usually improves all that much if you restrict attention to effect sizes that share the same value(s) of one or more key moderator variables.
Bottom line: if you do an ecological study, there appears to be at least a 20% chance that whatever effect you’re measuring will have the opposite sign to the one you expected. Does that worry you? (“Ugh, even the signs of our effects don’t replicate consistently.”) Do you feel like it just goes with the territory? (“Welp, that’s ecology for you, weird stuff happens at least 20% of the time.”) Does it actually reassure you? (“This is actually more sign-predictability than I thought ecology had! Hooray!”) Looking forward to your comments.
*Note that there are a couple of meta-analyses in the database that used the absolute value of Hedge’s d as the measure of effect size, so all the effect sizes were positive. I was too lazy to drop those meta-analyses from this post. It’s only a couple of meta-analyses out of 476–too few to make any real difference.