My fairly comprehensive compilation of all 114,000+ effect sizes from 476 ecological meta-analyses also includes the sampling variances of those effect sizes. I was curious if those sampling variances have been dropping over time. For instance, maybe studies these days tend to have larger sample sizes, and thus smaller sampling variances?

No, not as far as I can tell from my preliminary analyses. As a first pass, for each meta-analysis I just calculated the correlation between the effect size sampling variance, and effect size publication year. That gives me 476 correlation coefficients. Negative correlations indicate that sampling variances of effect sizes dropped over time, positive correlations indicate the opposite. Here’s a histogram of those 476 correlation coefficients:

As you can see, the mean is very close to zero, and the distribution is roughly symmetrical around zero. So there’s no systemic tendency in ecology for sampling variances to drop over time. Rather, this distribution looks exactly like you’d expect if effect sizes tend to be published in random order with respect to their sampling variances. Sometimes, just by chance, it will just so happen that more recently published effect sizes will have lower sampling variances. And sometimes, it will just so happen that more recently published effect sizes have higher sampling variances.

So if you think that more recent ecological studies usually are more “rigorous” than older ones, well, are you sure about that? I mean, maybe they are, depending on what exactly you mean by “rigorous.” But if what you mean by “rigorous” is “low sampling variance”, I don’t see any sign that ecological studies of topic X tend to get more rigorous over time. At least not in any major way. If there’s a signal there, it seems like it must be pretty subtle.

1 thought on “Does the sampling variance of ecological studies generally improve over time?”

Nice post Jeremy. I agree that it completely depends on what you mean by rigorous. For lots of field studies in the area that I work in (effects of conservation management) I would argue that rigour probably has more to do with the study design. i.e. BACI and Randomized control trials are better than space-for-time and before-after studies. Alec Christie has done some nice work comparing different study methods in this field https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111/1365-2664.13499. Would be interested to see how use of different methods has changed over time, my gut feeling is that it probably hasn’t changed much either…

Nice post Jeremy. I agree that it completely depends on what you mean by rigorous. For lots of field studies in the area that I work in (effects of conservation management) I would argue that rigour probably has more to do with the study design. i.e. BACI and Randomized control trials are better than space-for-time and before-after studies. Alec Christie has done some nice work comparing different study methods in this field https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111/1365-2664.13499. Would be interested to see how use of different methods has changed over time, my gut feeling is that it probably hasn’t changed much either…