Does the sampling variance of ecological studies generally improve over time?

Source: https://i.pinimg.com/originals/ee/b6/5d/eeb65d2597e81282c15fa95b3375db96.png

For the details, read on!

My fairly comprehensive compilation of all 114,000+ effect sizes from 476 ecological meta-analyses also includes the sampling variances of those effect sizes. I was curious if those sampling variances have been dropping over time. For instance, maybe studies these days tend to have larger sample sizes, and thus smaller sampling variances?

No, not as far as I can tell from my preliminary analyses. As a first pass, for each meta-analysis I just calculated the correlation between the effect size sampling variance, and effect size publication year. That gives me 476 correlation coefficients. Negative correlations indicate that sampling variances of effect sizes dropped over time, positive correlations indicate the opposite. Here’s a histogram of those 476 correlation coefficients:

As you can see, the mean is very close to zero, and the distribution is roughly symmetrical around zero. So there’s no systemic tendency in ecology for sampling variances to drop over time. Rather, this distribution looks exactly like you’d expect if effect sizes tend to be published in random order with respect to their sampling variances. Sometimes, just by chance, it will just so happen that more recently published effect sizes will have lower sampling variances. And sometimes, it will just so happen that more recently published effect sizes have higher sampling variances.

So if you think that more recent ecological studies usually are more “rigorous” than older ones, well, are you sure about that? I mean, maybe they are, depending on what exactly you mean by “rigorous.” But if what you mean by “rigorous” is “low sampling variance”, I don’t see any sign that ecological studies of topic X tend to get more rigorous over time. At least not in any major way. If there’s a signal there, it seems like it must be pretty subtle.