Measuring biological generality in meta-analysis: a pluralistic approach to heterogeneity quantification and stratification

Uncovering general rules enhances the predictive capabilities in ecology and evolution. Meta-analytic approaches play a critical role in this endeavour, examining the extent to which phenomena can be replicated, generalized, and transferred. However, ecologists and evolutionary biologists have largely overlooked the role of meta-analytic heterogeneity in informing generality. To reform this situation, we introduce a pluralistic approach aimed at quantifying and stratifying various heterogeneity metrics, such as I^2, CV, M, and predictive distribution. These metrics offer complementary information, revealing the source, magnitude, and visual representation of heterogeneity. Our analysis of 512 meta-analyses demonstrates that heterogeneity is, on average, ten times larger than statistical noise, contributing to 91% of the observed variance (median I2 = 91%). This amount of heterogeneity is nearly twice the size of the meta-analytic mean effect (median CV = 1.8, M = 0.6), indicating substantial total heterogeneity in ecology and evolution. Surprisingly, in half of the cases, focal effects could generalize across studies even with high total heterogeneity by controlling for within-study variation. Our synthesis also visualises empirical distributions of various heterogeneity metrics, potentially serving as new benchmarks for informed interpretation. Our proposed pluralistic approach will accelerate the future quest for general rules via meta-analyses.