Meta-regression to explain shrinkage and heterogeneity in large-scale replication projects

Recent large-scale replication projects (RPs) have estimated concerningly low reproducibility rates. Further, they all reported substantial degrees of shrinkage of effect size, where the replica tion effect size was found to be, on average, much smaller than the original effect size. Within these RPs, the included original-replication study-pairs can vary substantially with respect to aspects of study design, outcome measures, and descriptive features of both original and replication study population and study team. When broader claims about the reproducibility of an entire field are based on aggregations of such heterogeneous data, it becomes imperative to conduct a rigorous analysis of the amount and sources of shrinkage and heterogeneity within and between study-pairs included. Methodology from the meta-analysis literature provides an approach for quantifying the heterogeneity present in RPs, as additive or multiplicative parameter. Meta-regression methodology further allows for an investigation of the sources of shrinkage and heterogeneity. We propose the use of location-scale meta-regressions as a means to directly relate the identified characteristics with shrinkage (represented by the location) and the heterogeneity variance (represented by the scale). This could also provide valuable insights into drivers and factors associated with high or low reproducibility rates and therefore contextualise results of PRs. The proposed methodology is illustrated using data from the Replication Project Psychology and the Replication Project Experimental Economics. All analysis scripts and data are available online.