Reliability, Replicability, and Validity: A Meta-Scientific Analysis

The role of measurement error in replicability of psychological findings has become of increasing interest, with some researchers suggesting it is critical to replicability, and others arguing that it is likely secondary to other effects on generalizability of findings. This work examined the relationship between reliability and effect size in published registered replication (i.e., many-labs) projects (313 samples from 44 studies) and a meta-analysis (16 samples from 8 studies). Among multiple-item registered replication designs, at lower reliabilities effect size was near zero regardless of reliability; at greater reliabilities (above approximately 0.80), effect size appeared to increase with reliability for some effects but not others. However, among the broader set of registered replication studies, including single-item designs, number of items was not associated with greater effect size, and in fact decreased with measure length. In the meta-analysis, greater reliability was associated with greater validity coefficients throughout the range of reliability. Results point to the importance of measurement precision in replicability of psychological findings, but also to the importance of precision per se and not proxies such as measure length.