Calibration experiments: an experimentalist alternative to multi-method approaches for measurement validation

Most theories in psychological, educational, and social sciences deal with latent, not directly observable, variables such as memory, cognitive skills, or job satisfaction. Measurement validation ensures that the actually taken measurements relate to the latent variable one is interested in, and is becoming ever more pervasive as a criterion for pre-registering specific sets of observables or data processing steps. According to the existing research canon, quantitative validation is ideally performed within the context of a multi-trait multi-method matrix (MTMM). While statistically fairly well developed, this approach has several conceptual and practical shortcomings that limit its domain of application: (1) the requirement for discriminant latent variables that can be measured with the same methods, (2) the requirement for conceptually different methods to disambiguate trait from methods factors, and most seriously (3) the difficulty in identifying a more valid over a less valid method. Here, we compare this to experiment-based calibration, a complementary approach for latent variables that can be externally manipulated. Inspired by the practice of experimental calibration in modern metrology (measurement science), this approach relies on an experimental treatment that introduces an external criterion, which is procedurally unrelated to the measurement methods. We show how calibration can compare even closely related measurement methods, dispenses with a need for discriminant latent variables, and allows optimizing the psychometric procedures themselves. Calibration can thus complement classical psychometrics for latent variables that are amenable to external manipulation, and could become part of an integrative validity argument in the psychological, educational, and social sciences.