When numbers fail: Do researchers agree on operationalisation of published research?

Current discussions on improving the reproducibility of science often revolve around statistical innovations. However, equally important for improving methodological rigour is a valid operationalisation of phenomena. Operationalisation is the process of translating theoretical constructs into measurable lab-quantities. Thus, the validity of operationalisation is central for the quality of empirical studies. But do differences in the validity of operationalisation affect the way scientists evaluate scientific literature? To investigate this, we adapted three published studies and manipulated the strength of operationalisation. We preregistered our planned study as a registered report. For each study, we designed two tasks in which the validity of the used operationalisation is manipulated and sent them to researchers via email. In the first task, researchers were presented with a summary of the Method and Result section from one of the studies and were asked to guess the hypothesis that was investigated via a multiple-choice questionnaire. In a second task, researchers were asked to rate the perceived quality of the study. Our results show that (1) researchers are better at inferring the underlying research question from empirical results if the operationalisation is more valid; but (2) the different validity is only to some extent reflected in a different judgment of the quality of the study. These results combined give partial evidence for the notion that researchers’ evaluations of research results are not affected by operationalisation validity.