Implementing statcheck during peer review is related to a steep decline in statistical reporting inconsistencies

We investigated whether statistical reporting inconsistencies could be avoided if journals implement the tool statcheck in the peer review process. In a preregistered pretest-posttest quasi-experiment covering over 7000 articles and over 147,000 extracted statistics, we compared the prevalence of reported p-values that were inconsistent with their degrees of freedom and test statistic in two journals that implemented statcheck in their peer review process (Psychological Science and Journal of Experimental and Social Psychology) and two matched control journals (Journal of Experimental Psychology: General and Journal of Personality and Social Psychology, respectively), before and after statcheck was implemented. Preregistered multilevel logistic regression analyses showed that the decrease in both inconsistencies and decision inconsistencies around p = .05 is considerably steeper in statcheck journals than in control journals, offering preliminary support for the notion that statcheck can be a useful tool for journals to avoid statistical reporting inconsistencies in published articles. We discuss limitations and implications of these findings.