Some Data Indicating that Editors and Reviewers Do Not Check Preregistrations during the Review Process

If a study is preregistered, but nobody checks it during review, does it mean anything? The purpose of this preregistered study was to better understand how editors and reviewers engage with preregistration plans during the review process. The article set was retrieved from the PLOS of family of journals, which a) are cross-disciplinary, b) are open access, c) have a relatively high frequency of preregistered studies, and d) most importantly, in 2019 began publishing manuscript reviews alongside published articles. A total of 693 articles were identified via the initial search, which was reduced to 201 that met the review criteria. For each article, the review history was coded for whether editors or reviewers a) mentioned the preregistration, b) accessed the preregistration, and c) made specific mention of the relation between the preregistration and reporting in the paper. Analyses were conducted at the article level (i.e., did any reviewer or editor mention/access/compare; n = 201) and the editor/reviewer level (i.e., overall, how many editors/reviewers mention/access/compare; n = 689). In 43% of articles at least one editor/reviewer mentioned the preregistration, but in only 14% did someone access the preregistration and in 10% did someone make a clear statement about the relation between the preregistration and the manuscript. At the editor/reviewer level things look much worse, with only 18% mentioning preregistration, 5% reporting accessing the preregistration, and 3% discussing the relation between the preregistration and the manuscript. These data suggest very low levels of engagement with the preregistration during the review process, undermining its effectiveness.