Machine Learning Reduced Workload for the Cochrane COVID-19 Study Register: Development and Evaluation of the Cochrane COVID-19 Study Classifier

Abstract Background: This study developed, calibrated, and evaluated a machine learning (ML) classifier designed to reduce study identification workload inmaintainingthe Cochrane COVID-19 Study Register (CCSR), a continuously updated register of COVID-19 research studies.Methods: A ML classifier for retrieving COVID-19 research studies (the “Cochrane COVID-19 Study Classifier”) was developedusing a data set of title-abstract records ‘included’ in, or‘excluded’ from,the CCSR up to18th October 2020, manually labelled byinformation and data curation specialists or the Cochrane Crowd. The classifier was then calibrated using a second data set of similar records ‘included’ in, or ‘excluded’ from,theCCSRbetween 19th October and 2ndDecember 2020, aiming for 99% recall. Finally, the calibrated classifier was evaluated using a third data set of similar records ‘included’ in, or ‘excluded’ from, the CCSR between4thand 19thJanuary 2021.Results: The Cochrane COVID-19 Study Classifier was trained using 59,513 records (20,878 of which were ‘included’ in the CCSR). A classification threshold was set using 16,123 calibration records (6,005 of which were ‘included’ in the CCSR) and the classifier hada precision of 0.52in this dataset atthe target threshold recall >0.99. The final, calibrated COVID-19 classifier correctly retrieved 2,285 (98.9%) of 2,310eligible study reports but missed 25 (1%), with aprecision of 0.638 and a netscreening workload reduction of 24.1% (1,113 records correctly excluded). Conclusions: The Cochrane COVID-19 Study Classifierreduces manual screening workload for identifying COVID-19 research studies, with a very low and acceptable risk of missing eligible studies. It is now deployed in the live study identification workflow for the Cochrane COVID-19 Study Register.