Developing Deep Learning Models for the Triage of Pediatric Elbow Radiographs: Strategies for Model Optimization With Small Datasets

Abstract Background and Objective: To compare the performance of an AI model developed based on strategies designed to accommodate small sized development sets to pediatric ER physicians on a binomial classification task of pediatric elbow radiographs.Materials and Methods: 1,314 pediatric elbow lateral radiographs (mean age: 8.2 years) were retrospectively retrieved, binomially classified based on annotation as normal or abnormal (with pathology), and randomly partitioned into a development set (993 images), tuning set (109 images), second tuning set (100 images) and test set (112 images). An AI model was trained on the development set utilizing the EfficientNet B1 compound scaling network architecture and augmentations. Its performance on the test set was compared to a group of five physicians (inter-rater agreement: fair). Statistical analysis: AUC of AI model - DeLong method. Performance of AI model and physician groups - McNemar test.Results: Accuracy of the model on the test set - 0.804 (95% CI, 0.718 - 0.873), Area under Receiver Operating Characteristic (AUROC) - 0.872 (95% CI, 0.831 - 0.947). AI model performance compared to the physician group on the test set - sensitivity 0.790 (95% CI 0.684 to 0.895) vs 0.649 (95% CI 0.525 to 0.773), p value 0.088; specificity 0.818 (95% CI 0.716 to 0.920) vs 0.873 (95% CI 0.785 to 0.961), p value 0.439.Conclusions: An AI model for elbow radiograph triage designed with strategies to optimize performance for a small sized development set performed comparably to physicians.