Physician Consensus Stratification Reveals Performance Limits of Deep Learning for Smartphone-Based Pharyngitis Diagnosis

Background: Overprescription of antibiotics for pharyngitis—driven by diagnostic uncertainty and physician disagreement on ambiguous presentations—contributes significantly to antimicrobial resistance. Existing AI diagnostic models report aggregate accuracy metrics that obscure performance variability across cases with different levels of diagnostic certainty, limiting their clinical utility. Objective: To develop and validate a consensus-based evaluation framework for deep learning models in pharyngitis diagnosis, stratifying performance by physician agreement levels (High vs. Low Consensus), and to compare image-only, symptom-only, and multimodal fusion strategies for smartphone-based diagnosis. Methods: We curated a dataset of 742 multimodal cases (symptoms and throat images) independently evaluated by 4–9 physicians. After excluding 102 cases where physicians could not reach a clear majority, 640 cases were used for model training and evaluation. We compared six architectures: Image-Only (CNN, Vision Transformer), Symptoms-Only, and Multimodal Fusion (Late Fusion, Gated Fusion, Cross-Attention). Models were evaluated using 5-fold stratified cross-validation, with separate analysis for High-Consensus (N=199) and Low-Consensus (N=441) cases. Results: On High-Consensus cases, all image-based models achieved excellent accuracy (95–96%) with high specificity (>96%). On Low-Consensus cases, accuracy dropped to 76–78%, matching the inter-physician agreement rate. Vision Transformer demonstrated the best overall performance (84.84% accuracy, 70.5% AUC). Multimodal fusion provided minimal benefit over image-only approaches. All models maintained high specificity (>96%) but exhibited low sensitivity (2–15%) for bacterial pharyngitis, reflecting conservative prediction patterns suitable for rule-out screening but requiring human oversight for suspected bacterial cases. Conclusions: Our consensus-based evaluation framework reveals that AI models, like human physicians, perform reliably only on diagnostically clear pharyngitis cases. The high specificity (96–98%) suggests strong potential for reducing unnecessary antibiotic prescriptions through AI-assisted triage, while the low sensitivity (2–15%) necessitates physician review for suspected bacterial infections. This stratified approach provides clinically meaningful insights for safe AI deployment in antibiotic stewardship programs.