Clinician-Trained Artificial Intelligence for Enhanced Routing of Patient Portal Messages in the Electronic Health Record

ABSTRACT Background Increased use of electronic health records (EHR) and patient-initiated messaging has led to inefficiency, staff shortages, and provider burnout. We evaluated the impact of a natural language processing (NLP) algorithm for message routing on communication dynamics. Methods We developed an NLP model to accurately label inbound EHR messages from patients using a pre-trained classifier with fine-tuning based on clinician feedback. In a real-world study, the model was deployed to prospectively label and route messages sent to participating physicians at an integrated health system. A parallel control group of unrouted messages was generated for comparative analysis. The primary endpoints for assessing model performance were the time to first message interaction, the time to conversation resolution, and the total number of message interactions by staff, compared with the control group. Secondary endpoints were the precision, recall, F1 score (a measure of positive predictive value and sensitivity), and accuracy for correct message classification. Results The model prospectively labeled and routed 469 unique conversations over 14 days from the inbaskets of participating physicians. Compared to a control group of 402 unrouted conversations from the same time period, conversations in the routed group had an 83.3% reduction in the time to first interaction (median difference [MD], −1 hour; P<0.001), an 84.4% reduction in the time to conversation resolution (MD, −22.5 hours; P<0.001), and a 40% reduction in the total number of staff interactions after application of intervention (MD, −2.0 interactions; P97.6%), recall (>95%), and F1 scores (>96.5%) for accurate prediction of all five message classes, with a total accuracy of 97.8%. Conclusions Real-time message routing using advanced NLP was associated with significantly reduced message response and resolution times.