Implementation Report: Integrating Natural Language Processing Approaches in Child and Adolescent Mental Health Services Research Using Normalisation Process Theory

Objective: This implementation report documents the integration of an NLP-based data extraction tool within the Oxford Health NHS Foundation Trust’s Child and Adolescent Mental Health Services (CAMHS) to identify and analyse treatment options and outcomes for anxiety in autistic and ADHD youth using electronic health records (EHRs). We evaluate the implementation process through the lens of Normalisation Process Theory (NPT) to understand how the documented interventions were adopted and to identify factors influencing its normalisation into routine research practice. Methods: A multidisciplinary team implemented an NLP pipeline to automatically extract information on presenting problems, treatments, and outcomes (e.g., engagement, clinical outcomes, side effects) from pseudonymised EHR free-text notes. The implementation followed an iterative seven-phase roadmap: (1) defining research questions; (2) identifying key constructs; (3) co-developing an annotation schema; (4) training human annotators; (5) resolving annotation discrepancies; (6) large-scale annotation of EHR data; and (7) NLP model development and evaluation. Throughout, we applied NPT to guide implementation and evaluate progress. Quantitative outcomes (such as volume of data processed, annotation reliability, and NLP model performance) and qualitative feedback were collected to assess implementation success. Results: The NLP intervention was successfully deployed in a real-world clinical research setting, extracted data from 454,127 clinical notes across 5,906 patients. The annotation process achieved a high inter-annotator agreement (Krippendorff’s α = 0.86; 95% CI 0.83–0.88) after iterative schema refinement, and the trained NLP model demonstrated acceptable accuracy in identifying medications (F1-score = 0.9; 95% CI 0.89, 0.91) and anxiety-related information (F1-score = 0.79; 95% CI 0.77, 0.8). The NPT-based evaluation revealed that coherence of a shared understanding of the NLP tool’s purpose and value was established among the team. Cognitive participation was generally strong, clinicians and researchers showed growing interest and involvement in the NLP-enabled research workflow. Collective action was facilitated by clear protocols, though challenges such as variability in data quality and the learning curve for new technology impeded smooth operation at times. Through reflective monitoring, the team identified needs for ongoing training and infrastructure improvements. No major adverse event or harm was reported, though additional time was required for data access and computational setup, which considerably extended the project timeline.