Replicability and Validity of a New AI Assessment of PTSD from Patient Language: A Sequential Evaluation with Model Pre-registration

AI shows promise in identifying psychopathology through language, but replicability in AI-models remains challenging. We develop a AI-based language assessment of PTSD severity and introduce the Sequential Evaluation with Model Pre-registration to rigorously evaluate its validity and replicability. This design includes two phases: Development with pre-registration and Evaluation. Data included development (N=1437) and prospective (N=346) samples, where participants described their lives during automated interviews. In the prospective sample, pre-registered models correlated with PTSD CheckList scores (r=.38, p<.001); converged with PTSD diagnosis (AUC=.76; outperforming demographics and trauma exposures: AUC=.61, p<.01). We found that for each standard deviation increase, mental healthcare expenditure rose by $696.5 (p<.001). Our pre-registered PTSD model assessments are replicable in prospectively collected clinical data and showed external validity against expense criteria. With further development, such models can be used to screen for PTSD or monitor treatment response, especially in telehealth or automated interviews, where deployment can be seamless.