Is the structure of psychopathology encoded in language? Comparing large language models and self-reports

Quantitative nosology aims to capture psychopathology structure that is more fundamental than any particular method or theoretical tradition. However, this structural research has relied on one method: individual responses to standardized questionnaire items (e.g., self-report, clinical interview). The present study takes different approach by asking whether the empirical psychopathology structure reflects constructs that are so socially salient they have been engrained in language. We addressed this question by comparing psychopathology structure derived from self-report to the semantic structure accessed by large language models (LLMs), which represent observations and experiences expressed by millions of people across billions of documents, spanning theoretical and scientific paradigms. Psychopathology constructs were identified from a novel measure—new to LLMs and participants—covering 71 symptom dimensions. The self-report structure (N = 780 adults) was compared to the structure in language derived by 1) factor-analyzing item text embedding similarity from two LLMs, and 2) grouping items with a conversational LLM. Results showed the self-report and LLM structures largely align (77% factors were similar). The few cases of misalignment indicate some psychopathology concepts persist in scientific and cultural discourse that ignore functional symptom relationships. For example, insomnia and nightmares were separate in self-reports yet lumped together by LLMs, suggesting these symptoms are often discussed in similar contexts but are not usually reported by the same people. Overall, our study adds novel insight into the nature of psychopathology dimensions identified by structural research.