Do societal concepts and experiences of psychopathology align? Comparing large language models and self-reports

Society has constructs to understand psychopathology, which do not necessarily correspond to people’s experience of these problems. This study examined alignment between societal and psychological constructs by comparing concepts encoded in natural language to reported symptoms. We used large language models (LLMs) to assess social consensus because they’re trained on ideas expressed in text by millions of individuals. Societal and psychological constructs were identified from a novel measure—new to LLMs and participants—covering 71 psychopathology dimensions. Psychological constructs were estimated by factor-analyzing self-reports (N=780). Multiple methods were used for societal constructs: 1) factor-analyzing item semantic embedding similarity from two LLMs, 2) grouping items with ChatGPT. In results replicated across methods, societal constructs aligned with 77% of self-reported factors with 16 cases of misalignment suggesting constructs based on superficial symptom/behavior (dis)similarity rather than psychological mechanisms. Findings indicate that although common knowledge of psychopathology is largely accurate, some symptoms/behaviors are misunderstood.