Can Large Language Model Embeddings Predict Test Item Correlations? A Multilevel Analysis

Large language model embeddings show promise for modeling psychometric properties directly from item text. This study provides a rigorous, condition-sensitive validation by testing whether the cosine similarity of sentence embeddings predicts empirical inter-item correlations, and whether this relationship is uniform across different psychological constructs. With a sample of 1010 participants, we analyzed 463 item pairs nested within nine dimensions across four scales (assessing internet gaming disorder, dark triad traits, aggression, and life satisfaction). A multilevel model was fitted with empirical correlation as the outcome, LLM cosine similarity as a fixed effect, and random slopes and intercepts across dimensions. Results confirmed a significant positive fixed effect, indicating that greater semantic similarity predicts stronger empirical correlations. Crucially, significant random slope variance revealed that the strength of this predictive relationship differed substantially across dimensions. Furthermore, a strong negative correlation was found between the random intercepts and slopes, suggesting that for constructs with higher baseline inter-item cohesion, semantic similarity plays a proportionally smaller role. The findings offer conditional support for the theory that semantic overlap is a systematic source of item covariance while highlighting the moderating role of the psychological construct itself.