SEMbeddings: how to evaluate model misfit before data collection using large-language models

Recent developments suggest that Large Language Models (LLMs) offer a viable approach to approximate empirical correlation matrices of item responses through the utilization of item embeddings and their cosine similarities. In this paper, we introduce a novel tool, that we label SEMbeddings, which integrates prior findings with the application of latent measurement models to assess model fit or misfit prior to data collection. To support our statement, we use the 96 items of the VIA-IS-P which measures 24 different character strengths and responses from 31,697 participants to those items. Our analysis reveals a significant correlation (r = .56) between cosine similarities of embeddings and empirical correlations among items. Moreover, we demonstrate the feasibility of fitting confirmatory factor analyses on cosine similarity matrices and interpreting their outcomes using modification indices: We found consistency of the results obtained with SEMbeddings procedures and classical procedures on empirical data. While not always identical, these results suggest that SEMbeddings could serve as a screening tool to inspect model fit and to make informed decisions about items before data collection. With the increasing precision of LLMs, these procedures will produce increasingly reliable results, potentially transforming the way we develop new questionnaires.