Can Generative AI Infer Thinking Style from Language? Evaluating the Utility of AI as a Psychological Text Analysis Tool

Generative AI are not currently the choice technology for text analysis, but prior work suggests they may have some utility to assess dynamics like emotion. The current work builds on this empirical foundation to consider how analytic thinking scores from a large language model chatbot, ChatGPT, are linked to analytic thinking scores from dictionary-based approaches like Linguistic Inquiry and Word Count (LIWC). Using over 16,000 texts from four samples and tested against three prompts and two models (GPT-3.5, GPT-4), the evidence suggests there were small associations between ChatGPT and LIWC analytic thinking scores (meta-analytic effect sizes: .058 < rs < .304; ps < .001). Critically, when given the formula to calculate the LIWC analytic thinking index, ChatGPT performed incorrect mathematical operations in 22.1% of the cases, suggesting basic word and number processing may be unreliable with large language models. Researchers should be cautious when using AI for text analysis.