Can LLMs be used to Quantify the Emotional Salience of Text Statements using an Elo Rating System?

When studying mood or affective state, it is useful to collect free-text input from participants relevant to their mood alongside traditional assessment batteries. Quantifying free-text data in relation to quantitative measures remains a challenge. We propose a novel application of an Elo rating system for analysing verbatim human text descriptions of emotionally salient experiences. By using crowdsourced pairwise comparisons, our approach preserves the richness of free-text data while generating a quantitative representation. We apply the same approach to various LLMs to compare their performance against humans. Regression analyses indicate that rankings generated by LLMs predict human rankings, demonstrating strong explanatory power across models. Applying LLMs to this task offers the added advantages of greater efficiency and lower cost. We discuss the potential applications of this method in human research and explore critical considerations regarding the use of LLMs in emotionally relevant tasks.