Bayesian analysis of subjective ranking data using Thurstonian Models: Tutorial, novel methods, and an open-source library

Subjective ranking data are ubiquitous in behavioural research, design and marketing. Subjective ranking data arise when subjects are asked to order a set of items or stimuli according to some criteria. For example, a psychologist may ask subjects to order a set of faces according to their perceived 'friendliness', while an automotive engineer may rank a set of vehicle prototypes according to their subjectively perceived stability. Thurstonian models provide a powerful framework for analysing ranking data, but have not seen widespread use in the behavioural sciences. By modelling discrete rankings as arising from a set of continuous latent variables, Thurstonian models can extend the flexibility of generalised linear models to ranking data. This allows us to estimate aggregate (or "consensus") ranks for a group of participants, incorporate covariates into the models, test for differences between conditions and populations, and assess the degree of agreement between rankings from multiple subjects. Here we provide an introduction to Thurstonian models, and illustrate how these, when coupled with Bayesian estimation approaches, can provide a powerful tool for analysing ranking data. Specifically, using three example datasets, including both simulated data and data from our own research on subjective impressions of driving simulators, we illustrate how we can fit and interpret Thurstonian models. We demonstrate a number of novel approaches for summarising the Bayesian Thurstonian model fits using distance metrics, and provide an open source code library which allows users to flexibly specify and fit Thurstonian models. This implementation uses a novel reformulation of the Thurstonian model, which admits the use of Hamiltonian Monte Carlo, a fast and efficient algorithm for fitting Bayesian models.