Comparing Perceptual Judgments in Large Multimodal Models and Humans

Cognitive scientists commonly collect participants' judgments regarding perceptual characteristics of stimuli to develop and evaluate memory, learning, and decision-making models. For instance, to model human responses in tasks of category learning and item recognition, researchers often have to collect perceptual judgments of images to embed the images in multidimensional feature spaces. This process is time-consuming and costly. Recent advancements in Large Multimodal Models (LMMs) provide a potential alternative since such models can respond to prompts that include both text and images and could potentially replace human participants. To test whether the available LMMs can indeed be useful for this purpose, we evaluated their judgments on a dataset consisting of rock images that has been widely used by cognitive scientists. The dataset includes human perceptual judgments along ten dimensions considered important for classifying rock images. While the models exhibited a strong positive correlation with human responses, we found that they fell short in replacing an average of a set of judgments from human participants. The models provided correlations with these averaged data that were roughly the same magnitude as observed for individual participants, especially for dimensions that are relatively general (such as lightness and chromaticity) as opposed to domain-specific dimensions (such as pegmatitic structure), where they struggled more. We also found that modifying prompts and providing additional examples of images with corresponding ratings had a positive but relatively modest impact on model performance. Our study provides a benchmark for evaluating future LMMs on human perceptual judgment data.