Quantifying the informational value of classification images

Reverse correlation is an influential psychophysical paradigm that uses participant’s responses to randomly varying images to build a Classification Image (CI), which is commonly interpreted as a visualization of a participant’s mental representation. It is unclear, however, how to statistically quantify the amount of signal present in CIs, which limits the interpretability of these images. In this paper, we propose a novel metric, infoVal, which assess informational value relative to a resampled random distribution, and can be interpreted as a z-score. In the first part, we define the infoVal metric and show, through simulations, that it adheres to typical Type I error-rates rated under various task conditions (internal validity). In the second part, we show that the metric correlates with markers of data quality in empirical RC data, such as the subjective recognizability, objective discriminability and test-retest reliability of CIs (convergent validity). In the final part, we demonstrate how the infoVal metric can be used to compare the informational value of reverse correlation datasets, comparing data acquired online versus data acquired in a controlled lab environment. We recommend a new standard of good practice where researchers assess infoVal scores of reverse correlation data to ensure that they do not read signal in CIs, where no signal is present. The infoVal metric is implemented in the open source rcicr R-package to facilitate its adoption. This work has now been published in Behavior Research Methods, and can be found here: https://link.springer.com/article/10.3758%2Fs13428-019-01232-2