Computational reconstruction of mental representations using human behavior

Revealing the contents of mental representations is a longstanding goal of cognitive science. However, there is currently no framework for reconstructing representations of multiple high-level visual concepts. We asked participants to indicate what they perceived in images made of random visual features in a deep neural network. We then inferred associations between the semantic features of their responses and the visual features of the images. This allowed us to reconstruct the mental representation of virtually any common visual concept, both those supplied by participants and other concepts extrapolated from the same semantic space. We successfully validated many of these reconstructions in separate participants. We further generalized the approach to predict behavior for new stimuli and in a new task, and to reconstruct representations of individual observers and of a neural network. This framework enables a large-scale investigation of conceptual representations.