Shared Representations of Stability in Humans, Supervised & Unsupervised Deep Neural Networks

At a glance, the human visual system transforms complex retinal images into generic feature representations useful for guiding a wide range of behaviors. Here, we provide evidence that the feature representations embedded in purely feed- forward neural networks are sufficient to explain seemingly high-level human judgments: in this case, the stability of randomly arranged block towers. To do this, we first show that we can linearly decode stability from the features of two deep neural networks – a supervised network trained on ImageNet, and a varia- tional autoencoder trained only to reconstruct images of block towers from various perspectives – neither of which were ever taught stability per se. We then derive a set of image-computable features to use as predictors of performance, finding that the stability judgments of both human subjects and the neural net decoders are best predicted by the same feature. Our findings suggest overall that at least some aspects of seemingly higher-level reasoning in a now paradigmatic intuitive physics task may be grounded in direct readouts of purely perceptual features.