Emergence of Emotion Selectivity in Deep Neural Networks Trained to Recognize Visual Objects

Recent neuroimaging studies have shown that the visual cortex plays an important role in representing the affective significance of visual input. The origin of these affect-specific visual representations is debated: they are intrinsic to the visual system versus they arise through reentry from frontal emotion processing structures such as the amygdala. We examined this problem by combining convolutional neural network (CNN) models of the human ventral visual cortex pre-trained on ImageNet with two datasets of affective images. Our results show that (1) in all layers of the CNN models, there were artificial neurons that responded consistently and selectively to neutral, pleasant, or unpleasant images and (2) lesioning these neurons by setting their output to 0 or enhancing these neurons by increasing their gain led to decreased or increased emotion recognition performance respectively. These results support the idea that the visual system may have the intrinsic ability to represent the affective significance of visual input and suggest that CNNs offer a fruitful platform for testing neuroscientific theories. Author Summary The present study shows that emotion selectivity can emerge in deep neural networks trained to recognize visual objects and the existence of the emotion-selective neurons underlies the ability of the network to recognize the emotional qualities in visual images. Obtained using two affective datasets (IAPS and NAPS) and replicated on two CNNs (VGG-16 and AlexNet), these results support the idea that the visual system may have an intrinsic ability to represent the motivational significance of sensory input and CNNs are a valuable platform for testing neuroscience ideas in a way that is not practical in empirical studies.