Deep generative networks reveal the tuning of neurons in IT and predict their influence on visual perception

Finding: the tuning of visual neurons has kept neuroscientists busy for decades. One approach to this problem has been to test specific hypotheses on the relevance of a visual property (e.g. orientation or color), build a set of “artificial” stimuli that vary along that property and then record neural responses to those stimuli. Here, we present a complementary, data-driven method to retrieve the tuning properties of visual neurons. Exploiting deep generative networks and electrophysiology in monkeys, we first used a method to reconstruct any stimulus from its evoked neuronal activity in the inferotemporal cortex (IT). Then, by arbitrarily perturbing the response of individual cortical sites in the model, we generated naturalistic and interpretable sequences of images that strongly influence neural activity of that site. This method enables the discovery of previously unknown tuning properties of high-level visual neurons that are easily interpretable, which we tested with carefully controlled stimuli. When we knew which images drove the neurons, we activated the cells with electrical microstimulation and observed a predicable shift of the monkey perception in the direction of the preferred image. By allowing the brain to tell us what it cares about, we are no longer limited by our experimental imagination.