Deep Neural Network Models of Object Recognition Exhibit Human-Like Limitations when Performing Visual Search Tasks

ABSTRACT To find an object we are looking for, we must recognize it. Prevailing models of visual search neglect recognition, focusing instead on selective attention mechanisms. These models account for performance limitations that participants exhibit when searching highly simplified stimuli often used in laboratory tasks. However, it is unclear how to apply these models to complex natural images of real-world objects. Deep neural networks (DNN) can be applied to any image, and recently have emerged as state-of-the-art models of object recognition in the primate ventral visual pathway. Using these DNN models, we ask whether object recognition explains limitations on performance across visual search tasks. First, we show that DNNs exhibit a hallmark effect seen when participants search simplified stimuli. Further experiments show this effect results from optimizing for object recognition: DNNs trained from randomly-initialized weights do not exhibit the same performance limitations. Next, we test DNN models of object recognition with natural images, using a dataset where each image has a visual search difficulty score, derived from human reaction times. We find DNN accuracy is inversely correlated with visual search difficulty score. Our findings suggest that to a large extent visual search performance is explained by object recognition.