Trade-off between performance and human-like perception in face recognition models

Deep learning models for face recognition are widely adopted in cognitive neuroscience, yet it is not well understood whether their excellent performance means they “see” faces like humans. Here, we collected a relatively large dataset of human face-similarity judgments to assess how well prominent face recognition models align with human perception. We found that models with high — but not the highest — recognition performance are often best aligned with human similarity judgments. We further tested to what extent a linear transformation could better align model representations with human perceptual similarity. Models with superior recognition performance benefited the least from this transformation, suggesting a deeper mismatch with human perception. Notably, the transformation reduced recognition performance for higher-performing models but gave low-performing models a slight recognition boost. Overall, our results indicate that a trade-off exists between recognition ability and human-like perceptual similarity. This may inform future work on developing more human-like models.