Manifold geometry underlies a unified code for category and category-independent features

Abstract In everyday vision, animals routinely extract from the same visual stimulus both object identity and continuous identity-independent variables such as position and size. It has been shown that linear decoding performance of both kinds of information increases along the ventral stream, suggesting that inferior temporal cortex may be implementing a joint code for object category and category-independent features. A central open question is whether such a code can indeed exist within a single representation and, if so, what geometric properties enable it. Here, we show that convolutional neural networks can develop such a code. We then derive a theory of regression on category manifolds, identifying the key manifold-geometry measures that enable accurate readout of category-independent features, and showing how they can be optimized while preserving manifold properties known to support classification performance. We further characterize how common experimental constraints, such as subsampling neural units and using a limited number of categories, affect the empirical estimation of regression performance. Our findings thus provide a principled understanding of the geometry underlying joint codes and yield testable predictions for future neural recordings probing the joint-code hypothesis in the ventral stream. Competing Interest Statement The authors have declared no competing interest. Footnotes Figure 2 has been both updated and split into two separate figures. The overall presentation of the results has been improved throughout the text. Figure 8 shows slightly different data that was previously reported in the SI, and vice versa. The SI has been updated with more supplementary results, as well as a section reporting more example images and image statistics for our image dataset.