{"paper_id":"0529deb7-9edf-41ee-bc5c-356768bf1c93","body_text":"FACE-SELECTIVE RESPONSES CORRELATE WITH DEEP\nNETWORKS THAT LEARN FROM ENVIRONMENT FEEDBACK\nA PREPRINT\nMo Zhou\nDepartment of Psychology and Neuroscience\nBoston College\nBoston, MA 02467\nzhouabx@bc.edu\nEmily Schwartz\nDepartment of Psychology and Neuroscience\nBoston College\nBoston, MA 02467\nArish Alreja\nNeuroscience Institute and Machine Learning Department\nCarnegie Mellon University\nPittsburgh, PA 15213\nR. Mark Richardson\nMassachusetts General Hospital\nand Harvard Medical School\nBoston, MA 02115\nAvniel Ghuman\nDepartment of Neurological Surgery\nUniversity of Pittsburgh\nPittsburgh, PA 15260\nStefano Anzellotti\nDepartment of Psychology and Neuroscience\nBoston College\nBoston, MA 02467\nstefano.anzellotti@bc.edu\nABSTRACT\nDeep neural networks have shown high accuracy in modeling neural responses in the visual system,\nbut most models rely on supervised learning, which requires training on ground-truth labels that are\ntypically unavailable in real-world settings. While unsupervised models can address this limitation,\nthey miss another key aspect: visual representations are shaped by feedback from the environment.\nWe introduce a reinforcement learning (RL) model of face perception that incorporates both input\nstimuli and feedback from the environment. Inspired by human interactions, we train the model\nto approach faces yielding positive interactions and avoid faces yielding negative interactions. Us-\ning intracortical electroencephalography (iEEG) data and Representational Dissimilarity Matrices\n(RDMs), we evaluate the model’s ability to account for neural responses. Our RL model performs\nat the same level as supervised and unsupervised models, capturing neural responses to complex\nvisual stimuli. The findings suggest that RL models are a promising approach for understanding\nperception.\nSignificance Statement\nUnderstanding how the brain encodes faces is central to vision science. Existing models rely on supervised learning,\nwhich requires ground-truth labels that are often unavailable in real-world settings, or on unsupervised learning, which\nignores the role of environmental-feedback in shaping visual representations. We introduce a reinforcement learning\n(RL) model that learns through environmental feedback, simulating human interactions by associating approaching\nfaces with positive interactions and avoiding faces with negative interactions. Using intracortical electroencephalogra-\nphy (iEEG) data from face-selective regions, we show that an RL model with a variational DenseNet encoder accounts\nfor neural representations comparably to supervised and unsupervised models. Task and architecture jointly shaped\nrepresentational geometry, highlighting the importance of both learning objective and encoder design. These findings\nsuggest the potential of RL-based approaches to understand neural representations of naturalistic faces.\n.CC-BY 4.0 International licenseavailable under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprintthis version posted February 27, 2026. ; https://doi.org/10.64898/2026.02.25.703652doi: bioRxiv preprint \n\nFace-selective responses correlate with deep networks that learn from environment feedback A P REPRINT\n1 Introduction\nFaces encode a wealth of socially-relevant information, such as a person’s identity and expressions. What are the\ncomputational mechanisms that give rise to neural representations of faces? Recent work modeled neural responses\nto faces and other objects using supervised deep neural network models that learn by comparing the outputs of the\nmodels to ground-truth labels (Khaligh-Razavi and Kriegeskorte, 2014; Cadieu et al., 2014; Schwartz et al., 2023a).\nThese models yielded promising results, but rely on ground-truth labels that are not generally available to human\nobservers. To address this limitation, more recent studies used unsupervised models - that do not need ground-truth\nlabels for training - (Higgins et al., 2021; Zhuang et al., 2021; Konkle and Alvarez, 2022), succeeding at predicting\nneural responses with accuracy that matches that of supervised models.\nCurrent unsupervised models typically learn face and object representations without taking into consideration the\ndownstream behavioral tasks that observers need to perform. For example, some models are trained to build accurate\nimage reconstructions (Higgins et al., 2021), other models are trained to minimize the dissimilarity between the repre-\nsentation of different images that correspond to a same object (Konkle and Alvarez, 2022). However, several studies\nindicate that visual representations are also shaped by the tasks animals need to perform, and in particular by the\nbehavioral relevance of distinguishing between different sets of stimuli. For example, when macaques are trained to\ndistinguish between different categories of stimuli to receive a reward, more neurons in inferotemporal cortex become\ntuned for the stimulus properties that are relevant to make that categorization (even when counterbalancing which\nproperties are relevant across subjects; Sigala and Logothetis, 2002; De Baene et al., 2008). Converging evidence\ncomes from recent work in mice, that used two-photon imaging to show that after category learning ventral visual\nregions are associated with an increase in the fraction of neurons responsive to the trained task (Goltstein et al., 2021).\nTogether, these studies indicate that the task-relevant feedback an animal receives from the environment contributes to\nshaping their visual representations.\nUnsupervised models of neural responses have difficulty accounting for this kind of dependence of visual represen-\ntations on the feedback an observer receives from the environment. We sought to address this gap in the case of face\nperception, using a model trained with a simple approach-avoidance task. While such approach-avoidance does not\ncome close to the full complexity of the behaviors that are informed by face perception in naturalistic settings, rep-\nresentations learned using this task can provide a lower bound of the correspondence that can be obtained between\nface-selective neural responses and models trained with information that is available in real-world social interactions.\nTo evaluate this, we trained a deep neural network model to approach individuals with whom they have positive interac-\ntions, and to avoid individuals with whom they have negative interactions (see Materials and Methods for details of the\ninteractions). We then asked whether this model could account for neural responses to faces recorded by intracranial\nelectrodes.\nThe model was trained in a simulated environment that included individuals with different identities. In order to\naccount for the variability in the valence of repeated interactions with the same individual, each identity was associated\nwith a probability distribution representing the reward that would be obtained by approaching it. Each identity was\nalso associated with a set of face images varying in viewpoint, illumination, and other properties. At each trial, the\nmodel received a face image as input, and chose whether or not to approach the associated identity, receiving a reward\nas a result of its choice. The model was trained to maximize its reward.\nWe used representational similarity analysis (Kriegeskorte et al., 2008) to quantify the correspondence between the\nrepresentations learned by this model and neural responses to faces recorded with intracortical electroencephalography\n(iEEG). We then compared the correspondence obtained with the model to those obtained with deep supervised and\nunsupervised models that were built with matching architectures and trained with the same dataset.\n2 Materials and Methods\n2.1 Participants\nThe experimental protocols were approved by the University of Pittsburgh’s Institutional Review Board. Informed\nconsent was acquired from all participants (see also Li et al., 2019; Boring et al., 2021 for details, which used the\nsame data). The sample comprised 11 participants (mean age = 31.8 years, SD = 9.89; 7 females) who underwent\nintracranial electroencephalography (iEEG) electrode (surface and depth) implantation for seizure onset localization.\nOne participant was removed before performing additional analysis due to noisy data. None of the participants showed\nevidence of epileptic activity on electrodes located in the ventral and lateral temporal lobes.\n2\n.CC-BY 4.0 International licenseavailable under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprintthis version posted February 27, 2026. ; https://doi.org/10.64898/2026.02.25.703652doi: bioRxiv preprint \n\nFace-selective responses correlate with deep networks that learn from environment feedback A P REPRINT\n2.2 Experimental Design\n2.2.1 Experiment paradigm\nAt the beginning of the experiment, all participants completed a functional localizer task that was used to identify\nface-selective electrodes. Next, they completed the main task. In the main task, each trial consisted of a face image\n(presented for 1000ms) followed by a 500ms inter-trial interval during which a fixation cross was displayed. Partici-\npants were asked to identify the gender of the presented faces, as rapidly and accurately as possible. The face images\npresented to participants were chosen from the Karolinska Directed Emotional Faces (KDEF) dataset (Lundqvist et al.,\n1998). This dataset contains 4900 images featuring 70 individuals (50% female) displaying seven distinct expressions\nfrom five angles. In this study, we used a subset of expressions, including happy, sad, fearful, angry, and neutral\nexpressions. Each face identity and expression combination was depicted from various angles, including frontal view\n(0 degrees), left and right 45-degree views, and left and right 90-degree profiles.\nIn addition, the participants were divided into two subsets, that completed slightly different versions of the experiment\n(A and B). In version A, participants completed one 200-trial session, viewing a stimuli set comprising eight identities,\nfive expressions, and five viewpoint angles (left/right profile, left/right 45 degree, and frontal). Each stimulus was\ndisplayed three times. In version B, participants completed at least two sessions viewing a distinct KDEF subset of\n600 images, comprising 40 identities, five expressions, and three viewpoint angles (profile, 45 degree, and frontal).\nEach stimulus was presented only once.\n2.2.2 Data processing\nData was pre-processed at the University of Pittsburgh (Li et al., 2019; Boring et al., 2021). The data encompasses\n14 depth and 11 surface electrodes that recorded local field potentials at 1000 Hz. Both types of electrodes had\ncomparable recording site surface areas and recorded similar neural responses.\nTo extract single-trial potential signals, the raw data underwent band-pass filtering using a fourth order Butterworth\nfilter, with frequencies between 0.2 Hz and 115 Hz preserved. Subsequently, slow and linear drift components, as\nwell as high-frequency noise, were removed. Additionally, a 60 Hz line noise was eliminated using a stop-band\nencompassing the range of 55-65 Hz. Single-trial potentials (referred to as stP) were time-locked to the onset of each\nstimulus, with the signal sampled at 1000 Hz. Artifacts were reduced by examining raw data, with no ictal events\nobserved. Trials exceeding 5 standard deviations above the mean maximum amplitude were excluded, as were trials\nwith a difference of ≥25 µV between consecutive sampling instances, resulting in <1% trial removed.\n2.2.3 Electrode localization\nElectrode location (Fig. 1C) was ascertained using an automated co-registering method (Hermes et al., 2010). Post-\noperative CT scans were aligned with anatomical MRI scans to sectionalize electrode contacts pre-surgery. Pre- and\npost-operative imaging scans were also used to localize SEEG electrodes. Face-selective electrodes were determined\nby analyzing functional localizer data. An electrode was deemed face-selective if it significantly differentiated faces\nfrom other object categories (Li et al., 2019; Boring et al., 2021).\nFace-selective electrode localization. Across the 11 consented participants, a total of 1,079 electrodes were im-\nplanted, and 25 electrodes (2.3%) were face selective. Of the 25 electrodes, 12 were in the ventral stream (ventral\ntemporal and occipital cortex anterior to V2), including 10 in fusiform gyrus (localization determined with Neu-\nrosynth; Yarkoni et al., 2011). One of these electrodes did not pass the reliability analysis and was excluded from\nfurther analyses. The remaining 24 electrodes are shown in Fig. 1C. Eight were in the lateral stream (lateral temporal\nand lateral occipital cortex anterior to V2, including V3d, V5/MT, and STS); and four were outside these streams\n(labeled as “other”).\nElectrode coverage. A total of 24 face-selective electrodes were identified across 10 subjects (after excluding one\nsubject due to noisy data). The number of electrodes contributed by each subject was as follows: P16 (1 electrode),\nP23 (1 electrode), P27 (5 electrodes), P28 (1 electrode), P30 (1 electrode), P34 (4 electrodes), P36 (3 electrodes), P39\n(1 electrode), P41 (6 electrodes), and P47 (1 electrode).\n2.3 Statistical Analysis\n2.3.1 Deep convolutional neural network models\nWe deployed distinct deep convolutional neural networks (DCNNs), varying in architecture and learning mechanisms,\nto model the neural data. In order to compare different learning mechanisms while keeping the architectures as\n3\n.CC-BY 4.0 International licenseavailable under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprintthis version posted February 27, 2026. ; https://doi.org/10.64898/2026.02.25.703652doi: bioRxiv preprint \n\nFace-selective responses correlate with deep networks that learn from environment feedback A P REPRINT\ncomparable as possible, three networks used the same encoder architecture, but varied in their learning mechanism: a\nsupervised model (SUP focused on identity (ID) classification, an unsupervised model (UNSUP) aimed to reconstruct\nthe original image, and a reinforcement learning model (RL) was trained to predict the expected reward associated\nwith interacting with a particular person (identity). We used identity classification rather than gender as the supervised\nobjective because identity is more challenging and more informative for characterizing face-selective representations,\nwhereas gender is typically easier and less informative. These models were constructed based on a residual neural\nnetwork architecture – ResNet-18 (He et al., 2016; see Fig. 1A), that was also used in recent work on face perception\n(Schwartz et al., 2023a). In addition to these three models trained on the ResNet encoder architecture, in order\nto facilitate comparison to different encoder architectures, we also included three models (SUP, UNSUP, and RL)\nconstructed using a combination of densely connected architecture – DenseNet (Huang et al., 2017), and Variational\nAutoencoders – V AEs (Kingma and Welling, 2013). In addition – since the RL model with this encoder architecture\nshowed competitive correlations with neural responses (see Fig. 1D) – we used this architecture as the basis to test\na multi-task model (VIB UNSUP+RL) which combined image reconstruction and reward prediction objectives using\nthis architecture (see Fig. 1A, B & Fig. 5A). The details of the models’ architectures are described in the following\nsection.\nFig. 1 | Deep network models and correlations with neural responses. A. Two types of encoder architectures.\nB. Neural network architectures for the supervised (SUP), unsupervised (UNSUP), and reinforcement learning (RL)\nmodels. C. Face-selective electrode locations (n = 24). D. Cumulative Kendall τ correlations between face-selective\niEEG RDMs and the RDMs from each model averaged over electrodes (n = 24).\n2.3.2 DCNN architectures\nIn a first set of three models, we used a standard residual neural network (ResNet-18) architecture as the encoder\n(He et al., 2016). In such encoder, input images were processed through a series of convolutional layers with batch\nnormalization and ReLU activation functions (see Table 1).\nWhile we used the same encoder architecture for the three models, each model was based on a different decoder\narchitecture that varied depending on the task the model needed to perform. Each decoder operated on the represen-\ntations derived from the encoder module. For the ResNet SUP model, a classifier was utilized to perform identity\nclassification based on the feature representations. The classifier consisted of one fully connected layer (see Table 2).\nFor the ResNet UNSUP model, the decoder component first generated a vector of means (µ) and a vector of standard\ndeviations (σ) using fully connected layers. These two parameters are essential for the image reconstruction process\n(see details in the following paragraph on V AEs). The rest of the decoder component mirrored a symmetric structure\n4\n.CC-BY 4.0 International licenseavailable under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprintthis version posted February 27, 2026. ; https://doi.org/10.64898/2026.02.25.703652doi: bioRxiv preprint \n\nFace-selective responses correlate with deep networks that learn from environment feedback A P REPRINT\nTable 1 | ResNet encoder architecture.\nLayer Output Size ResNet-18\nencConv1 128 × 128 3 × 3, 64, stride 1\nencConv2 128 × 128\n\u0014\n3 × 3, 64\n3 × 3, 64\n\u0015\n× 2\nencConv3 64 × 64\n\u0014\n3 × 3, 128\n3 × 3, 128\n\u0015\n× 2\nencConv4 32 × 32\n\u0014\n3 × 3, 256\n3 × 3, 256\n\u0015\n× 2\nencConv5 16 × 16\n\u0014\n3 × 3, 512\n3 × 3, 512\n\u0015\n× 2\nto that of the encoder. Transpose convolutional layers were utilized for upsampling, ensuring the preservation of spa-\ntial information (see Table 3). Finally, the decoder component of the ResNet RL model was based on a network that\ngenerated as output the predicted reward that would be derived from interacting with a face. Based on this output, the\nnetwork then chose whether or not to interact with the face, and a loss was computed based on the chosen interaction\n(or lack thereof). The details of the loss function and training procedure are described in the following section. The\nnetwork utilized five fully connected layers, with the first four including ReLU activation functions (see Table 4).\nTable 2 | ResNet SUP model: decoder architecture.\nLayer In features Out features Input Source\nlayer1 512 1503 encConv5\nTable 3 | ResNet UNSUP model: decoder architecture. bn: batch normalization.\nLayer Kernel Stride Input Size Output Size Input Source\nµ N/A N/A 512 × 16 × 16 256 encConv5\nσ N/A N/A 512 × 16 × 16 256 encConv5\ndecFC N/A N/A 256 512 × 16 × 16 z\ndecConv1 4 × 4 2 512 × 16 × 16 256 × 32 × 32 bn(256)\nbn(256) N/A N/A 256 × 32 × 32 256 × 32 × 32 decConv1\ndecConv2 4 × 4 2 256 × 32 × 32 128 × 64 × 64 bn(256)\nbn(128) N/A N/A 128 × 64 × 64 128 × 64 × 64 decConv2\ndecConv3 4 × 4 2 128 × 64 × 64 64 × 128 × 128 bn(128)\nbn(64) N/A N/A 64 × 128 × 128 64 × 128 × 128 decConv3\ndecConv4 3 × 3 1 3 × 128 × 128 3 × 128 × 128 bn(64)\nbn(3) N/A N/A 3 × 128 × 128 3 × 128 × 128 decConv4\nIn a second set of three additional models, we used an encoder architecture that combines the inherent strengths of\nDenseNet (Huang et al., 2017) and variational autoencoders – V AEs (Kingma and Welling, 2013). The inclusion\nof DenseNet-like skip-connections in the encoder improved identity classification performance based on the latent\nrepresentations generated by the encoder, making it possible to compare the learning mechanisms building on the\nsame encoder architecture.\nIn the encoder, input images were processed through a series of convolutional layers with batch normalization and\nReLU activation functions (see Table 5). The encoder consists of three dense blocks, each comprising three con-\nvolutional layers. Within each dense block, feature maps from different layers are concatenated along the channels\ndimension. This design choice promotes feature reuse and alleviates the vanishing gradient problem. After the dense\n5\n.CC-BY 4.0 International licenseavailable under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprintthis version posted February 27, 2026. ; https://doi.org/10.64898/2026.02.25.703652doi: bioRxiv preprint \n\nFace-selective responses correlate with deep networks that learn from environment feedback A P REPRINT\nTable 4 | ResNet RL model: decoder architecture\nLayer In features Out features Input Source\nlayer1 512 1024 FC\nrelu1 1024 1024 layer1\nlayer2 1024 512 relu1\nrelu2 512 512 layer2\nlayer3 512 256 relu2\nrelu3 256 256 layer3\nlayer4 256 128 relu3\nrelu4 128 128 layer4\nlayer5 128 1 relu4\nblocks, the resulting feature maps are then fed into fully connected layers, which generate a vector of means (µ) and\na vector of standard deviations (σ ) of the latent space (as in a typical variational autoencoder; Kingma and Welling,\n2013). These vectors represent the parameters of a Gaussian distribution associated with a particular input stimulus.\nTo incorporate stochasticity in the model and enable probabilistic sampling, the reparameterization trick is applied.\nThis trick involves sampling latent variables by adding random noise ϵ drawn from a Gaussian distribution with zero\nmean and unit standard deviation. The latent variablez is computed as z = µ + σ · ϵ. Then, z is used as the input to the\ndecoder network, which varies depending on the task (details are provided in the following paragraph). This stochastic\nlayer introduces regularization and enables the use of the network as a generative model (Kingma and Welling, 2013).\nAs in the case of the models using the ResNet encoder, for the VIB DenseNet models, each model was based on a\ndifferent decoder architecture that varied depending on the task. Each decoder operated on the latent representations\nderived from the encoder module. For the VIB SUP model, fully connected layers were employed to generate the\nmean (µ) and standard deviation (σ ) for the latent space after the encoder. Then, a classifier was utilized to perform\nidentity classification based on the feature representations. The classifier consisted of two fully connected layers, with\nthe first layer including a ReLU activation function (see Table 6). For the VIB UNSUP model, the decoder component\nmirrored a symmetric structure to that of the encoder. Transpose convolutional layers were utilized for upsampling,\nensuring the preservation of spatial information (see Table 7). Finally, the decoder component of the VIB RL model\nwas based on a network built to predict reward using the latent variable z from the encoder as input. It utilized five\nfully connected layers, with the first four including ReLU activation functions (see Table 8).\n2.3.3 Loss functions\nEach of the models had a corresponding loss function designed for the task the model needed to complete. The SUP\nmodels used a standard Cross-Entropy loss for classification. The UNSUP models used a reconstruction loss, and the\nRL models used an RL loss. In addition, all of the models using the VIB encoder architecture have a KL-divergence\nloss component added to the overall loss function. The VIB UNSUP+RL model used the following loss function:\nL = α||x − ˆ x||2 + βDKL (q||p)|\n{z }\nV AEloss\n+ γL(ω)RL| {z }\nRLloss\n.\nWhere x is an image of a person’s face,α = 4000, β = 0.0000001, and γ = 1.5.\nThe loss L(ω)RL is a reinforcement learning loss constructed as follows. Each person identity i is associated with\na reward distribution N (µi, σi). We modeled the reward distribution per identity as Gaussian to capture stochastic\nfeedback expected from natural social interactions. The mean µi encodes identity-specific expected value, while σi\ncaptures uncertainty. The network h produces as output the expected reward ˆr. The model chooses to interact with\nthe person in the image with probability eˆr\n1+eˆr . If the model chooses to interact with the person (ω = 1), it obtains\na reward r ∼ N (µi, σi), and computes a ‘prediction error’ loss given by the square of the difference between the\nobtained reward and the expected reward. Instead, if the model chooses to not interact with the person (ω = 0), it\ncomputes an ‘opportunity cost’ loss associated with not interacting.\n6\n.CC-BY 4.0 International licenseavailable under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprintthis version posted February 27, 2026. ; https://doi.org/10.64898/2026.02.25.703652doi: bioRxiv preprint \n\nFace-selective responses correlate with deep networks that learn from environment feedback A P REPRINT\nTable 5 | VIB DenseNet encoder architecture. bn: batch normalization.\nBlock Layer Kernel Stride Input Size Output Size Input Source\nConv 1 bn1(3) N/A N/A 3 × 128 × 128 3 × 128 × 128 Images\nencConv1 4 × 4 2 3 × 128 × 128 64 × 64 × 64 bn1\nbn2(64) N/A N/A 64 × 64 × 64 64 × 64 × 64 encConv1\nDense 1 encConv2 3 × 3 1 64 × 64 × 64 64 × 64 × 64 bn2\nbn3(128) N/A N/A 128 × 64 × 64 128 × 64 × 64 encConv1, encConv2\nencConv3 3 × 3 1 128 × 64 × 64 64 × 64 × 64 bn3\nbn4(192) N/A N/A 192 × 64 × 64 192 × 64 × 64 encConv1, encConv2, encConv3\nencConv4 3 × 3 1 192 × 64 × 64 64 × 64 × 64 bn4\nbn5(64) N/A N/A 64 × 64 × 64 64 × 64 × 64 encConv4\nAvgPool2d(2) N/A N/A 64 × 64 × 64 64 × 32 × 32 bn5\nDense 2 encConv5 3 × 3 1 64 × 32 × 32 64 × 32 × 32 AvgPool2d(2)\nbn6(128) N/A N/A 128 × 32 × 32 128 × 32 × 32 encConv4, encConv5\nencConv6 3 × 3 1 128 × 32 × 32 64 × 32 × 32 bn6\nbn7(192) N/A N/A 192 × 32 × 32 192 × 32 × 32 encConv4, encConv5, encConv6\nencConv7 3 × 3 1 192 × 32 × 32 64 × 32 × 32 bn7\nbn8(64) N/A N/A 64 × 32 × 32 64 × 32 × 32 encConv7\nDense 3 encConv8 3 × 3 1 64 × 32 × 32 64 × 32 × 32 bn8\nbn9(128) N/A N/A 128 × 32 × 32 128 × 32 × 32 encConv7, encConv8\nencConv9 3 × 3 1 128 × 32 × 32 64 × 32 × 32 bn9\nbn10(192) N/A N/A 192 × 32 × 32 192 × 32 × 32 encConv7, encConv8, encConv9\nencConv10 3 × 3 1 192 × 32 × 32 64 × 32 × 32 bn10\nbn11(64) N/A N/A 64 × 32 × 32 64 × 32 × 32 encConv10\nFC µ N/A N/A 64 × 32 × 32 2048 bn11\nσ N/A N/A 64 × 32 × 32 2048 bn11\nTable 6 | VIB SUP model: decoder architecture.\nLayer In features Out features Input Source\nlayer1 2048 2048 FC\nrelu1 2048 2048 layer1\nlayer2 2048 1503 relu1\nL(ω)RL =\n\u001a(r − ˆr)2 if ω = 1\nˆr + λ · c otherwise\nThe ‘opportunity cost’ is computed with:\nλ = 1 − current epoch\ntotal epoch c = 1 − min\nidentities\nˆr\nTherefore, at the start of training, the opportunity cost was positive and greater or equal to1 for all identities, promoting\nexploration. This choice was inspired by the idea of the optimistic initial value in reinforcement learning. As training\n7\n.CC-BY 4.0 International licenseavailable under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprintthis version posted February 27, 2026. ; https://doi.org/10.64898/2026.02.25.703652doi: bioRxiv preprint \n\nFace-selective responses correlate with deep networks that learn from environment feedback A P REPRINT\nTable 7 | VIB UNSUP model: decoder architecture. bn: batch normalization.\nBlock Layer Kernel Stride Input Size Output Size Input Source\ndecFC decFC1 N/A N/A 2048 64 × 32 × 32 z\nbn12(64) N/A N/A 64 × 32 × 32 64 × 32 × 32 decFC1\nDense 1 decConv1 3 × 3 1 64 × 32 × 32 64 × 32 × 32 bn12\nbn13(128) N/A N/A 128 × 32 × 32 128 × 32 × 32 decFC1, decConv1\ndecConv2 3 × 3 1 128 × 32 × 32 64 × 32 × 32 bn13\nbn14(192) N/A N/A 192 × 32 × 32 192 × 32 × 32 decFC1, decConv1, decConv2\ndecConv3 3 × 3 1 192 × 32 × 32 64 × 32 × 32 bn14\nbn15(64) N/A N/A 64 × 32 × 32 64 × 32 × 32 decConv3\nDense 2 decConv4 3 × 3 1 64 × 32 × 32 64 × 32 × 32 bn15\nbn16(128) N/A N/A 128 × 32 × 32 128 × 32 × 32 decConv3, decConv4\ndecConv5 3 × 3 1 128 × 32 × 32 64 × 32 × 32 bn16\nbn17(192) N/A N/A 192 × 32 × 32 192 × 32 × 32 decConv3, decConv4, decConv5\ndecConv6 3 × 3 1 192 × 32 × 32 64 × 32 × 32 bn17\nbn18(64) N/A N/A 64 × 32 × 32 64 × 32 × 32 decConv6\nDense 3 decConv7 3 × 3 1 64 × 32 × 32 64 × 32 × 32 bn18\nbn19(128) N/A N/A 128 × 32 × 32 128 × 32 × 32 decConv6, decConv7\ndecConv8 3 × 3 1 128 × 32 × 32 64 × 32 × 32 bn19\nbn20(192) N/A N/A 192 × 32 × 32 192 × 32 × 32 decConv6, decConv7, decConv8\ndecConv9 3 × 3 1 192 × 32 × 32 64 × 32 × 32 bn20\ninterpolate N/A N/A 64 × 32 × 32 64 × 64 × 64 decConv9\nbn21(64) N/A N/A 64 × 64 × 64 64 × 64 × 64 interpolate\ndecConv decConv10 4 × 4 2 64 × 64 × 64 3 × 128 × 128 bn21\nbn22(3) N/A N/A 3 × 128 × 128 3 × 128 × 128 decConv10\nTable 8 | VIB RL model: decoder architecture.\nLayer In features Out features Input Source\nlayer1 2048 1024 FC\nrelu1 1024 1024 layer1\nlayer2 1024 512 relu1\nrelu2 512 512 layer2\nlayer3 512 256 relu2\nrelu3 256 256 layer3\nlayer4 256 128 relu3\nrelu4 128 128 layer4\nlayer5 128 1 relu4\nprogressed, and the model’s predictions of the rewards that would result from interacting with different face images\nimproved, the decay of λ led the opportunity cost to converge to the predicted rewards ˆr.\nNote that the opportunity cost loss is essential to prevent the network’s outputs to collapse to small numbers. In fact,\nin its absence, the network learns to predict small values of ˆr: this prevents the model from incurring in the prediction\n8\n.CC-BY 4.0 International licenseavailable under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprintthis version posted February 27, 2026. ; https://doi.org/10.64898/2026.02.25.703652doi: bioRxiv preprint \n\nFace-selective responses correlate with deep networks that learn from environment feedback A P REPRINT\nerror loss, because the prediction error loss is incurred by the model only if it chooses to interact, and the probability\nof interacting is eˆr\n1+eˆr , which is low for low ˆr values.\n2.3.4 Training and testing\nAll models were trained using the CelebA dataset (Liu et al., 2018) consisting of over 300,000 images. In order to\nmatch the size of the dataset with the recent work (Schwartz et al., 2023a) to facilitate comparison between studies, a\nsubset of CelebA was utilized. This subset contained 28,709 training images and an additional 3,589 labeled testing\nimages, featuring a total of 1,503 unique identities. Random selection was employed to ensure a minimum of 20\nimages per identity. All images were uniformly resized to dimensions of 128 × 128 pixels and RGB scale.\nFollowing the training, the ResNet SUP and VIB SUP networks were tested on their ability to perform identity recog-\nnition using the CelebA dataset. The testing extracts the output from each network for every image in the dataset.\nResNet SUP model was able to recognize identity, achieving an accuracy of 22.76 % on a left-out subset of CelebA\n(chance level = 0.07%). VIB SUP model was able to recognize identity, achieving an accuracy of 18.5% on a left-\nout subset of CelebA (the classification accuracy was 6.88% without the DenseNet architecture). Other results about\nmodel performances will be described in the “Results” section below.\n2.3.5 Temporal localizer\nIn order to compare the representations in DCNNs to neural representations, we first aimed to identify temporal\nwindows during which face-selective electrodes yielded the most consistent responses. The data time-series was\nsegmented into successive, non-overlapping 50ms time windows, starting from 25ms pre-stimulus onset up to 525ms\npost-onset. We correlated all instances of a stimulus’s neural response within each time window and calculated an\naverage correlation across all time windows for that stimulus. The overall average correlation was then subtracted\nfrom the time-window-specific correlations. We then conducted a paired t-test to distinguish time windows with above-\naverage test-retest reliability (p < 0.05). Following Bonferroni correction for multiple comparisons, one electrode was\nomitted from the RDM analysis due to a lack of reliable response time windows (p > 0.05).\n2.3.6 Computing representational similarity analysis for neural data\nAiming to preserve as much data as possible, we initially analyzed all face-selective electrodes, including those ex-\nposed to every stimulus once. We calculated Representational Dissimilarity Matrices (RDMs) for three temporal\nwindows (125ms - 175ms, 175ms - 225ms, 225ms - 275ms) based on previous research on visual face perception’s\ntiming. A 50-dimensional vector was derived for each temporal window per electrode, with each dimension represent-\ning the response level per millisecond of the 50ms window. Using correlation distance, we measured the dissimilarity\nbetween each pair of stimulus response patterns to obtain neural RDMs.\nFollowing this procedure, RDMs of dimensions 200 × 200 were generated for Experiment A (5 expressions times 5\nviewpoints times 8 identities), while RDMs of dimensions 600 × 600 were obtained for Experiment B (5 expressions\ntimes 3 viewpoints times 40 identities). It is important to note that the sizes of the RDMs differed between the two\nexperiments because different subsets of KDEF images were used in Experiment A and Experiment B, as described\nin the “Experimental paradigm” section. Specifically, in Experiment B, the information regarding viewpoint only\nincluded the viewpoint angle, without distinguishing between left and right viewpoints. Consequently, the feature\nvectors for the left and right viewpoints were averaged, resulting in the averaging of left and right profile views as well\nas left and right 45 degree views.\n2.3.7 Computing representational similarity for DCNNs\nTo study the similarity between the representations in different networks, we used representational similarity analysis\n(RSA). Specifically, for the VIB DenseNet encoder architecture, features from the Variational Information Bottleneck\n(VIB) representations were extracted; for the ResNet encoder architecture, features from the final layer in the last\nresidual blocks were extracted. For each of the models, we calculated representational dissimilarity matrices (RDMs)\nusing a three-step procedure that has been used in previous studies (Schwartz et al., 2023a). First, feature vectors were\nextracted for all KDEF images employed in the experiment. Subsequently, the feature vectors were mean-centered\nby subtracting the mean feature vector across all KDEF images. Finally, for each pair of images, the correlation\ndistance between their mean-centered feature vectors was computed using Pearson’s correlation coefficient ( r), and\nthe correlation distance was defined as 1 − r.\n9\n.CC-BY 4.0 International licenseavailable under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprintthis version posted February 27, 2026. ; https://doi.org/10.64898/2026.02.25.703652doi: bioRxiv preprint \n\nFace-selective responses correlate with deep networks that learn from environment feedback A P REPRINT\n2.3.8 Comparing RDMs\nOnce the RDMs for each of the electrodes and time windows were computed, as well as the RDMs for the different\ncomputational models, the similarity between neural RDMs and model RDMs was calculated using the Kendallτ rank\ncorrelation coefficient (τB) (Fig. 1D). In addition, Kendall’s τB was also used to calculate the similarity between the\nRDMs of different computational models (Fig. 2B).\n3 Results\n3.1 Deep neural network models of face perception\nSix deep neural network models of face perception were trained using a subset of the CelebA dataset (Liu et al., 2018).\nIn order to evaluate models that learn based on feedback from the environment, we implemented a deep convolutional\nneural network (DCNN) trained with the approach-avoidance task described in the Introduction. Because the model\nwas trained to maximize its reward through interactions with its environment, it is a type of reinforcement learning\nmodel, and we will refer to it as such throughout the article (“RL model”). However, it should not be confused with\nother kinds of reinforcement learning models, such as deep Q-learning, that use different training mechanisms.\nIn addition to the RL model, we implemented two comparison DCNNs: one trained with a supervised task (“SUP\nmodel”), and one trained with an unsupervised task (“UNSUP model”). Given an image of a person’s face as input,\nthe SUP model was trained to classify the identity of that face. The UNSUP model was trained to reconstruct the input\nimage as accurately as possible. The RL model was trained to predict the reward that would result from interacting\nwith the identity associated with that face image, and to approach identities yielding positive rewards. The RL model\ndiffers from the SUP model in two important ways. First, in the RL model, during training, different identities\ncan be associated with similar rewards. By contrast, in the SUP models, different identities are always associated\nwith different labels. Second, for the RL model there is a trade-off between acquiring information and receiving\nreward: choosing to approach identities associated with negative rewards provides information, but it also has a cost.\nSupervised and unsupervised DCNNs have been used to model neural responses to faces in previous work (Tsantani\net al., 2021; Schwartz et al., 2023a; Higgins et al., 2021). Here, we aimed to test whether RL models, that do not have\naccess to the ground truth labels but can receive feedback from the environment, can also provide comparable results\nwhen they are equipped with the same encoder architecture.\nThe performance of a model and its correspondence with neural responses can depend not only on the task the model\nis trained with (e.g. SUP, UNSUP, or RL), but also on the model’s architecture. In order to evaluate the impact of\nthe architecture, we tested models built on two different encoder architectures (Fig. 1A). ResNet-18 was selected as\none of the encoder architectures because ResNet architectures are widely used in neuroscience (Wen et al., 2018;\nSchwartz et al., 2023a; Dobs et al., 2023); therefore, facilitating comparison with previous studies. A modified version\nof a variational model was selected as the other: this choice was motivated by recent results showing that variational\narchitectures learn factors that show correspondence with the tuning properties of neurons in inferotemporal cortex\n(Higgins et al., 2021). The original version of this variational architecture did not perform well at the supervised\ntask; therefore, we modified it with the addition of dense connections (Huang et al., 2017); this change improved its\nperformance at supervised identity classification. For each encoder, we used three different decoders: each decoder\nwas tailored to the type of output the model needed to produce (Fig. 1B). While some of the models shared the same\nencoder architecture, each model was individually trained end-to-end to perform one task only. Thus, even models\nwith the same encoder architecture ultimately learned different encoder weights.\nFirst, we ensured that the RL models successfully learned to interact with the faces yielding positive rewards. Indeed,\nthere were significant increase in rewards after training the RL models. We trained the RL models ten times using both\nResNet and VIB DenseNet encoder. For the ResNet RL model, the average reward before training was -0.255 (SEM\n= 0.033), and the average reward after training was 12.604 (SEM = 0.047). For the VIB RL model, the average reward\nbefore training was 0.500 (SEM = 0.041), and the average reward after training was 10.671 (SEM = 0.209). The SUP\nand UNSUP models also learned successfully to perform their tasks – details of their loss function can be found in the\nSupplemental Material.\n3.2 Comparing face-selective neural responses to deep networks trained with feedback from the environment\nTo quantify the correspondence between models and neural responses, we collected intracortical electroencephalog-\nraphy (iEEG) data from 11 participants while they observed images of faces varying in identity, expression, and\nviewpoint (from the Karolinska Directed Emotional Faces dataset: KDEF; Lundqvist et al., 1998). Participants were\nasked to classify the gender of the faces (see Rossion et al., 2011; Ghuman et al., 2014; Li et al., 2019; Boring et al.,\n10\n.CC-BY 4.0 International licenseavailable under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprintthis version posted February 27, 2026. ; https://doi.org/10.64898/2026.02.25.703652doi: bioRxiv preprint \n\nFace-selective responses correlate with deep networks that learn from environment feedback A P REPRINT\n2021). We analyzed 24 face-selective electrodes from 10 participants (for details, see Materials and Methods and\nFig. 1C).\nHaving identified the face-selective electrodes, we next sought to compare representations measured by these elec-\ntrodes to representations in the DCNN models. Because neural responses change dynamically over time, and their\ntimecourse can vary from region to region, we compared models to neural responses separately for different time win-\ndows (see Materials and Methods and Schwartz et al., 2023a for more details), consistent with previous research on the\ntiming of face processing in the human brain (Rossion et al., 2011). For each electrode and time window, we computed\nneural representational dissimilarity matrices (RDMs). We also computed RDMs obtained from the DCNN models,\nby feeding as input to the DCNNs the same KDEF images that were shown to the participants. Finally, Kendall’s\nτ was used to compare the neural RDMs to the RDMs obtained from the DCNNs (Fig. 1D). For all six models we\ntrained, the DCNN RDMs were computed based on features extracted from the final layer of the models’ encoders.\nWe first tested the correspondence between the RL models’ RDMs, and the RDMs obtained from neural responses\nin face selective electrodes. Kendall τ correlations between the model RDMs and the neural RDMs were positive\n(Fig. 1D). The Kendall τ correlations between model RDMs and neural RDMs were different for the SUP, UNSUP,\nand RL models using the ResNet encoder. The model trained with the RL task showed lower correlations with neu-\nral responses. The main effect of model type was significant (F (2, 213) = 5.626, p = 0.004, see Fig. 1D). In\ncontrast, when using the VIB DenseNet encoder, the Kendall τ correlations were similar across all models, with a\nnon-significant main effect for the model type (F (2, 213) = 1.098, p = 0.336, see Fig. 1D). This indicates that,\nwhen using the VIB DenseNet encoder, the RL model achieved comparable performance to the SUP and UNSUP\nmodels. Additionally, to further investigate the influence of architecture within the RL training framework, we directly\ncompared the correlations of the VIB RL and ResNet RL models with neural RDMs using a pairedt-test across partic-\nipants. For this analysis, the correlation values for each participant were averaged across all face-selective electrodes\nand time windows. The VIB RL model showed significantly higher correlations with neural RDMs compared to the\nResNet RL model (t(9) = 2 .565, p = 0.030). Since this analysis was a pre-specified comparison limited to the RL\nmodels, we did not apply a correction for multiple comparisons.\nThe correspondence between the model RDMs and neural RDMs was highest in the first time window (125ms -\n175ms), and decreased in subsequent time windows (Fig. 1D). This decrease was consistent across all six models\n(ResNet SUP, ResNet UNSUP, ResNet RL, VIB SUP, VIB UNSUP, and VIB RL). In order to determine whether\nthe models captured the similarity between neural responses to different images more accurately than simple pixel-\nbased metrics, we additionally performed a comparison between the neural RDMs and RDMs computed based on the\nPearson’s correlation distance between the pixel values for each pair of images. The results (represented by the gray\nbars in Fig. 1D) showed considerably lower Kendallτ values when using the pixel-based similarity compared with the\nsix deep network models.\nIn sum, three main findings emerged from the analyses: first, features from deep network models captured neural\nrepresentations more accurately than pixel-based metrics, second, the correspondence between deep network models\nand neural responses was highest in the 125ms - 175ms time window, third, most importantly, reinforcement learning\nmodels with a VIB DenseNet encoder architecture performed comparably to supervised and unsupervised models at\naccounting for neural responses in face-selective electrodes.\n3.3 Studying the impact of architecture and task mechanisms on representational geometry\nSince we trained models with the same encoder architecture using different tasks, as well as models that differ in\nterms of their encoder architecture but share the same task, we were able to study the impact of the architecture and\nof the task on the models’ representational geometry, and on their correspondence with neural responses. To do this,\nwe first computed representational dissimilarity matrices for each model (Fig. 2A), and then the dissimilarity between\nthe dissimilarity matrices themselves (Fig. 2B). In addition, we computed the dissimilarity between models in terms\nof how they correspond to neural responses. Specifically, for each model, we computed a vector of Kendall’s τ\ncorrelations between the model RDM and neural RDMs across all face-selective electrodes and time windows. These\nvectors reflect the extent to which each model matches the representational structure of neural data. We then computed\nthe euclidean distance between these vectors (Fig. 2C) to quantify how similarly two models explain the neural data.\nThe choice of euclidean distance is motivated by the observation that the magnitude of the Kendallτ values is important\nto establish the dissimilarity between models (in addition to the pattern ofτ values across electrodes), and dissimilarity\nmetrics based on correlation or cosine distance would not take into account such differences in magnitude. Note that\ntwo models that have intermediate levels of dissimilarity to each other (i.e. models that correlate with Kendallτ values\nsmaller than 1) might be similar in terms of their correspondence to neural responses if the correspondence is driven\nby overlapping parts of the models’ variance, alternatively, they might be different in their correspondence to neural\n11\n.CC-BY 4.0 International licenseavailable under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprintthis version posted February 27, 2026. ; https://doi.org/10.64898/2026.02.25.703652doi: bioRxiv preprint \n\nFace-selective responses correlate with deep networks that learn from environment feedback A P REPRINT\nresponses if it is driven by non-overlapping parts of their variance. Thus, the direct similarity between two models\nmight be different from the similarity in the extent to which those models match with neural responses.\nThe RDMs obtained from the ResNet SUP and ResNet RL models were different from each other and from those\nobtained with all other models. Conversely, the ResNet UNSUP model, despite having the same encoder architecture,\nlearned more similar representations to the VIB models than to the other ResNet models (Fig. 2B). These patterns\nindicate that the task used to train the models had an impact on the learned representations within a given encoder\narchitecture, but models trained with same tasks need not be most similar across architectures. In addition, model\narchitecture had an impact on the results as well. Models using the VIB DenseNet encoder showed more similar\nrepresentations to each other, despite the differences in the decoder architectures and the tasks (Fig. 2B).\nIn terms of the models’ ability to capture neural responses, task was an important driver of the similarity between\nmodels. The SUP models were similar to each other, despite their different encoder architectures (ResNet vs VIB\nDenseNet). This was also the case for the two UNSUP models, but not for the RL models: there was a greater\ndifference between the ResNet RL model and the VIB RL model in terms of their ability to account for neural RDMs.\nIn addition, the ResNet RL model was very different from the SUP models in how it accounted for neural responses,\nregardless of the encoder architecture being used (Fig. 2C).\n3.4 Using deep networks to study the differences between ventral and lateral face-selective regions\nAfter comparing face-selective neural responses to deep networks trained with different learning mechanisms, we\nstudied whether these models provide insights into the distinct functional roles of face-selective representations en-\ncoded in ventral temporal regions and lateral temporal regions. The ventral and lateral streams have been hypothesized\nto be specialized respectively for the recognition of face identity and facial expressions. However, this view has been\nrecently challenged (Duchaine and Yovel, 2015; Li et al., 2019; Schwartz et al., 2023b), raising the question of what\nare the functional roles of the two streams. Instead of differing based on their role for identity and expressions, rep-\nresentations in ventral and lateral regions might differ in the extent to which they are shaped by feedback from the\nenvironment. Feedback-dependent neural tuning has been reported in inferior temporal cortex (Sigala and Logothetis,\n2002), and thus ventral regions might be better captured by the model trained with the approach-avoidance task. Alter-\nnatively, all models could account similarly well for responses in ventral and lateral regions, and these regions might\ndiffer along other dimensions. To determine this, we tested whether the multivariate patterns of correspondences be-\ntween neural responses and multiple different model types provide information that differentiates between ventral and\nlateral electrodes.\nTogether, a total of six models were evaluated. Since for each model we computed the correspondence with neural\nresponses in three time windows, each electrode was associated with 18 Kendallτ values (one for each of the 6 models\nand 3 time windows). We used a logistic regression to classify between ventral and lateral electrodes based on these\n18-dimensional patterns of Kendallτ values, using a leave-one-electrode-out cross-validation procedure. This resulted\nin a classification accuracy of 75% – the datapoints and the separating hyperplane are shown in Fig. 3A, projected on\nthe first three principal components for visualization purposes. This result shows that it was possible to distinguish\nbetween ventral and lateral electrodes based on the 18-dimensional patterns of correspondences across models and\ntime windows.\nTo investigate more precisely what information contributes to distinguishing between ventral and lateral electrodes,\nwe extracted the β values from the logistic regression (Fig. 3B). A positive β value for a particular model and time\nwindow means that greater Kendall τ values between that model and neural responses in that time window in a\nparticular electrode indicate that the electrode is in ventral temporal cortex. Conversely, if theβ value for a model and\ntime window is negative, greater correspondence between the model and neural responses in that window indicates\nthat the electrode is in lateral temporal cortex. Therefore, if β values exhibit significant variation across model types,\nit would show that different models contribute differently to distinguishing between ventral and lateral responses.\nFollowing the same logic, significant variation in β values across time windows would indicate that different time\nwindows contribute differently to distinguishing between ventral and lateral responses.\nIn order to quantify these effects, we performed a three-way ANOV A testing the variation in β values across model\ntypes, time windows, and brain regions. The three-way interaction was non-significant (F (10, 324) = 0.744, p =\n0.682), as were the interaction between model type and time window (F (10, 324) = 1 .137, p = 0.333) and the\ninteraction between model type and brain region (F (5, 324) = 1.746, p = 0 .124). The interaction between time\nwindow and brain region (ventral vs lateral) was significant (F (2, 324) = 24.897, p < 0.001). This indicates that\nacross time, ventral and lateral electrodes varied in terms of their correspondence to the models. All main effects were\nsignificant (model type: F (5, 324) = 4 .874, p < 0.001; time window: F (2, 324) = 48 .696, p < 0.001; brain region:\nF (1, 324) = 55 .974, p < 0.001).\n12\n.CC-BY 4.0 International licenseavailable under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprintthis version posted February 27, 2026. ; https://doi.org/10.64898/2026.02.25.703652doi: bioRxiv preprint \n\nFace-selective responses correlate with deep networks that learn from environment feedback A P REPRINT\nFig. 2 | Model comparisons. A. RDMs of the latent representations of all models (based on KDEF images used in\nversion A of the experiment).B. Kendall’sτ ranking distance between the RDMs of the different models.C. Euclidean\ndistance between the models’ match with neural responses. The match was computed as cumulative Kendall τ values\nbetween model RDMs and neural RDMs.\nIn the first two time windows all models had positiveβ values, indicating a greater correspondence with ventral RDMs.\nWhat drives this effect? Previous work has demonstrated that lateral regions exhibit stronger responses to dynamic\nstimuli (Pitcher et al., 2019). We hypothesize that experiments using dynamic stimuli might lead to more reliable\nresponses in lateral regions, and that relatively lower Kendall τ values with lateral electrodes might be due to the lack\nof dynamic information in our stimuli. In the third time windowβ values dropped to near-zero. This drop is likely due\nto a floor effect: Kendall τ values between the models and neural responses were overall low in the third time window.\n3.5 Studying the unique contributions of different models\nDifferent models can account for distinct or overlapping variance in the neural RDMs. To investigate the degree of\noverlap between different models, and their unique contributions to capturing the representational structure of neural\ndata, we employed a semi-partial Kendall τ analysis. This approach allowed us to isolate and quantify the correlation\nbetween each model and neural RDMs, controlling for each other model one at a time. The semipartial Kendall\n13\n.CC-BY 4.0 International licenseavailable under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprintthis version posted February 27, 2026. ; https://doi.org/10.64898/2026.02.25.703652doi: bioRxiv preprint \n\nFace-selective responses correlate with deep networks that learn from environment feedback A P REPRINT\nFig. 3 | Differences between ventral and lateral electrodes. A.Separating hyperplane for the classification of ventral\n(green) vs lateral (blue) electrodes based on Kendall τ values, plotted in the space spanned by the first three principal\ncomponents. B. Coefficients (β) of the logistic regression.\nτ analysis thus offers a more nuanced understanding of how each model aligns with, or diverges from, the neural\nrepresentations observed in the iEEG data.\nFig. 4 | Semi-partial Kendall τ correlations. Heatmaps show semi-partial Kendall τ between face-selective iEEG\nRDMs and model RDMs across time windows. Rows denote the predictor (row model’s unique contribution); columns\ndenote the controlled model (partialled out).\nTo compute semi-partial Kendall τ, we regressed out the RDMs of the computational models in the columns of Fig. 4\nfrom those in the rows, and correlated the residuals with the neural data. The results showed that the supervised models\n(ResNet SUP and VIB SUP) contain a considerable amount of unique information compared to models trained with the\nother tasks. This finding underscored the ability of SUP models to explain neural variance that remain unaccounted for\nby other models. Furthermore, the unique contribution of the models using VIB DenseNet encoder architecture also\nsuggested their ability to capture unique neural activity. Together, these findings pointed to the ability of supervised\nmodels and VIB DenseNet architectures to explain neural responses that other models and architectures could not.\n3.6 Modeling face representations with a combination of unsupervised learning and feedback from the\nenvironment\nIn realistic settings, rewards are encountered only occasionally. By contrast, unsupervised experience is plentiful.\nConsidering that visual representations are shaped by the feedback an animal receives (Sigala and Logothetis, 2002),\n14\n.CC-BY 4.0 International licenseavailable under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprintthis version posted February 27, 2026. ; https://doi.org/10.64898/2026.02.25.703652doi: bioRxiv preprint \n\nFace-selective responses correlate with deep networks that learn from environment feedback A P REPRINT\nbut also that unsupervised models have been reported to do well at accounting for neural responses to objects (Zhuang\net al., 2021; Konkle and Alvarez, 2022), the visual system might rely on a combination of unsupervised learning\nand feedback from the environment. Combining these different types of learning, however, can pose challenges.\nChanges in neural representations that improve performance on the unsupervised task might worsen performance\non the approach-avoidance task, and vice versa. To evaluate this, we tested whether a model of face perception\nthat combines these two types of learning can still complete both types of tasks accurately, and whether it acquires\nrepresentations that correspond more closely to neural responses.\nSpecifically, we trained a model that uses the VIB DenseNet encoder architecture, but differs from previous models\nwe tested in that the output of the encoder is then fed as input to two decoders (VIB UNSUP+RL, Fig. 5A). The first\ndecoder reconstructs the image (from which we calculated an unsupervised loss), while the second decoder determines\nwhether or not to interact with the person in the image (from which we calculated a reinforcement learning loss). This\nnetwork is trained to minimize the sum of the two losses, so that the encoder is pushed to learn representations that\ncan be used to reconstruct the original images, and also to perform the approach-avoidance task.\nFig. 5 | Combination of unsupervised learning and reinforcement learning. A. Neural network architectures for\nthe VIB UNSUP+RL model. B. Example of image reconstruction from VIB UNSUP model and VIB UNSUP+RL\nmodel. C. Average reward obtained before and after training for VIB RL model and VIB UNSUP+RL model. D.\nCumulative Kendall τ correlations between face-selective iEEG RDMs and the RDMs from each model averaged over\nelectrodes (n = 24).\nAfter training this “combined” model, we first aimed to evaluate its performance on the tasks for which it was trained.\nWe compared its ability to reconstruct images to the model trained exclusively to reconstruct images (the unsupervised\nmodel), finding that the reconstructions of the combined model were comparatively less accurate, but preserved the\noverall appearance of the face (Fig. 5B). We also compared the combined model’s ability to maximize reward, finding\nthat the model was able to improve its decisions during training and to obtain significantly larger amounts of reward at\nthe end of the training procedure (Fig. 5C, VIB UNSUP+RL), although the amount of reward it obtained after training\nwas significantly lower than the reward obtained by the model trained exclusively with approach-avoidance (Fig. 5C,\nVIB RL). In sum, although the combined model did not perform as accurately as the models trained exclusively with\none task, it was nonetheless able to perform both tasks.\nNext, we evaluated the correspondence between the combined model and neural responses. Compared to the unsu-\npervised model trained exclusively with image reconstruction and to the reinforcement learning model, the combined\nmodels’ correlations with neural responses were numerically higher (Fig. 5D). These findings underscore that while\nthe combined model did not perform as accurately as the models trained exclusively with one task, it nonetheless\nshowed a performance that was at least equivalent to that of the VIB UNSUP and VIB RL models in accounting for\nneural responses in face-selective electrodes, with a non-significant main effect for the model type (F(2, 213) = 0 .484,\np = 0.617). More evidence will be needed to determine whether combining unsupervised learning and feedback from\nthe environment can yield representation that provide a closer match to neural responses.\n15\n.CC-BY 4.0 International licenseavailable under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprintthis version posted February 27, 2026. ; https://doi.org/10.64898/2026.02.25.703652doi: bioRxiv preprint \n\nFace-selective responses correlate with deep networks that learn from environment feedback A P REPRINT\n4 Discussion\nDeep network models trained based on feedback from the environment (RL) were able to account for neural responses\ncomparably for both supervised and unsupervised models. This is particularly significant considering that the super-\nvised models tested in this study were trained to recognize face identity: such supervised models have been shown to\nperform well at capturing face-selective neural responses, outperforming supervised models trained with other tasks\nsuch as facial expression classification (Schwartz et al., 2023a).\nUnlike SUP models, that require ground-truth labels which are not usually available in realistic settings, RL models can\nlearn through more realistic interactions with the environment. In addition, unlike UNSUP models, the representations\nin RL models do not depend exclusively on the input stimuli, but also on the reward the models receives from the\nenvironment. In this respect, VIB RL models can account for previous results showing that the visual system is shaped\nby animals’ interactions with the environment and by the tasks they are trained to perform (Sigala and Logothetis,\n2002; De Baene et al., 2008).\nModels trained based on feedback from the environment were only able to obtain comparable performance to super-\nvised and unsupervised models when using a novel encoder architecture, that combines the strengths of variational\nencoders (Kingma and Welling, 2013) and of densely connected neural networks – DenseNets (Huang et al., 2017).\nBy contrast, when using a ResNet architecture (He et al., 2016), representations learned with a supervised identity\nclassification task correlated more strongly with neural responses compared to the representations learned with the\nother models tested (UNSUP and RL).\nRecent work found that deep RL models such as deep Q networks can capture neural responses across multiple brain\nregions while participants play Atari games (Cross et al., 2021). In addition to correlating with neural responses in\nfrontal and parietal regions, the models’ representations also correlated with responses in regions in ventral temporal\ncortex known to encode perceptual representations (Cross et al., 2021). Temporal and frontal regions interact during\nreinforcement learning (Gershman and Daw, 2017). Previous studies focused on Atari games that display simple\nstylized visual stimuli. Therefore, a model could capture neural responses to such stimuli, but might still be unable to\naccount for responses to more complex and nuanced stimuli like human faces. While the present work uses a different\nmodel architecture (VIB DenseNet), it indicates that models that learn based on feedback from the environment have\nthe potential to account for the representation of naturalistic images.\nThe task performed by the RL model, while being more realistic than categorization based on ground-truth labels,\nis still a simple approximation of real-life interactions. Despite this, the VIB RL model was still able to account\nfor neural responses comparably to the VIB SUP and VIB UNSUP models. This finding suggests that, when using\nthe VIB DenseNet encoder, RL models are a viable approach to capture perceptual representations. We hypothesize\nthat improving the realism and complexity of the reinforcement learning task in future studies will also improve the\ncorrespondence between the models’ representations and neural responses.\nThe architecture of the models had an impact on the results. Having been trained on the same dataset, models em-\nploying the ResNet encoder architecture and models employing the VIB DenseNet encoder architecture demonstrated\na different degree of correspondence with neural responses. This was especially true in the RL models: when training\non the same dataset and task, the VIB RL model showed significantly higher correlations with neural responses than\nthe ResNet RL model. Two aspects of the architecture differed between the two models. First, unlike the ResNet\nRL model, the VIB RL model is based on a combination of DenseNet and variational autoencoders (V AEs) architec-\nture. Second, the VIB RL model includes a probabilistic bottleneck, which injects stochasticity during training, and\na regularization term in the loss (the KL-divergence). Either of these features (or both) could have contributed to the\ndifference in performance. The task the networks were trained on (and the associated decoder architecture) also had\nan impact on the results, even when the encoder architecture was matched. In fact, both the ResNet UNSUP model\nand the ResNet RL model did not perform as well as the ResNet SUP model, showing that the training task (and the\nassociated decoder) contributed to the correspondence between models and neural responses even when the encoder\narchitecture was held constant.\nWe anticipated the VIB UNSUP+RL combined model to exhibit greater correspondence to neural responses, but its\ncorrelation with neural data was not significantly different from that of the VIB UNSUP model and the VIB RL model.\nThe combined model’s reconstruction performance and the reward it achieved after training were lower compared to\nthe models trained specifically for one single task. Within the multi-task model, the optimization process for one task\nmight have affected negatively the optimization of the other task (Sener and Koltun, 2018). This finding encourages\nthe search for other approaches to combine unsupervised learning and reinforcement learning that can perform well at\nboth tasks. Such approaches might also yield representations that provide a closer match to neural responses.\n16\n.CC-BY 4.0 International licenseavailable under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprintthis version posted February 27, 2026. ; https://doi.org/10.64898/2026.02.25.703652doi: bioRxiv preprint \n\nFace-selective responses correlate with deep networks that learn from environment feedback A P REPRINT\nOur analysis also revealed insights about the differences between neural representations in ventral and lateral temporal\nregions. Thanks to the temporal resolution of iEEG, we were able to investigate differences between the two streams\nacross multiple time windows. First, we found a significant variation in the correspondence of these regions with the\nmodels across different time windows. Second, we observed that overall the models’ correspondence with ventral\nelectrodes was greater than with lateral ones, particularly in the initial time windows. This disparity might be due to\nthe nature of the stimuli used to train our DCNNs. Previous research has reported that lateral regions have stronger\nresponses to dynamic stimuli (Pitcher et al., 2019), whereas our models were trained on static images. This might\naccount for the lower correspondence observed with the lateral electrodes. This discrepancy could potentially be\naddressed by making use of deep networks that can process dynamic stimuli (Lotter et al., 2016; Feichtenhofer et al.,\n2019).\nReferences\nBoring, M. J., Silson, E. H., Ward, M. J., Richardson, R. M., Fiez, J. A., Baker, C. I., and Ghuman, A. S. (2021).\nMultiple adjoining word-and face-selective regions in ventral temporal cortex exhibit distinct dynamics. Journal of\nNeuroscience, 41(29):6314–6327.\nCadieu, C. F., Hong, H., Yamins, D. L., Pinto, N., Ardila, D., Solomon, E. A., Majaj, N. J., and DiCarlo, J. J.\n(2014). Deep neural networks rival the representation of primate it cortex for core visual object recognition. PLoS\ncomputational biology, 10(12):e1003963.\nCross, L., Cockburn, J., Yue, Y ., and O’Doherty, J. P. (2021). Using deep reinforcement learning to reveal how the\nbrain encodes abstract state-space representations in high-dimensional environments. Neuron, 109(4):724–738.\nDe Baene, W., Ons, B., Wagemans, J., and V ogels, R. (2008). Effects of category learning on the stimulus selectivity\nof macaque inferior temporal neurons. Learning & Memory, 15(9):717–727.\nDobs, K., Yuan, J., Martinez, J., and Kanwisher, N. (2023). Behavioral signatures of face perception emerge\nin deep neural networks optimized for face recognition. Proceedings of the National Academy of Sciences,\n120(32):e2220642120.\nDuchaine, B. and Yovel, G. (2015). A revised neural framework for face processing. Annual review of vision science,\n1:393–416.\nFeichtenhofer, C., Fan, H., Malik, J., and He, K. (2019). Slowfast networks for video recognition. In Proceedings of\nthe IEEE/CVF international conference on computer vision, pages 6202–6211.\nGershman, S. J. and Daw, N. D. (2017). Reinforcement learning and episodic memory in humans and animals: an\nintegrative framework. Annual review of psychology, 68:101–128.\nGhuman, A. S., Brunet, N. M., Li, Y ., Konecky, R. O., Pyles, J. A., Walls, S. A., Destefino, V ., Wang, W., and Richard-\nson, R. M. (2014). Dynamic encoding of face information in the human fusiform gyrus. Nature communications,\n5(1):1–10.\nGoltstein, P. M., Reinert, S., Bonhoeffer, T., and H¨ubener, M. (2021). Mouse visual cortex areas represent perceptual\nand semantic features of learned visual categories. Nature neuroscience, 24(10):1441–1451.\nHe, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the\nIEEE conference on computer vision and pattern recognition, pages 770–778.\nHermes, D., Miller, K. J., Noordmans, H. J., Vansteensel, M. J., and Ramsey, N. F. (2010). Automated electro-\ncorticographic electrode localization on individually rendered brain surfaces. Journal of neuroscience methods,\n185(2):293–298.\nHiggins, I., Chang, L., Langston, V ., Hassabis, D., Summerfield, C., Tsao, D., and Botvinick, M. (2021). Unsu-\npervised deep learning identifies semantic disentanglement in single inferotemporal face patch neurons. Nature\ncommunications, 12(1):6456.\nHuang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. (2017). Densely connected convolutional networks.\nIn Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708.\nKhaligh-Razavi, S.-M. and Kriegeskorte, N. (2014). Deep supervised, but not unsupervised, models may explain it\ncortical representation. PLoS computational biology, 10(11):e1003915.\nKingma, D. P. and Welling, M. (2013). Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114.\nKonkle, T. and Alvarez, G. A. (2022). A self-supervised domain-general learning framework for human ventral stream\nrepresentation. Nature communications, 13(1):491.\n17\n.CC-BY 4.0 International licenseavailable under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprintthis version posted February 27, 2026. ; https://doi.org/10.64898/2026.02.25.703652doi: bioRxiv preprint \n\nFace-selective responses correlate with deep networks that learn from environment feedback A P REPRINT\nKriegeskorte, N., Mur, M., and Bandettini, P. A. (2008). Representational similarity analysis-connecting the branches\nof systems neuroscience. Frontiers in systems neuroscience, page 4.\nLi, Y ., Richardson, R. M., and Ghuman, A. S. (2019). Posterior fusiform and midfusiform contribute to distinct stages\nof facial expression processing. Cerebral Cortex, 29(7):3209–3219.\nLiu, Z., Luo, P., Wang, X., and Tang, X. (2018). Large-scale celebfaces attributes (celeba) dataset. Retrieved August,\n15(2018):11.\nLotter, W., Kreiman, G., and Cox, D. (2016). Deep predictive coding networks for video prediction and unsupervised\nlearning. arXiv preprint arXiv:1605.08104.\nLundqvist, D., Flykt, A., and ¨Ohman, A. (1998). Karolinska directed emotional faces. Cognition and Emotion.\nPitcher, D., Ianni, G., and Ungerleider, L. G. (2019). A functional dissociation of face-, body-and scene-selective brain\nareas based on their response to moving and static stimuli. Scientific reports, 9(1):8242.\nRossion, B., Jacques, C., et al. (2011). The n170: Understanding the time course of face perception in the human\nbrain. The Oxford handbook of ERP components, pages 115–142.\nSchwartz, E., Alreja, A., Richardson, R. M., Ghuman, A., and Anzellotti, S. (2023a). Intracranial electroencephalogra-\nphy and deep neural networks reveal shared substrates for representations of face identity and expressions. Journal\nof Neuroscience, 43(23):4291–4303.\nSchwartz, E., O’Nell, K., Saxe, R., and Anzellotti, S. (2023b). Challenging the classical view: recognition of identity\nand expression as integrated processes. Brain Sciences, 13(2):296.\nSener, O. and Koltun, V . (2018). Multi-task learning as multi-objective optimization.Advances in neural information\nprocessing systems, 31.\nSigala, N. and Logothetis, N. K. (2002). Visual categorization shapes feature selectivity in the primate temporal cortex.\nNature, 415(6869):318–320.\nTsantani, M., Kriegeskorte, N., Storrs, K., Williams, A. L., McGettigan, C., and Garrido, L. (2021). Ffa and ofa\nencode distinct types of face identity information. Journal of Neuroscience, 41(9):1952–1969.\nWen, H., Shi, J., Chen, W., and Liu, Z. (2018). Deep residual network predicts cortical representation and organization\nof visual features for rapid categorization. Scientific reports, 8(1):3752.\nYarkoni, T., Poldrack, R. A., Nichols, T. E., Van Essen, D. C., and Wager, T. D. (2011). Large-scale automated\nsynthesis of human functional neuroimaging data. Nature methods, 8(8):665–670.\nZhuang, C., Yan, S., Nayebi, A., Schrimpf, M., Frank, M. C., DiCarlo, J. J., and Yamins, D. L. (2021). Unsu-\npervised neural network models of the ventral visual stream. Proceedings of the National Academy of Sciences,\n118(3):e2014196118.\nSupplemental Material\n18\n.CC-BY 4.0 International licenseavailable under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprintthis version posted February 27, 2026. ; https://doi.org/10.64898/2026.02.25.703652doi: bioRxiv preprint \n\nFace-selective responses correlate with deep networks that learn from environment feedback A P REPRINT\nFig. 6 | Model training losses.\n19\n.CC-BY 4.0 International licenseavailable under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprintthis version posted February 27, 2026. ; https://doi.org/10.64898/2026.02.25.703652doi: bioRxiv preprint \n\nFace-selective responses correlate with deep networks that learn from environment feedback A P REPRINT\nFig. 7 | Model comparisons. A. RDMs of the latent representations of all models (based on KDEF images used in\nversion B of the experiment).B. Kendall’sτ ranking distance between the RDMs of the different models.C. Euclidean\ndistance between the models’ match with neural responses. The match was computed as cumulative Kendall τ values\nbetween model RDMs and neural RDMs.\n20\n.CC-BY 4.0 International licenseavailable under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprintthis version posted February 27, 2026. ; https://doi.org/10.64898/2026.02.25.703652doi: bioRxiv preprint \n\nFace-selective responses correlate with deep networks that learn from environment feedback A P REPRINT\nFig. 8 | Average reward obtained before and after training for ResNet RL model.\n21\n.CC-BY 4.0 International licenseavailable under a \n(which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made \nThe copyright holder for this preprintthis version posted February 27, 2026. ; https://doi.org/10.64898/2026.02.25.703652doi: bioRxiv preprint","source_license":"CC-BY-4.0","license_restricted":false}