Behaviorally-relevant features of observed actions dominate cortical representational geometry in natural vision

Behaviorally-relevant features of observed actions dominate cortical representational geometry in natural vision | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Behaviorally-relevant features of observed actions dominate cortical representational geometry in natural vision Jane Han, Vassiki Chauhan, Rebecca Philip, Morgan Taylor, Heejung Jung, and 4 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5478816/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted You are reading this latest preprint version Abstract We effortlessly extract behaviorally relevant information from dynamic visual input in order to understand the actions of others. In the current study, we develop and test a number of models to better understand the neural representational geometries supporting action understanding. Using fMRI, we measured brain activity as participants viewed a diverse set of 90 different video clips depicting social and nonsocial actions in real-world contexts. We developed five behavioral models using arrangement tasks: two models reflecting behavioral judgments of the purpose (transitivity) and the social content (sociality) of the actions depicted in the video stimuli; and three models reflecting behavioral judgments of the visual content (people, objects, and scene) depicted in still frames of the stimuli. We evaluated how well these models predict neural representational geometry and tested them against semantic models based on verb and nonverb embeddings and visual models based on gaze and motion energy. Our results revealed that behavioral judgments of similarity better reflect neural representational geometry than semantic or visual models throughout much of cortex. The sociality and transitivity models in particular captured a large portion of unique variance throughout the action observation network, extending into regions not typically associated with action perception, like ventral temporal cortex. Overall, our findings expand the action observation network and indicate that the social content and purpose of observed actions are predominant in cortical representation. Biological sciences/Neuroscience/Cognitive neuroscience/Perception Biological sciences/Psychology/Human behaviour Full Text Additional Declarations There is NO Competing Interest. Cite Share Download PDF Status: Under Review Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5478816","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":384241064,"identity":"3a3ab676-49f0-494f-8027-af549936ac63","order_by":0,"name":"Jane Han","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA6ElEQVRIiWNgGAWjYBACxgYGBmYgLcMgwXwASEvIEK2Fh0GCLQGkhYcom6BaeAwYwAyCytvPHnxcUHOHh392z+dXN2oseBjYDx/dgNdhPXnJxjOOPeORuHN2m3XOMaDDeNLSbuD3S46ZNA/bYR6GG7nbjHPYJEAuNMOvpf+N+W+ef4d55G/kPDPO+UeMlhk5Zsy8bYd5DG7kMD/ObSNKyxtjad6+wzyGd46ZMef2SfCwEfKLYX+O4Weeb4fl5G43P/6c861Ojp/98DH8WhoQbDYJMIlPOQjII7GZPxBSPQpGwSgYBSMTAABkiUW9HxovfgAAAABJRU5ErkJggg==","orcid":"https://orcid.org/0000-0002-7110-6502","institution":"Dartmouth College","correspondingAuthor":true,"prefix":"","firstName":"Jane","middleName":"","lastName":"Han","suffix":""},{"id":384241065,"identity":"447e0c17-2ca9-465b-96a5-c7e47cdfbb35","order_by":1,"name":"Vassiki Chauhan","email":"","orcid":"https://orcid.org/0000-0002-9612-2402","institution":"Dartmouth College","correspondingAuthor":false,"prefix":"","firstName":"Vassiki","middleName":"","lastName":"Chauhan","suffix":""},{"id":384241066,"identity":"9128c332-db79-42dc-a57a-04b832d55a49","order_by":2,"name":"Rebecca Philip","email":"","orcid":"","institution":"Dartmouth College","correspondingAuthor":false,"prefix":"","firstName":"Rebecca","middleName":"","lastName":"Philip","suffix":""},{"id":384241067,"identity":"d1697565-7c1d-46bf-84de-11373105ba60","order_by":3,"name":"Morgan Taylor","email":"","orcid":"","institution":"Brandeis University","correspondingAuthor":false,"prefix":"","firstName":"Morgan","middleName":"","lastName":"Taylor","suffix":""},{"id":384241068,"identity":"626bf024-6122-4f63-88eb-091dc53d411a","order_by":4,"name":"Heejung Jung","email":"","orcid":"","institution":"Dartmouth College","correspondingAuthor":false,"prefix":"","firstName":"Heejung","middleName":"","lastName":"Jung","suffix":""},{"id":384241069,"identity":"a0953842-7211-4a69-bc7f-7af3a0720b6e","order_by":5,"name":"Yaroslav Halchenko","email":"","orcid":"","institution":"Dartmouth College","correspondingAuthor":false,"prefix":"","firstName":"Yaroslav","middleName":"","lastName":"Halchenko","suffix":""},{"id":384241070,"identity":"ebe745a3-218b-4533-8f89-1178cc7e1ec5","order_by":6,"name":"M. Ida Gobbini","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"M.","middleName":"Ida","lastName":"Gobbini","suffix":""},{"id":384241071,"identity":"88b84a7f-ac8a-41cb-9a9e-2095dfeaa876","order_by":7,"name":"James Haxby","email":"","orcid":"https://orcid.org/0000-0002-6558-3118","institution":"Dartmouth College","correspondingAuthor":false,"prefix":"","firstName":"James","middleName":"","lastName":"Haxby","suffix":""},{"id":384241072,"identity":"e130165e-263f-489d-8720-ef1840806eab","order_by":8,"name":"Samuel Nastase","email":"","orcid":"https://orcid.org/0000-0001-7013-5275","institution":"Princeton University","correspondingAuthor":false,"prefix":"","firstName":"Samuel","middleName":"","lastName":"Nastase","suffix":""}],"badges":[],"createdAt":"2024-11-18 22:10:16","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5478816/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5478816/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":73176695,"identity":"6f66647c-619c-4b65-a514-d120a7997088","added_by":"auto","created_at":"2025-01-07 12:15:23","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2174951,"visible":true,"origin":"","legend":"Article File","description":"","filename":"ActionHanNastase.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5478816/v1_covered_72487e07-e5c0-4dbd-9f90-f93935dd5d1d.pdf"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"Behaviorally-relevant features of observed actions dominate cortical representational geometry in natural vision","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"nature-portfolio","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"","title":"Nature Portfolio","twitterHandle":"","acdcEnabled":false,"dfaEnabled":false,"editorialSystem":"ejp","reportingPortfolio":"","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-5478816/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5478816/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"We effortlessly extract behaviorally relevant information from dynamic visual input in order to understand the actions of others. In the current study, we develop and test a number of models to better understand the neural representational geometries supporting action understanding. Using fMRI, we measured brain activity as participants viewed a diverse set of 90 different video clips depicting social and nonsocial actions in real-world contexts. We developed five behavioral models using arrangement tasks: two models reflecting behavioral judgments of the purpose (transitivity) and the social content (sociality) of the actions depicted in the video stimuli; and three models reflecting behavioral judgments of the visual content (people, objects, and scene) depicted in still frames of the stimuli. We evaluated how well these models predict neural representational geometry and tested them against semantic models based on verb and nonverb embeddings and visual models based on gaze and motion energy. Our results revealed that behavioral judgments of similarity better reflect neural representational geometry than semantic or visual models throughout much of cortex. The sociality and transitivity models in particular captured a large portion of unique variance throughout the action observation network, extending into regions not typically associated with action perception, like ventral temporal cortex. Overall, our findings expand the action observation network and indicate that the social content and purpose of observed actions are predominant in cortical representation.","manuscriptTitle":"Behaviorally-relevant features of observed actions dominate cortical representational geometry in natural vision","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-12-03 02:28:24","doi":"10.21203/rs.3.rs-5478816/v1","editorialEvents":[],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"nature-communications","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"NCOMMS","sideBox":"Learn more about [Nature Communications](http://www.nature.com/ncomms/)","snPcode":"","submissionUrl":"https://mts-ncomms.nature.com/","title":"Nature Communications","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"ejp","reportingPortfolio":"Nature Communications","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"1f358665-74a7-4de1-a771-d01986e83c67","owner":[],"postedDate":"December 3rd, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[{"id":40935333,"name":"Biological sciences/Neuroscience/Cognitive neuroscience/Perception"},{"id":40935334,"name":"Biological sciences/Psychology/Human behaviour"}],"tags":[],"updatedAt":"2025-10-06T16:15:21+00:00","versionOfRecord":[],"versionCreatedAt":"2024-12-03 02:28:24","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5478816","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5478816","identity":"rs-5478816","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}