Generative learning with multimodal prompts as computational model for brain responses

Generative learning with multimodal prompts as computational model for brain responses | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Generative learning with multimodal prompts as computational model for brain responses Ya-Li Li, Xin Liu, Jichuan Zhang, Shengjin Wang This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8322486/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Exploring brain activity via high-resolution functional magnetic resonance imaging (fMRI) is important in the field of neuroscience. Traditional methodology leverages statistical analysis to associate the stimulus and brain responses, as well as identify cortical selectivity, restricted by insufficient data-driven learning. Here we address the brain activity analysis from the perspective of deep generative learning. We develop a diffusion model, NeoDiffuser, for generating and recovering fMRI data of visual cortex areas with controllable conditions. Technically, NeoDiffuser is composed of multimodal encoders, and a conditional diffusion model accommodating multimodal prompts. We demonstrate the capabilities of NeoDiffuser in simulating fMRI responses with compact guidance encoded from visual stimuli and contextual layouts, to perform stimulus-to-cortex functional association. NeoDiffuser also exhibits the ability in recovering brain signals of missing vertices, to further analyze cortex-to-cortex association. Owing to the controllable prompts, NeoDiffuser shows the feasibility of exploratory factor analysis on what impacts the neural responses. We explore the impacts of geometrical features and hemisphere-specific properties. We demonstrate that the areal cortex associations revealed by fMRI generation and recovery show high consistency with streamed visual processing. The development of NeoDiffuser displays the potential of bridging the human cognitive process and artificial neural networks. Biological sciences/Computational biology and bioinformatics/Computational models Physical sciences/Engineering/Biomedical engineering Full Text Additional Declarations There is NO Competing Interest. Supplementary Files supp.pdf Supplementary for "Generative learning with multimodal prompts as computational model for brain responses" Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8322486","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":561424286,"identity":"7d24b2a5-b508-491b-af76-b63b18f214c6","order_by":0,"name":"Ya-Li Li","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABCklEQVRIiWNgGAWjYBACPmYg8cBAwo4fSB8AIyCQwKeFDaQlwcAiWbKNaC0gIoGhgnHDMTCfGC3sPIYfEgokmI3vtz88XPDrTmJ/A/PB2zwMdnm4HcZjLJFgIMFndozH4PDMvmeJMw6wJVvzMCQX49FiANLCDNTCcJi353BiwwEeM2kehgOJDXhs+QHUwri5jf0BWMv8A/zfCGkxA9nCuIGNweAwz4/DiRsO8LAR0MJWZgHUkixxLMfgMG/DYeONh9mMLecYJOPUws9/ePOND3/q7Pibjz/+zPPnsOy8480Pb7ypsMOphYGBwwDBZgTGJwMochkMcKgGA/YHSJw/+FSOglEwCkbBSAUAZ5NU2y3Z4GwAAAAASUVORK5CYII=","orcid":"","institution":"Tsinghua University","correspondingAuthor":true,"prefix":"","firstName":"Ya-Li","middleName":"","lastName":"Li","suffix":""},{"id":561424287,"identity":"a8bb06c7-80ab-44bf-bad2-3aa8e67e9699","order_by":1,"name":"Xin Liu","email":"","orcid":"","institution":"Tsinghua University","correspondingAuthor":false,"prefix":"","firstName":"Xin","middleName":"","lastName":"Liu","suffix":""},{"id":561424288,"identity":"9e2993eb-641a-486e-84e1-fe23b4b1280e","order_by":2,"name":"Jichuan Zhang","email":"","orcid":"","institution":"Tsinghua University","correspondingAuthor":false,"prefix":"","firstName":"Jichuan","middleName":"","lastName":"Zhang","suffix":""},{"id":561424289,"identity":"4011d484-660c-44eb-b50d-2138670ad1ba","order_by":3,"name":"Shengjin Wang","email":"","orcid":"","institution":"Tsinghua University","correspondingAuthor":false,"prefix":"","firstName":"Shengjin","middleName":"","lastName":"Wang","suffix":""}],"badges":[],"createdAt":"2025-12-10 03:15:15","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8322486/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8322486/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":98477778,"identity":"55787fb1-3698-4cfa-bb43-be289784e789","added_by":"auto","created_at":"2025-12-18 04:22:41","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":42494194,"visible":true,"origin":"","legend":"","description":"","filename":"main2.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8322486/v1/13d0b4749995ead8dfeef89f.pdf"},{"id":98477777,"identity":"b87ffe69-0ad6-4bb4-8b35-b9aa5ef5bfe7","added_by":"auto","created_at":"2025-12-18 04:22:40","extension":"json","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":5760,"visible":true,"origin":"","legend":"","description":"","filename":"NCOMMS25099956.json","url":"https://assets-eu.researchsquare.com/files/rs-8322486/v1/152fa707821a1658b19f640f.json"},{"id":98477780,"identity":"48de1c3b-4695-43a9-b21f-db7be4ad0c4b","added_by":"auto","created_at":"2025-12-18 04:22:41","extension":"pdf","order_by":2,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":46158924,"visible":true,"origin":"","legend":"","description":"","filename":"supp.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8322486/v1/51d0f1d83a924582d1492a55.pdf"},{"id":104402171,"identity":"35523eba-5950-4e49-87f7-cb61adcf53a8","added_by":"auto","created_at":"2026-03-11 12:14:33","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":28853636,"visible":true,"origin":"","legend":"Article File","description":"","filename":"main2.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8322486/v1_covered_fece56f4-bd25-4b87-8f76-796432f05353.pdf"},{"id":98477779,"identity":"aed22955-0122-449f-9915-b9e7138974a7","added_by":"auto","created_at":"2025-12-18 04:22:41","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":46158924,"visible":true,"origin":"","legend":"Supplementary for \"Generative learning with multimodal prompts as computational model for brain responses\"","description":"","filename":"supp.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8322486/v1/ad40d3fe6f2ed74edeff7806.pdf"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"Generative learning with multimodal prompts as computational model for brain responses","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-8322486/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8322486/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Exploring brain activity via high-resolution functional magnetic resonance imaging (fMRI) is important in the field of neuroscience. Traditional methodology leverages statistical analysis to associate the stimulus and brain responses, as well as identify cortical selectivity, restricted by insufficient data-driven learning. Here we address the brain activity analysis from the perspective of deep generative learning. We develop a diffusion model, NeoDiffuser, for generating and recovering fMRI data of visual cortex areas with controllable conditions. Technically, NeoDiffuser is composed of multimodal encoders, and a conditional diffusion model accommodating multimodal prompts. We demonstrate the capabilities of NeoDiffuser in simulating fMRI responses with compact guidance encoded from visual stimuli and contextual layouts, to perform stimulus-to-cortex functional association. NeoDiffuser also exhibits the ability in recovering brain signals of missing vertices, to further analyze cortex-to-cortex association. Owing to the controllable prompts, NeoDiffuser shows the feasibility of exploratory factor analysis on what impacts the neural responses. We explore the impacts of geometrical features and hemisphere-specific properties. We demonstrate that the areal cortex associations revealed by fMRI generation and recovery show high consistency with streamed visual processing. The development of NeoDiffuser displays the potential of bridging the human cognitive process and artificial neural networks.","manuscriptTitle":"Generative learning with multimodal prompts as computational model for brain responses","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-12-18 04:22:35","doi":"10.21203/rs.3.rs-8322486/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"55eb1ce5-84d7-43a7-824a-e5a65cab6062","owner":[],"postedDate":"December 18th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":59779546,"name":"Biological sciences/Computational biology and bioinformatics/Computational models"},{"id":59779547,"name":"Physical sciences/Engineering/Biomedical engineering"}],"tags":[],"updatedAt":"2026-03-04T20:30:44+00:00","versionOfRecord":[],"versionCreatedAt":"2025-12-18 04:22:35","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8322486","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8322486","identity":"rs-8322486","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}