spEMO: Leveraging Multi-Modal Foundation Models for Analyzing Spatial Multi-Omic and Histopathology Data

spEMO: Leveraging Multi-Modal Foundation Models for Analyzing Spatial Multi-Omic and Histopathology Data | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article spEMO: Leveraging Multi-Modal Foundation Models for Analyzing Spatial Multi-Omic and Histopathology Data Hongyu Zhao, Tianyu Liu, Tinglin Huang, Tong Ding, Hao Wu, Peter Humphrey, and 6 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6941589/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 05 Feb, 2026 Read the published version in Nature Biomedical Engineering → Version 1 posted You are reading this latest preprint version Abstract Recent advances in pathology foundation models (PFMs), which are pretrained on large-scale histopathological images, have significantly accelerated progress in disease-centered applications. In parallel, spatial multi-omic technologies collect gene and protein expression levels at high spatial resolution, offering rich understanding of tissue context. However, current models fall short in effectively integrating these complementary data modalities. To fill in this gap, we introduce spEMO, a novel computational system that unifies embeddings from pathology foundation models and large language models (LLMs) to analyze spatial multi-omic data. By incorporating multimodal representations and information, spEMO outperforms models trained on single-modality data across a broad range of downstream tasks, including spatial domain identification, spot-type classification, whole-slide disease-state prediction and interpretation, inference of multicellular interactions, and automated medical report generation. The outstanding performances of spEMO in these tasks demonstrate its strength in both biological and clinical applications. Additionally, we propose a new evaluation task, known as multi-modal alignment, to assess the information retrieval capabilities of pathology foundation models. This task provides a principled benchmark for evaluating and improving model architectures. Collectively, spEMO represents a step forward in building holistic, interpretable, and generalizable AI systems for spatial biology and pathology. Biological sciences/Computational biology and bioinformatics/Computational models Biological sciences/Computational biology and bioinformatics/Machine learning Spatial Multi-Omic Data Analysis Foundation Model Spatial Domain Identification Spot-Type Prediction Disease-Associated Prediction Cell-Cell Interaction Medical Report Generation Full Text Additional Declarations There is NO Competing Interest. Cite Share Download PDF Status: Published Journal Publication published 05 Feb, 2026 Read the published version in Nature Biomedical Engineering → Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6941589","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":481961671,"identity":"008119a8-1b78-4e8b-90ab-de2c9f7de5cd","order_by":0,"name":"Hongyu Zhao","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA5ElEQVRIiWNgGAWjYBACAyQG4wMGBgsUQYJamIFYgjQtbBJEaTFn7z38mreNwd6c/fCxap4aCTkG9uZtEvi0WPacS7MGamG27ElLu81zTMKYgedYGV4tBjdyzIyBWtgMDuSY3eZhk0hskMgxI0oLj8H5N2bFPP8k6hvk3xDUYvwYqEUCpJeZt00igUGCh4CWM2fMGOeckzAwuPEsWXJun4RhG09asQVeLcd7jD+8KbOxNziffPDDm2828vzshzfewKcFCNikeCDRwcDEA+ISUA4CzB9/QFmMP/AqHAWjYBSMgpEKAAR3QO4h1DXsAAAAAElFTkSuQmCC","orcid":"https://orcid.org/0000-0003-1195-9607","institution":"Yale University","correspondingAuthor":true,"prefix":"","firstName":"Hongyu","middleName":"","lastName":"Zhao","suffix":""},{"id":481961672,"identity":"4bbe1cbb-64da-40b5-af5d-492f05c9826f","order_by":1,"name":"Tianyu Liu","email":"","orcid":"https://orcid.org/0000-0002-9412-6573","institution":"Yale University","correspondingAuthor":false,"prefix":"","firstName":"Tianyu","middleName":"","lastName":"Liu","suffix":""},{"id":481961673,"identity":"321670aa-89d6-48c8-9229-fcae7f9edd15","order_by":2,"name":"Tinglin Huang","email":"","orcid":"","institution":"Yale University","correspondingAuthor":false,"prefix":"","firstName":"Tinglin","middleName":"","lastName":"Huang","suffix":""},{"id":481961674,"identity":"21425472-6b78-4fa0-8559-ecee980aeb84","order_by":3,"name":"Tong Ding","email":"","orcid":"","institution":"Harvard University","correspondingAuthor":false,"prefix":"","firstName":"Tong","middleName":"","lastName":"Ding","suffix":""},{"id":481961675,"identity":"8d5f4419-018e-414c-a696-26ab42bfa23e","order_by":4,"name":"Hao Wu","email":"","orcid":"","institution":"Yale University","correspondingAuthor":false,"prefix":"","firstName":"Hao","middleName":"","lastName":"Wu","suffix":""},{"id":481961676,"identity":"3fdd5df9-60f3-4893-a3fd-df0cc5ca254a","order_by":5,"name":"Peter Humphrey","email":"","orcid":"","institution":"Yale University","correspondingAuthor":false,"prefix":"","firstName":"Peter","middleName":"","lastName":"Humphrey","suffix":""},{"id":481961677,"identity":"95f27c90-cdba-494d-a144-458441d11fc0","order_by":6,"name":"Sudhir Perincheri","email":"","orcid":"","institution":"Yale University","correspondingAuthor":false,"prefix":"","firstName":"Sudhir","middleName":"","lastName":"Perincheri","suffix":""},{"id":481961678,"identity":"8c127c1f-2f7e-45ce-9541-39841332d372","order_by":7,"name":"Kurt Schalper","email":"","orcid":"","institution":"Yale University","correspondingAuthor":false,"prefix":"","firstName":"Kurt","middleName":"","lastName":"Schalper","suffix":""},{"id":481961679,"identity":"1eae96dd-d1cc-4923-b8aa-66c5c8cf36d4","order_by":8,"name":"Rex Ying","email":"","orcid":"","institution":"Yale University","correspondingAuthor":false,"prefix":"","firstName":"Rex","middleName":"","lastName":"Ying","suffix":""},{"id":481961680,"identity":"a14a3458-fedd-4c53-a5aa-b4781f1a6b50","order_by":9,"name":"Hua Xu","email":"","orcid":"","institution":"Yale University","correspondingAuthor":false,"prefix":"","firstName":"Hua","middleName":"","lastName":"Xu","suffix":""},{"id":481961681,"identity":"005bde21-86d0-4b45-9c92-8e4a003b64f0","order_by":10,"name":"James Zou","email":"","orcid":"https://orcid.org/0000-0001-8880-4764","institution":"Stanford University","correspondingAuthor":false,"prefix":"","firstName":"James","middleName":"","lastName":"Zou","suffix":""},{"id":481961682,"identity":"eeb36fdd-da07-4bff-8ea9-bc5e38129c27","order_by":11,"name":"Faisal Mahmood","email":"","orcid":"https://orcid.org/0000-0001-7587-1562","institution":"Harvard Medical School, USA","correspondingAuthor":false,"prefix":"","firstName":"Faisal","middleName":"","lastName":"Mahmood","suffix":""}],"badges":[],"createdAt":"2025-06-20 21:35:24","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6941589/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6941589/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1038/s41551-025-01602-6","type":"published","date":"2026-02-05T05:00:00+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":101994395,"identity":"06d9feea-0d7d-424a-8e08-043591f0b9fe","added_by":"auto","created_at":"2026-02-05 21:42:04","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":7801706,"visible":true,"origin":"","legend":"Article File","description":"","filename":"spemosubmit.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6941589/v1_covered_de25dd91-43bc-495f-9033-305ed81b16df.pdf"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"spEMO: Leveraging Multi-Modal Foundation Models for Analyzing Spatial Multi-Omic and Histopathology Data","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"nature-portfolio","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"","title":"Nature Portfolio","twitterHandle":"","acdcEnabled":false,"dfaEnabled":false,"editorialSystem":"ejp","reportingPortfolio":"","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Spatial Multi-Omic Data Analysis, Foundation Model, Spatial Domain Identification, Spot-Type Prediction, Disease-Associated Prediction, Cell-Cell Interaction, Medical Report Generation","lastPublishedDoi":"10.21203/rs.3.rs-6941589/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6941589/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Recent advances in pathology foundation models (PFMs), which are pretrained on large-scale histopathological images, have significantly accelerated progress in disease-centered applications. In parallel, spatial multi-omic technologies collect gene and protein expression levels at high spatial resolution, offering rich understanding of tissue context. However, current models fall short in effectively integrating these complementary data modalities. To fill in this gap, we introduce spEMO, a novel computational system that unifies embeddings from pathology foundation models and large language models (LLMs) to analyze spatial multi-omic data. By incorporating multimodal representations and information, spEMO outperforms models trained on single-modality data across a broad range of downstream tasks, including spatial domain identification, spot-type classification, whole-slide disease-state prediction and interpretation, inference of multicellular interactions, and automated medical report generation. The outstanding performances of spEMO in these tasks demonstrate its strength in both biological and clinical applications. Additionally, we propose a new evaluation task, known as multi-modal alignment, to assess the information retrieval capabilities of pathology foundation models. This task provides a principled benchmark for evaluating and improving model architectures. Collectively, spEMO represents a step forward in building holistic, interpretable, and generalizable AI systems for spatial biology and pathology.","manuscriptTitle":"spEMO: Leveraging Multi-Modal Foundation Models for Analyzing Spatial Multi-Omic and Histopathology Data","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-07-08 16:38:49","doi":"10.21203/rs.3.rs-6941589/v1","editorialEvents":[],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"nature-biomedical-engineering","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"natbiomedeng","sideBox":"Learn more about [Nature Biomedical Engineering](http://www.nature.com/natbiomedeng/)","snPcode":"41551","submissionUrl":"https://mts-natbiomedeng.nature.com/cgi-bin/main.plex","title":"Nature Biomedical Engineering","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"ejp","reportingPortfolio":"Nature Research","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"1c447e0a-270e-461a-a008-85902551d43c","owner":[],"postedDate":"July 8th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":51172498,"name":"Biological sciences/Computational biology and bioinformatics/Computational models"},{"id":51172499,"name":"Biological sciences/Computational biology and bioinformatics/Machine learning"}],"tags":[],"updatedAt":"2026-02-05T21:41:56+00:00","versionOfRecord":{"articleIdentity":"rs-6941589","link":"https://doi.org/10.1038/s41551-025-01602-6","journal":{"identity":"nature-biomedical-engineering","isVorOnly":false,"title":"Nature Biomedical Engineering"},"publishedOn":"2026-02-05 05:00:00","publishedOnDateReadable":"February 5th, 2026"},"versionCreatedAt":"2025-07-08 16:38:49","video":"","vorDoi":"10.1038/s41551-025-01602-6","vorDoiUrl":"https://doi.org/10.1038/s41551-025-01602-6","workflowStages":[]},"version":"v1","identity":"rs-6941589","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6941589","identity":"rs-6941589","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}