StereoMM: A Graph Fusion Model for Integrating Spatial Transcriptomic Data and Pathological Images

StereoMM: A Graph Fusion Model for Integrating Spatial Transcriptomic Data and Pathological Images | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article StereoMM: A Graph Fusion Model for Integrating Spatial Transcriptomic Data and Pathological Images Jiajun Zhang, Bingying Luo, Fei Teng, Guo Tang, Weixuan Cen, Chi Qu, and 13 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4616611/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted You are reading this latest preprint version Abstract Spatially resolved omics technologies generating multimodal and high-throughput data necessitate the development of advanced analysis methods, and facilitate biological discoveries by comprehensively utilizing information from the multi-omics data. Spatial transcriptomic data and hematoxylin and eosin (H&E) images reveal abundant features which are different and complementary to each other. We presented a machine learning based toolchain called StereoMM, a graph based fusion model that can integrate gene expression, histological images, and spatial location. StereoMM interacts with the information revealed by the transcriptomic and imaging data through an attention module. The updated features are input into the graph autoencoder together with a graph of spatial position, so that multimodal features are fused in a self-supervised manner. Here, StereoMM was trained using mouse brain tissue, demonstrating its capability to discern fine tissue architecture, while highlighting its advantage in computational speed. Utilizing data from human lung adenosquamous carcinoma obtained using Stereo-seq and human breast cancer from 10X Visium, we showed the superior performance of StereoMM in spatial domain recognition over competing software, and its ability to reveal tumour heterogeneity. We also used StereoMM to accurately classify patients with colorectal cancer data, effectively differentiating between patients with deficient mismatch repair (dMMR) and proficient mismatch repair (pMMR). StereoMM’s approach for analysing gene expression data and imaging data aids in accurate identification of spatial domains in spatial transcriptomes, unveils critical molecular features, and elucidates the connections between different domains, thereby laying the groundwork for downstream analysis. Biological sciences/Computational biology and bioinformatics/Software Biological sciences/Cancer/Cancer microenvironment Biological sciences/Molecular biology/Transcriptomics Biological sciences/Computational biology and bioinformatics spatial omics multimodal data deep learning graph fusion molecular characteristics Full Text Additional Declarations There is NO Competing Interest. Supplementary Files Supplymentaryinformation0621.docx Cite Share Download PDF Status: Under Review Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4616611","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":318440812,"identity":"42bc4661-2ce0-48c2-9255-3b22af7101ee","order_by":0,"name":"Jiajun Zhang","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABCklEQVRIiWNgGAWjYDACCRDBI8HA2MDAcOADWAjEJFIL48EZxGuBAObDPMS4S35287OHX2Qs5Jln5B44bPPLJo9f7HDjxx8Mdnm4tDDOOWZuLMMjYdg4Iy/hcG5fWrHk7MRmaR6G5GJcWpglEsykJXgkGBtn5Bgczu05nLjhdmIbMzAoEnH5iE0i/RtIiz1Yi2XPf7AWxh94tPBI5JhJfuCRSARrYfhxAKyFgQePFgmJnDJpoMbkxp43Bgd7G5ITZ4L9YpCMU4v8jPRtkj976mw3tucYf/jxxy6xXzr94ccfFXY4tYCDgLeHgcEQpIKxDSZmgEc9SOGPH0DrwMw/+FWOglEwCkbByAQA/+ZbM3/mYrAAAAAASUVORK5CYII=","orcid":"https://orcid.org/0000-0003-2731-7158","institution":"BGI Research","correspondingAuthor":true,"prefix":"","firstName":"Jiajun","middleName":"","lastName":"Zhang","suffix":""},{"id":318440813,"identity":"388e1815-bbc9-4b1c-81fc-7591f7907471","order_by":1,"name":"Bingying Luo","email":"","orcid":"","institution":"BGI Research","correspondingAuthor":false,"prefix":"","firstName":"Bingying","middleName":"","lastName":"Luo","suffix":""},{"id":318440814,"identity":"05871427-563c-4bd3-b4eb-0dbd9a5694d3","order_by":2,"name":"Fei Teng","email":"","orcid":"","institution":"BGI Research","correspondingAuthor":false,"prefix":"","firstName":"Fei","middleName":"","lastName":"Teng","suffix":""},{"id":318440815,"identity":"cd0ee5ea-e780-47ee-bb6b-89e77cef13de","order_by":3,"name":"Guo Tang","email":"","orcid":"","institution":"BGI Research","correspondingAuthor":false,"prefix":"","firstName":"Guo","middleName":"","lastName":"Tang","suffix":""},{"id":318440816,"identity":"b6f26f34-bcf5-4af0-9dfa-3917eda7a7ce","order_by":4,"name":"Weixuan Cen","email":"","orcid":"","institution":"BGI Research","correspondingAuthor":false,"prefix":"","firstName":"Weixuan","middleName":"","lastName":"Cen","suffix":""},{"id":318440817,"identity":"eeecf71c-db1f-4886-aaed-7ef1f9c513cf","order_by":5,"name":"Chi Qu","email":"","orcid":"","institution":"BGI Research","correspondingAuthor":false,"prefix":"","firstName":"Chi","middleName":"","lastName":"Qu","suffix":""},{"id":318440818,"identity":"d0d7ab3b-6ce0-41ff-bd3c-27fdfb68cb95","order_by":6,"name":"Xuanzhu Liu","email":"","orcid":"","institution":"Beijing Genomics Institute at Shenzhen","correspondingAuthor":false,"prefix":"","firstName":"Xuanzhu","middleName":"","lastName":"Liu","suffix":""},{"id":318440819,"identity":"69f15305-7fa2-43c5-889a-d410876b94c6","order_by":7,"name":"Xin Liu","email":"","orcid":"","institution":"BGI-Shenzhen, Shenzhen, China","correspondingAuthor":false,"prefix":"","firstName":"Xin","middleName":"","lastName":"Liu","suffix":""},{"id":318440820,"identity":"bf334521-2cdd-4aad-8b06-cd115e11f048","order_by":8,"name":"Xing Liu","email":"","orcid":"","institution":"BGI Research","correspondingAuthor":false,"prefix":"","firstName":"Xing","middleName":"","lastName":"Liu","suffix":""},{"id":318440821,"identity":"3461b7e9-938f-4214-ae9d-68d3b44bbbe6","order_by":9,"name":"Huaqiang Huang","email":"","orcid":"","institution":"BGI Research","correspondingAuthor":false,"prefix":"","firstName":"Huaqiang","middleName":"","lastName":"Huang","suffix":""},{"id":318440822,"identity":"4afe557b-3d45-4067-83ec-3f8054e380af","order_by":10,"name":"Yu Feng","email":"","orcid":"","institution":"BGI Research","correspondingAuthor":false,"prefix":"","firstName":"Yu","middleName":"","lastName":"Feng","suffix":""},{"id":318440823,"identity":"26d557af-c1f4-466b-8021-fe25895a3e1d","order_by":11,"name":"Xue Zhang","email":"","orcid":"","institution":"BGI Research","correspondingAuthor":false,"prefix":"","firstName":"Xue","middleName":"","lastName":"Zhang","suffix":""},{"id":318440824,"identity":"b58010aa-f881-496c-9f28-c8591f8d7476","order_by":12,"name":"Min Jian","email":"","orcid":"","institution":"BGI-shenzhen","correspondingAuthor":false,"prefix":"","firstName":"Min","middleName":"","lastName":"Jian","suffix":""},{"id":318440825,"identity":"87238c0b-5fcb-4050-b99c-70aef778bcd5","order_by":13,"name":"Mei Li","email":"","orcid":"","institution":"BGI Research","correspondingAuthor":false,"prefix":"","firstName":"Mei","middleName":"","lastName":"Li","suffix":""},{"id":318440826,"identity":"e3166b4e-515b-4eea-a40a-457c64c34124","order_by":14,"name":"Feng Xi","email":"","orcid":"","institution":"BGI Research","correspondingAuthor":false,"prefix":"","firstName":"Feng","middleName":"","lastName":"Xi","suffix":""},{"id":318440827,"identity":"01692d2b-8c41-44c8-adde-3caf60b593e3","order_by":15,"name":"Guibo Li","email":"","orcid":"https://orcid.org/0000-0002-6141-4931","institution":"BGI-Shenzhen","correspondingAuthor":false,"prefix":"","firstName":"Guibo","middleName":"","lastName":"Li","suffix":""},{"id":318440828,"identity":"6ac5d080-0f4c-451f-9b42-9335ad73cc32","order_by":16,"name":"Sha Liao","email":"","orcid":"","institution":"BGI-Shenzhen, Shenzhen 518083, China.","correspondingAuthor":false,"prefix":"","firstName":"Sha","middleName":"","lastName":"Liao","suffix":""},{"id":318440829,"identity":"082f4663-ddba-49f7-9d62-ed1ced3d1a3e","order_by":17,"name":"Ao Chen","email":"","orcid":"","institution":"BGI-ShenZhen","correspondingAuthor":false,"prefix":"","firstName":"Ao","middleName":"","lastName":"Chen","suffix":""},{"id":318440830,"identity":"dd099deb-8888-4c1c-94f3-991d8661e819","order_by":18,"name":"Xun Xu","email":"","orcid":"https://orcid.org/0000-0002-5338-5173","institution":"BGI-Research Shenzhen","correspondingAuthor":false,"prefix":"","firstName":"Xun","middleName":"","lastName":"Xu","suffix":""}],"badges":[],"createdAt":"2024-06-21 10:01:24","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4616611/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4616611/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":60055369,"identity":"cea2dcbd-fd67-4313-a4cc-8d13e6f70345","added_by":"auto","created_at":"2024-07-11 07:18:46","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1901660,"visible":true,"origin":"","legend":"","description":"","filename":"StereoMMmanuscript20240621doubleanonymized.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4616611/v1_covered_fc7ed08e-c196-4562-b7ac-809c88525d6e.pdf"},{"id":60054818,"identity":"62d52403-7773-446f-844f-b2658e8e6b51","added_by":"auto","created_at":"2024-07-11 07:10:45","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":4257598,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cbr\u003e\u003c/p\u003e","description":"","filename":"Supplymentaryinformation0621.docx","url":"https://assets-eu.researchsquare.com/files/rs-4616611/v1/4ac660089f85d6ae92fb71a3.docx"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"StereoMM: A Graph Fusion Model for Integrating Spatial Transcriptomic Data and Pathological Images","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"nature-portfolio","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"","title":"Nature Portfolio","twitterHandle":"","acdcEnabled":false,"dfaEnabled":false,"editorialSystem":"ejp","reportingPortfolio":"","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"spatial omics, multimodal data, deep learning, graph fusion, molecular characteristics","lastPublishedDoi":"10.21203/rs.3.rs-4616611/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4616611/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eSpatially resolved omics technologies generating multimodal and high-throughput data necessitate the development of advanced analysis methods, and facilitate biological discoveries by comprehensively utilizing information from the multi-omics data. Spatial transcriptomic data and hematoxylin and eosin (H\u0026amp;E) images reveal abundant features which are different and complementary to each other. We presented a machine learning based toolchain called StereoMM, a graph based fusion model that can integrate gene expression, histological images, and spatial location. StereoMM interacts with the information revealed by the transcriptomic and imaging data through an attention module. The updated features are input into the graph autoencoder together with a graph of spatial position, so that multimodal features are fused in a self-supervised manner. Here, StereoMM was trained using mouse brain tissue, demonstrating its capability to discern fine tissue architecture, while highlighting its advantage in computational speed. Utilizing data from human lung adenosquamous carcinoma obtained using Stereo-seq and human breast cancer from 10X Visium, we showed the superior performance of StereoMM in spatial domain recognition over competing software, and its ability to reveal tumour heterogeneity. We also used StereoMM to accurately classify patients with colorectal cancer data, effectively differentiating between patients with deficient mismatch repair (dMMR) and proficient mismatch repair (pMMR). StereoMM\u0026rsquo;s approach for analysing gene expression data and imaging data aids in accurate identification of spatial domains in spatial transcriptomes, unveils critical molecular features, and elucidates the connections between different domains, thereby laying the groundwork for downstream analysis.\u003c/p\u003e","manuscriptTitle":"StereoMM: A Graph Fusion Model for Integrating Spatial Transcriptomic Data and Pathological Images","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-07-11 07:10:40","doi":"10.21203/rs.3.rs-4616611/v1","editorialEvents":[],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"nature-communications","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"NCOMMS","sideBox":"Learn more about [Nature Communications](http://www.nature.com/ncomms/)","snPcode":"","submissionUrl":"https://mts-ncomms.nature.com/","title":"Nature Communications","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"ejp","reportingPortfolio":"Nature Communications","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"49c9780b-6d28-4844-ac29-e71462207db3","owner":[],"postedDate":"July 11th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[{"id":33671268,"name":"Biological sciences/Computational biology and bioinformatics/Software"},{"id":33671269,"name":"Biological sciences/Cancer/Cancer microenvironment"},{"id":33671270,"name":"Biological sciences/Molecular biology/Transcriptomics"},{"id":33671271,"name":"Biological sciences/Computational biology and bioinformatics"}],"tags":[],"updatedAt":"2024-07-11T07:10:40+00:00","versionOfRecord":[],"versionCreatedAt":"2024-07-11 07:10:40","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4616611","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4616611","identity":"rs-4616611","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}