DenoiseST: A dual-channel unsupervised deep learning-based denoising method to identify spatial domains and functionally variable genes in spatial transcriptomics

DenoiseST: A dual-channel unsupervised deep learning-based denoising method to identify spatial domains and functionally variable genes in spatial transcriptomics | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Short Report DenoiseST: A dual-channel unsupervised deep learning-based denoising method to identify spatial domains and functionally variable genes in spatial transcriptomics Yaxuan Cui, Ruheng Wang, Xin Zeng, Yang Cui, Zheyong Zhu, Kenta Nakai, and 3 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4470472/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Spatial transcriptomics provides a unique opportunity for understanding cellular organization and function in a spatial context. However, spatial transcriptome exists the problem of dropout noise, exposing a major challenge for accurate downstream data analysis. Here, we proposed DenoiseST, a dual-channel unsupervised adaptive deep learning-based denoising method for data imputing, clustering, and identifying functionally variable genes in spatial transcriptomics. To leverage spatial information and gene expression profiles, we proposed a dual-channel joint learning strategy with graph convolutional networks to sufficiently explore both linear and nonlinear representation embeddings in an unsupervised manner, enhancing the discriminative information learning ability from the global perspectives of data distributions. In particular, DenoiseST enables the adaptively fitting of different gene distributions to the clustered domains and employs tissue-level spatial information to accurately identify functionally variable genes with different spatial resolutions, revealing their enrichment in corresponding gene pathways. Extensive validations on a total of 19 real spatial transcriptome datasets show that DenoiseST obtains excellent performance and results on brain tissue datasets indicate it outperforms the state-of-the-art methods when handling artificial dropout noise with a remarkable margin of ~ 15%, demonstrating its effectiveness and robustness. Case study results demonstrate that when applied to identify biological structural regions on human breast cancer spatial transcriptomic datasets, DenoiseST successfully detected biologically significant immune-related structural regions, which are subsequently validated through Gene Ontology (GO), cell-cell communication, and survival analysis. In conclusion, we expect that DenoiseST is a novel and efficient method for spatial transcriptome analysis, offering unique insights into spatial organization and function. spatial transcriptome dropout noise dual-channel joint learning functionally variable genes immune-related structural regions Full Text Additional Declarations No competing interests reported. Supplementary Files SupplementDenoiseST.docx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4470472","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Short Report","associatedPublications":[],"authors":[{"id":309443883,"identity":"b7167e5c-4989-4cea-b666-237b7b8521a7","order_by":0,"name":"Yaxuan Cui","email":"","orcid":"","institution":"University of Tsukuba","correspondingAuthor":false,"prefix":"","firstName":"Yaxuan","middleName":"","lastName":"Cui","suffix":""},{"id":309443884,"identity":"4e0f8dd0-8c6e-4934-9e66-8975ce425591","order_by":1,"name":"Ruheng Wang","email":"","orcid":"","institution":"University of Texas Southwestern Medical Center","correspondingAuthor":false,"prefix":"","firstName":"Ruheng","middleName":"","lastName":"Wang","suffix":""},{"id":309443885,"identity":"3ce48b04-6a91-4aad-815b-c9f32ae1cda1","order_by":2,"name":"Xin Zeng","email":"","orcid":"","institution":"The University of Tokyo","correspondingAuthor":false,"prefix":"","firstName":"Xin","middleName":"","lastName":"Zeng","suffix":""},{"id":309443886,"identity":"828debe6-72dd-4119-9037-85e5c517a1a7","order_by":3,"name":"Yang Cui","email":"","orcid":"","institution":"The University of Tokyo","correspondingAuthor":false,"prefix":"","firstName":"Yang","middleName":"","lastName":"Cui","suffix":""},{"id":309443887,"identity":"998153ec-52ea-484d-8ae7-f0c2dd9ff750","order_by":4,"name":"Zheyong Zhu","email":"","orcid":"","institution":"Shandong University","correspondingAuthor":false,"prefix":"","firstName":"Zheyong","middleName":"","lastName":"Zhu","suffix":""},{"id":309443888,"identity":"5d442c96-9af6-43bd-b7a1-7221e2d2dc3c","order_by":5,"name":"Kenta Nakai","email":"","orcid":"","institution":"The University of Tokyo","correspondingAuthor":false,"prefix":"","firstName":"Kenta","middleName":"","lastName":"Nakai","suffix":""},{"id":309443889,"identity":"8bc56f45-fabd-486b-a7e6-1057cd08aa62","order_by":6,"name":"Xiucai Ye","email":"","orcid":"","institution":"University of Tsukuba","correspondingAuthor":false,"prefix":"","firstName":"Xiucai","middleName":"","lastName":"Ye","suffix":""},{"id":309443890,"identity":"651f67eb-754a-4e93-bcdc-3eb6dca501e1","order_by":7,"name":"Tetsuya Sakurai","email":"","orcid":"","institution":"University of Tsukuba","correspondingAuthor":false,"prefix":"","firstName":"Tetsuya","middleName":"","lastName":"Sakurai","suffix":""},{"id":309443891,"identity":"a52c1228-c5d8-4ba4-9b3d-40c0e385dc27","order_by":8,"name":"Leyi Wei","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA10lEQVRIiWNgGAWjYFACxsYDDAwScgwMCUAOG3FaGkBajEnRwsAA1MKQ2EC0FoPjzQ0HPrZZpM9vzzFg+FB2mIF/dgMBLWcONhyc2SaRu+HMGwPGGecOM0jcOYBfi9mNxIbDvCAtEjkGzLxthxkMJBIIaLn/sOHw3zaJdPkZQC1/idJyg7HhMGMbUNkNoBZGYrTYn0lsONhzTsJww5lnBUBGOo/EDQJaJNuPP3zwo6xOXr49eSOQYS3HP4OAFjBghEbHASDmIUI9CPwhUt0oGAWjYBSMTAAArmNJp+ZBVmEAAAAASUVORK5CYII=","orcid":"","institution":"Shandong University","correspondingAuthor":true,"prefix":"","firstName":"Leyi","middleName":"","lastName":"Wei","suffix":""}],"badges":[],"createdAt":"2024-05-24 06:44:18","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4470472/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4470472/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":79268042,"identity":"cc3402ec-7df0-4729-a26b-18b96023271d","added_by":"auto","created_at":"2025-03-26 10:31:57","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2229588,"visible":true,"origin":"","legend":"","description":"","filename":"manuscriptdenoisest.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4470472/v1_covered_4137ecd8-3ac0-490f-b41e-ddebe54d5c01.pdf"},{"id":57844787,"identity":"9c82c430-83a4-4cbd-9f1e-47dc8d2c3cad","added_by":"auto","created_at":"2024-06-06 10:32:56","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":15353797,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementDenoiseST.docx","url":"https://assets-eu.researchsquare.com/files/rs-4470472/v1/bd3089799ce4c1a2fad06689.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"DenoiseST: A dual-channel unsupervised deep learning-based denoising method to identify spatial domains and functionally variable genes in spatial transcriptomics","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"spatial transcriptome, dropout noise, dual-channel joint learning, functionally variable genes, immune-related structural regions","lastPublishedDoi":"10.21203/rs.3.rs-4470472/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4470472/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eSpatial transcriptomics provides a unique opportunity for understanding cellular organization and function in a spatial context. However, spatial transcriptome exists the problem of dropout noise, exposing a major challenge for accurate downstream data analysis. Here, we proposed DenoiseST, a dual-channel unsupervised adaptive deep learning-based denoising method for data imputing, clustering, and identifying functionally variable genes in spatial transcriptomics. To leverage spatial information and gene expression profiles, we proposed a dual-channel joint learning strategy with graph convolutional networks to sufficiently explore both linear and nonlinear representation embeddings in an unsupervised manner, enhancing the discriminative information learning ability from the global perspectives of data distributions. In particular, DenoiseST enables the adaptively fitting of different gene distributions to the clustered domains and employs tissue-level spatial information to accurately identify functionally variable genes with different spatial resolutions, revealing their enrichment in corresponding gene pathways. Extensive validations on a total of 19 real spatial transcriptome datasets show that DenoiseST obtains excellent performance and results on brain tissue datasets indicate it outperforms the state-of-the-art methods when handling artificial dropout noise with a remarkable margin of ~\u0026thinsp;15%, demonstrating its effectiveness and robustness. Case study results demonstrate that when applied to identify biological structural regions on human breast cancer spatial transcriptomic datasets, DenoiseST successfully detected biologically significant immune-related structural regions, which are subsequently validated through Gene Ontology (GO), cell-cell communication, and survival analysis. In conclusion, we expect that DenoiseST is a novel and efficient method for spatial transcriptome analysis, offering unique insights into spatial organization and function.\u003c/p\u003e","manuscriptTitle":"DenoiseST: A dual-channel unsupervised deep learning-based denoising method to identify spatial domains and functionally variable genes in spatial transcriptomics","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-06-06 10:32:49","doi":"10.21203/rs.3.rs-4470472/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"cf0dc9f6-3fef-437d-a1b9-8afc85be0a73","owner":[],"postedDate":"June 6th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-03-26T10:23:46+00:00","versionOfRecord":[],"versionCreatedAt":"2024-06-06 10:32:49","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4470472","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4470472","identity":"rs-4470472","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}