scELMo: Embeddings from Language Models are Good Learners for Single-cell Data Analysis

scELMo: Embeddings from Language Models are Good Learners for Single-cell Data Analysis | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article scELMo: Embeddings from Language Models are Good Learners for Single-cell Data Analysis Hongyu Zhao, Tianyu Liu, Tianqi Chen, Wangjie Zheng, Xiao Luo This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4014784/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Various Foundation Models (FMs) have been built based on the pre-training and fine-tuning framework to analyze single-cell data with different degrees of success. In this manuscript, we propose a method named scELMo (Single-cell Embedding from Language Models), to analyze single cell data that utilizes Large Language Models (LLMs) as a generator for both the description of metadata information and the embeddings for such descriptions. We combine the embeddings from LLMs with the raw data under the zero-shot learning framework to further extend its function by using the fine-tuning framework to handle different tasks. We demonstrate that scELMo is capable of cell clustering, batch effect correction, and cell-type annotation without training a new model. Moreover, the fine-tuning framework of scELMo can help with more challenging tasks including in-silico treatment analysis or modeling perturbation. scELMo has a lighter structure and lower requirement for resources. Moreover, our method is comparable to recent large-scale FMs (such as scGPT, [1] Geneformer [2]) based on our evaluations, suggesting a promising path for developing domain-specific FMs. Biological sciences/Computational biology and bioinformatics/Computational models Biological sciences/Computational biology and bioinformatics/Machine learning Single-Cell Data Analysis Foundation Model Large Language Model In-silico Treatment Analysis Perturbation Analysis Full Text Additional Declarations There is NO Competing Interest. Supplementary Files scELMosupplementary.pdf Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4014784","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":283877151,"identity":"e58f8323-cac8-4083-808f-94ba1dca72a2","order_by":0,"name":"Hongyu Zhao","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA5ElEQVRIiWNgGAWjYBACAyQG4wMGBgsUQYJamIFYgjQtbBJEaTFn7z38mreNwd6c/fCxap4aCTkG9uZtEvi0WPacS7MGamG27ElLu81zTMKYgedYGV4tBjdyzIyBWtgMDuSY3eZhk0hskMgxI0oLj8H5N2bFPP8k6hvk3xDUYvwYqEUCpJeZt00igUGCh4CWM2fMGOeckzAwuPEsWXJun4RhG09asQVeLcd7jD+8KbOxNziffPDDm2828vzshzfewKcFCNikeCDRwcDEA+ISUA4CzB9/QFmMP/AqHAWjYBSMgpEKAAR3QO4h1DXsAAAAAElFTkSuQmCC","orcid":"https://orcid.org/0000-0003-1195-9607","institution":"Yale University","correspondingAuthor":true,"prefix":"","firstName":"Hongyu","middleName":"","lastName":"Zhao","suffix":""},{"id":283877152,"identity":"d57e9abe-618b-460d-bca0-7b81228851b9","order_by":1,"name":"Tianyu Liu","email":"","orcid":"","institution":"Yale University","correspondingAuthor":false,"prefix":"","firstName":"Tianyu","middleName":"","lastName":"Liu","suffix":""},{"id":283877153,"identity":"a641ccb2-23b9-49ad-b25b-e6d6a3ba4902","order_by":2,"name":"Tianqi Chen","email":"","orcid":"","institution":"Yale University","correspondingAuthor":false,"prefix":"","firstName":"Tianqi","middleName":"","lastName":"Chen","suffix":""},{"id":283877154,"identity":"75db938a-c40b-4f5a-a7c0-fb0efdd15378","order_by":3,"name":"Wangjie Zheng","email":"","orcid":"","institution":"Yale University","correspondingAuthor":false,"prefix":"","firstName":"Wangjie","middleName":"","lastName":"Zheng","suffix":""},{"id":283877155,"identity":"dce84c6e-c835-4185-8282-4ccc99b3398a","order_by":4,"name":"Xiao Luo","email":"","orcid":"","institution":"University of California, Los Angeles","correspondingAuthor":false,"prefix":"","firstName":"Xiao","middleName":"","lastName":"Luo","suffix":""}],"badges":[],"createdAt":"2024-03-05 01:40:38","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4014784/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4014784/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":67744835,"identity":"3b74d6b0-d503-4698-84e4-f70a901a7f13","added_by":"auto","created_at":"2024-10-29 09:28:36","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1841997,"visible":true,"origin":"","legend":"","description":"","filename":"scELMomanuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4014784/v1_covered_97d804ed-84d5-44a1-954f-4e244c29ddc5.pdf"},{"id":53514862,"identity":"a8f7b85e-df48-48a1-8099-65194d849afd","added_by":"auto","created_at":"2024-03-27 01:19:54","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":19491245,"visible":true,"origin":"","legend":"","description":"","filename":"scELMosupplementary.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4014784/v1/1b70173026328420fe00d271.pdf"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"scELMo: Embeddings from Language Models are Good Learners for Single-cell Data Analysis","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Single-Cell Data Analysis, Foundation Model, Large Language Model, In-silico Treatment Analysis, Perturbation Analysis","lastPublishedDoi":"10.21203/rs.3.rs-4014784/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4014784/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eVarious Foundation Models (FMs) have been built based on the pre-training and fine-tuning framework to analyze single-cell data with different degrees of success. In this manuscript, we propose a method named scELMo (Single-cell Embedding from Language Models), to analyze single cell data that utilizes Large Language Models (LLMs) as a generator for both the description of metadata information and the embeddings for such descriptions. We combine the embeddings from LLMs with the raw data under the zero-shot learning framework to further extend its function by using the fine-tuning framework to handle different tasks. We demonstrate that scELMo is capable of cell clustering, batch effect correction, and cell-type annotation without training a new model. Moreover, the fine-tuning framework of scELMo can help with more challenging tasks including in-silico treatment analysis or modeling perturbation. scELMo has a lighter structure and lower requirement for resources. Moreover, our method is comparable to recent large-scale FMs (such as scGPT, [1] Geneformer [2]) based on our evaluations, suggesting a promising path for developing domain-specific FMs.\u003c/p\u003e","manuscriptTitle":"scELMo: Embeddings from Language Models are Good Learners for Single-cell Data Analysis","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-03-27 01:19:47","doi":"10.21203/rs.3.rs-4014784/v1","editorialEvents":[],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"1c447e0a-270e-461a-a008-85902551d43c","owner":[],"postedDate":"March 27th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":29881191,"name":"Biological sciences/Computational biology and bioinformatics/Computational models"},{"id":29881192,"name":"Biological sciences/Computational biology and bioinformatics/Machine learning"}],"tags":[],"updatedAt":"2024-11-18T13:05:17+00:00","versionOfRecord":[],"versionCreatedAt":"2024-03-27 01:19:47","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4014784","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4014784","identity":"rs-4014784","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}