Harnessing the Power of Single Cell Large Language Models with Parameter Efficient Fine-Tuning using scPEFT

Harnessing the Power of Single Cell Large Language Models with Parameter Efficient Fine-Tuning using scPEFT | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Harnessing the Power of Single Cell Large Language Models with Parameter Efficient Fine-Tuning using scPEFT Dong Xu, Fei He, Ruixin Fei, Jordan Krull, Xinyu Zhang, Mingyue Gao, and 8 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5926885/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 31 Dec, 2025 Read the published version in Nature Machine Intelligence → Version 1 posted You are reading this latest preprint version Abstract Single-cell large language models (scLLMs) capture essential biological insights from vast single-cell atlases but struggle in out-of-context applications, where zero-shot predictions can be unreliable. To address this, we introduce a single-cell parameter-efficient fine-tuning (scPEFT) framework that integrates learnable, low-dimensional adapters into scLLMs. By freezing the backbone model and updating only the adapter parameters, scPEFT efficiently adapts to specific tasks using limited custom data. This approach mitigates catastrophic forgetting, reduces parameter tuning by over 96%, and decreases GPU memory usage by more than half, significantly enhancing scLLMs’s accessibility for resource-constrained researchers. Validated across diverse datasets, scPEFT outperformed zero-shot models and traditional fine-tuning in disease-specific, cross-species, and under-characterized cell population tasks. Its attention-mechanism analysis identified COVID-related genes associated with specific cell states and uncovered unique blood cell subpopulations, demonstrating scPEFT’s capacity for condition-specific interpretations. These findings position scPEFT as an efficient solution for improving scLLMs’ utilities in general single-cell analyses. Biological sciences/Computational biology and bioinformatics/Machine learning Biological sciences/Computational biology and bioinformatics/Software Full Text Additional Declarations There is NO Competing Interest. Supplementary Files finalscPEFTsupplementarytable.pdf Default hyper-parameter settings finalscPEFTsupplementaryf1.pdf Violin plots of diverse adapters in scPEFT across datasets finalscPEFTsupplementaryf2.pdf UMAP visualizations of cell type identification on the NSCLC dataset. finalscPEFTsupplementaryf3.pdf UMAP visualizations of cell type identification on the MS dataset. finalscPEFTsupplementaryf4.pdf UMAP visualizations of cell type identification on the COVID dataset. Cite Share Download PDF Status: Published Journal Publication published 31 Dec, 2025 Read the published version in Nature Machine Intelligence → Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5926885","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":420479707,"identity":"1fa4d437-7f3f-4539-8c37-cbf7a545c173","order_by":0,"name":"Dong Xu","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAzUlEQVRIiWNgGAWjYHACAyC2YWCDcCSAOIEoLWmkazmMLEBAi8GN5I2PC36dj+bjP/7wMe8eCwZ+9hwDvFokZ6QVG8/su53bJpFjbMzzTIJBsucNfi38Ejlm0rw9IC08bNI8BySA9hKwhQ2i5VxuG//xZ2At9oS0gG3h+XEgt40hwQxiiwQhv/Q8KzbmbUgG+8VwzgEJHokzzwrwajE4Dgwxnj92ufP7jz988OZAnRx/e/IGvFrAgLENweYhrBwM/hCpbhSMglEwCkYmAAChT0ADUukQqgAAAABJRU5ErkJggg==","orcid":"https://orcid.org/0000-0002-4809-0514","institution":"University of Missouri - Columbia","correspondingAuthor":true,"prefix":"","firstName":"Dong","middleName":"","lastName":"Xu","suffix":""},{"id":420479708,"identity":"1722e66e-0aec-4589-b8c4-d9b460738647","order_by":1,"name":"Fei He","email":"","orcid":"https://orcid.org/0000-0002-3284-9506","institution":"University of Missouri - Columbia","correspondingAuthor":false,"prefix":"","firstName":"Fei","middleName":"","lastName":"He","suffix":""},{"id":420479709,"identity":"0233752c-b1eb-414c-9314-ac00b9e56b98","order_by":2,"name":"Ruixin Fei","email":"","orcid":"","institution":"University of Missouri - Columbia","correspondingAuthor":false,"prefix":"","firstName":"Ruixin","middleName":"","lastName":"Fei","suffix":""},{"id":420479710,"identity":"8266de41-7745-4069-934e-4e34b64fa876","order_by":3,"name":"Jordan Krull","email":"","orcid":"https://orcid.org/0000-0001-6507-8085","institution":"The Ohio State University","correspondingAuthor":false,"prefix":"","firstName":"Jordan","middleName":"","lastName":"Krull","suffix":""},{"id":420479711,"identity":"5eb329ae-c570-42b8-af05-7df4acb0a2b7","order_by":4,"name":"Xinyu Zhang","email":"","orcid":"","institution":"University of Missouri-Columbia","correspondingAuthor":false,"prefix":"","firstName":"Xinyu","middleName":"","lastName":"Zhang","suffix":""},{"id":420479712,"identity":"bdf0598d-1ae0-4ad7-9b63-e5d8e942cd7f","order_by":5,"name":"Mingyue Gao","email":"","orcid":"","institution":"University of Missouri-Columbia","correspondingAuthor":false,"prefix":"","firstName":"Mingyue","middleName":"","lastName":"Gao","suffix":""},{"id":420479713,"identity":"f2375a44-dab4-4760-8ac7-54ec2fde0131","order_by":6,"name":"Li Su","email":"","orcid":"https://orcid.org/0000-0003-3735-4681","institution":"University of Missouri - Columbia","correspondingAuthor":false,"prefix":"","firstName":"Li","middleName":"","lastName":"Su","suffix":""},{"id":420479714,"identity":"56e1061c-b2ef-4915-9823-71f3bf66a9dc","order_by":7,"name":"Yibo Chen","email":"","orcid":"https://orcid.org/0000-0002-3969-2679","institution":"University of Missouri-Columbia","correspondingAuthor":false,"prefix":"","firstName":"Yibo","middleName":"","lastName":"Chen","suffix":""},{"id":420479715,"identity":"5f310968-fabb-453f-ac93-cd134c693ea6","order_by":8,"name":"Baichuan Jin","email":"","orcid":"","institution":"University of Missouri-Columbia","correspondingAuthor":false,"prefix":"","firstName":"Baichuan","middleName":"","lastName":"Jin","suffix":""},{"id":420479716,"identity":"2c5c9ded-b4f7-4d61-9509-8dc6c83d6bc2","order_by":9,"name":"Yang Yu","email":"","orcid":"https://orcid.org/0009-0006-1475-7487","institution":"university of missouri","correspondingAuthor":false,"prefix":"","firstName":"Yang","middleName":"","lastName":"Yu","suffix":""},{"id":420479717,"identity":"707021d6-5dbc-41a9-a240-61dda449ed3b","order_by":10,"name":"Yuzhou Chang","email":"","orcid":"","institution":"The Ohio State University","correspondingAuthor":false,"prefix":"","firstName":"Yuzhou","middleName":"","lastName":"Chang","suffix":""},{"id":420479718,"identity":"0a096082-4db1-4408-80a4-9369239eb860","order_by":11,"name":"Qin Ma","email":"","orcid":"https://orcid.org/0000-0002-3264-8392","institution":"The Ohio State University","correspondingAuthor":false,"prefix":"","firstName":"Qin","middleName":"","lastName":"Ma","suffix":""},{"id":420479719,"identity":"5d8e0c40-d1d5-4158-b208-9edfdf326749","order_by":12,"name":"Anjun Ma","email":"","orcid":"https://orcid.org/0000-0001-6269-398X","institution":"Ohio State University","correspondingAuthor":false,"prefix":"","firstName":"Anjun","middleName":"","lastName":"Ma","suffix":""},{"id":420479721,"identity":"ffb369fe-483d-42ff-ba9a-5b0b942b4039","order_by":13,"name":"Jinpu Li","email":"","orcid":"https://orcid.org/0000-0002-6656-2896","institution":"University of Missouri","correspondingAuthor":false,"prefix":"","firstName":"Jinpu","middleName":"","lastName":"Li","suffix":""}],"badges":[],"createdAt":"2025-01-29 23:15:32","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5926885/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5926885/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1038/s42256-025-01170-z","type":"published","date":"2025-12-31T05:00:00+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":99343806,"identity":"afbdb9bf-fb0e-4aca-b9b3-596c41d7cf05","added_by":"auto","created_at":"2026-01-01 08:05:40","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":4770369,"visible":true,"origin":"","legend":"","description":"","filename":"finalscPEFTmanuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5926885/v1_covered_2cd88293-9e0a-4bda-81e9-1255b2b8a713.pdf"},{"id":81344282,"identity":"4fc8c01d-ba08-45d7-9292-1b12ddd0864b","added_by":"auto","created_at":"2025-04-25 04:16:43","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":19166,"visible":true,"origin":"","legend":"Default hyper-parameter settings","description":"","filename":"finalscPEFTsupplementarytable.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5926885/v1/a57278ce761e9babea86390a.pdf"},{"id":81344283,"identity":"85ca3552-bceb-4f59-a918-4954c2c27b3a","added_by":"auto","created_at":"2025-04-25 04:16:44","extension":"pdf","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":449030,"visible":true,"origin":"","legend":"Violin plots of diverse adapters in scPEFT across datasets","description":"","filename":"finalscPEFTsupplementaryf1.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5926885/v1/e35651c82c9c3d42ff2d5db7.pdf"},{"id":81344286,"identity":"14909fab-4de2-43b6-8014-05f8905424ad","added_by":"auto","created_at":"2025-04-25 04:16:44","extension":"pdf","order_by":3,"title":"","display":"","copyAsset":false,"role":"supplement","size":3651012,"visible":true,"origin":"","legend":"\u003cp\u003eUMAP visualizations of cell type identification on the NSCLC dataset.\u003c/p\u003e","description":"","filename":"finalscPEFTsupplementaryf2.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5926885/v1/56543271590414e8a2c8aba2.pdf"},{"id":81344285,"identity":"b8d6ded8-3e67-42b9-a45e-992326778260","added_by":"auto","created_at":"2025-04-25 04:16:44","extension":"pdf","order_by":4,"title":"","display":"","copyAsset":false,"role":"supplement","size":2460173,"visible":true,"origin":"","legend":"\u003cp\u003eUMAP visualizations of cell type identification on the MS dataset.\u003c/p\u003e","description":"","filename":"finalscPEFTsupplementaryf3.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5926885/v1/91de744172c39bf75be90d0e.pdf"},{"id":81344293,"identity":"aa7094fa-4cbe-417c-88a9-558dd6983dc5","added_by":"auto","created_at":"2025-04-25 04:16:52","extension":"pdf","order_by":5,"title":"","display":"","copyAsset":false,"role":"supplement","size":2325164,"visible":true,"origin":"","legend":"\u003cp\u003eUMAP visualizations of cell type identification on the COVID dataset.\u003c/p\u003e","description":"","filename":"finalscPEFTsupplementaryf4.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5926885/v1/abd987b943228af4ae3a2f7c.pdf"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"Harnessing the Power of Single Cell Large Language Models with Parameter Efficient Fine-Tuning using scPEFT","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"nature-portfolio","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"","title":"Nature Portfolio","twitterHandle":"","acdcEnabled":false,"dfaEnabled":false,"editorialSystem":"ejp","reportingPortfolio":"","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-5926885/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5926885/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Single-cell large language models (scLLMs) capture essential biological insights from vast single-cell atlases but struggle in out-of-context applications, where zero-shot predictions can be unreliable. To address this, we introduce a single-cell parameter-efficient fine-tuning (scPEFT) framework that integrates learnable, low-dimensional adapters into scLLMs. By freezing the backbone model and updating only the adapter parameters, scPEFT efficiently adapts to specific tasks using limited custom data. This approach mitigates catastrophic forgetting, reduces parameter tuning by over 96%, and decreases GPU memory usage by more than half, significantly enhancing scLLMs’s accessibility for resource-constrained researchers. Validated across diverse datasets, scPEFT outperformed zero-shot models and traditional fine-tuning in disease-specific, cross-species, and under-characterized cell population tasks. Its attention-mechanism analysis identified COVID-related genes associated with specific cell states and uncovered unique blood cell subpopulations, demonstrating scPEFT’s capacity for condition-specific interpretations. These findings position scPEFT as an efficient solution for improving scLLMs’ utilities in general single-cell analyses.","manuscriptTitle":"Harnessing the Power of Single Cell Large Language Models with Parameter Efficient Fine-Tuning using scPEFT","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-04-25 04:16:39","doi":"10.21203/rs.3.rs-5926885/v1","editorialEvents":[],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"nature-machine-intelligence","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"natmachintell","sideBox":"Learn more about [Nature Machine Intelligence](http://www.nature.com/natmachintell/)","snPcode":"","submissionUrl":"","title":"Nature Machine Intelligence","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"ejp","reportingPortfolio":"Nature Research","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"edb73e9c-3b49-4129-a51f-f2720b625afb","owner":[],"postedDate":"April 25th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":44822077,"name":"Biological sciences/Computational biology and bioinformatics/Machine learning"},{"id":44822078,"name":"Biological sciences/Computational biology and bioinformatics/Software"}],"tags":[],"updatedAt":"2026-01-01T08:05:34+00:00","versionOfRecord":{"articleIdentity":"rs-5926885","link":"https://doi.org/10.1038/s42256-025-01170-z","journal":{"identity":"nature-machine-intelligence","isVorOnly":false,"title":"Nature Machine Intelligence"},"publishedOn":"2025-12-31 05:00:00","publishedOnDateReadable":"December 31st, 2025"},"versionCreatedAt":"2025-04-25 04:16:39","video":"","vorDoi":"10.1038/s42256-025-01170-z","vorDoiUrl":"https://doi.org/10.1038/s42256-025-01170-z","workflowStages":[]},"version":"v1","identity":"rs-5926885","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5926885","identity":"rs-5926885","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}