Sparse autoencoders reveal organized biological knowledge but minimal regulatory logic in single-cell foundation models: a comparative atlas of Geneformer and scGPT

Sparse autoencoders reveal organized biological knowledge but minimal regulatory logic in single-cell foundation models: a comparative atlas of Geneformer and scGPT | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Sparse autoencoders reveal organized biological knowledge but minimal regulatory logic in single-cell foundation models: a comparative atlas of Geneformer and scGPT Ihor Kendiukhov This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9082479/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Background: Single-cell foundation models such as Geneformer and scGPT encode rich biological information, but whether this includes causal regulatory logic rather than statistical co-expression remains unclear. Sparse autoencoders (SAEs) can resolve superposition in neural networks by decomposing dense activations into interpretable features, yet they have not been systematically applied to biological foundation models. Results: We trained TopK SAEs on residual stream activations from all layers of Geneformer V2-316M (18 layers, d=1,152) and scGPT whole-human (12 layers, d=512), producing atlases of 82,525 and 24,527 features, respectively. Both atlases confirm massive superposition, with 99.8% of features invisible to SVD. Systematic characterization reveals rich biological organization: 29–59% of features annotate to Gene Ontology, KEGG, Reactome, STRING, or TRRUST, with U-shaped layer profiles reflecting hierarchical abstraction. Features organize into co-activation modules (141 in Geneformer, 76 in scGPT), exhibit causal specificity (median 2.36×), and form cross-layer information highways (63–99.8%). When tested against genome-scale CRISPRi perturbation data, only 3 of 48 transcription factors (6.2%) show regulatory-target-specific feature responses. A multi-tissue control yields marginal improvement (10.4%, 5 of 48 TFs), establishing model representations as the bottleneck. Conclusions: These models have internalized organized biological knowledge, including pathway membership, protein interactions, functional modules, and hierarchical abstraction, yet they encode minimal causal regulatory logic. We release both feature atlases as interactive web platforms enabling exploration of more than 107,000 features across 30 layers of two leading single-cell foundation models. sparse autoencoders single-cell foundation models Geneformer scGPT mechanistic interpretability superposition gene regulatory networks co-expression feature atlas Full Text Additional Declarations No competing interests reported. Supplementary Files supplementary.pdf Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9082479","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":611482731,"identity":"02e55802-8aa7-49db-ad1e-5957ba424a21","order_by":0,"name":"Ihor Kendiukhov","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABaklEQVRIie3SwUrDMBgH8G8U6iXbuaOjfYWOgDA29iwNheayjUFBerMwyC7quTvtFSpePHYE6qUo3gp6mPgCG14UlZkubquooDeR/iFpwpdfk9AClCnzB6NqeR+LxkFRxMPI55osVoK8t1U5U74jGNQ1sSSJPxPYklgSEuyIrNjwkdT0UbKAtDOoXSCMhz6n0+lRM1uev4B1MwqUJRMlfXx9P4S28X6wRuKEkLlenSPshCnvRwnCrUlqgXU7C2DGXE9tpB4OgeLNXXpitOAkEoRXmduPVLSvV5kgGRk/zhgnTOu5OgJOgiJZbQk1mSCvkuS7rHJCnwU5LJIsXhOnyjo2JIJUdiQWhCaKIPbmYK4DduqQCVc9jNJOM0pcr3XMMKrnJL10PPFaRUcWbcpdzNDhsEi65OSKn+nI10xzxE+zJ2YYtYzOwT/oDsyQ3j0gv20WfgEbvggqjq3th/pp9ua/W1+mTJky/ytvy72Jl0JdDOoAAAAASUVORK5CYII=","orcid":"","institution":"University of Tübingen","correspondingAuthor":true,"prefix":"","firstName":"Ihor","middleName":"","lastName":"Kendiukhov","suffix":""}],"badges":[],"createdAt":"2026-03-10 10:09:48","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9082479/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9082479/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":108977495,"identity":"4660f56c-2997-4e17-935f-a3ed4331811f","added_by":"auto","created_at":"2026-05-11 11:31:54","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":352710,"visible":true,"origin":"","legend":"","description":"","filename":"bmcgenomicsrevision.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9082479/v1_covered_bd43c4bc-572c-45db-a52d-c0a4d5d269f9.pdf"},{"id":105425089,"identity":"8c66ce27-bfb8-45ae-9f4c-32f837520a7e","added_by":"auto","created_at":"2026-03-25 23:40:46","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":3880868,"visible":true,"origin":"","legend":"","description":"","filename":"supplementary.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9082479/v1/89b37551e42eaffbd4824a15.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Sparse autoencoders reveal organized biological knowledge but minimal regulatory logic in single-cell foundation models: a comparative atlas of Geneformer and scGPT","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"sparse autoencoders, single-cell foundation models, Geneformer, scGPT, mechanistic interpretability, superposition, gene regulatory networks, co-expression, feature atlas","lastPublishedDoi":"10.21203/rs.3.rs-9082479/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9082479/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Background: Single-cell foundation models such as Geneformer and scGPT encode rich biological information, but whether this includes causal regulatory logic rather than statistical co-expression remains unclear. Sparse autoencoders (SAEs) can resolve superposition in neural networks by decomposing dense activations into interpretable features, yet they have not been systematically applied to biological foundation models.\n\nResults: We trained TopK SAEs on residual stream activations from all layers of Geneformer V2-316M (18 layers, d=1,152) and scGPT whole-human (12 layers, d=512), producing atlases of 82,525 and 24,527 features, respectively. Both atlases confirm massive superposition, with 99.8% of features invisible to SVD. Systematic characterization reveals rich biological organization: 29–59% of features annotate to Gene Ontology, KEGG, Reactome, STRING, or TRRUST, with U-shaped layer profiles reflecting hierarchical abstraction. Features organize into co-activation modules (141 in Geneformer, 76 in scGPT), exhibit causal specificity (median 2.36×), and form cross-layer information highways (63–99.8%). When tested against genome-scale CRISPRi perturbation data, only 3 of 48 transcription factors (6.2%) show regulatory-target-specific feature responses. A multi-tissue control yields marginal improvement (10.4%, 5 of 48 TFs), establishing model representations as the bottleneck.\n\nConclusions: These models have internalized organized biological knowledge, including pathway membership, protein interactions, functional modules, and hierarchical abstraction, yet they encode minimal causal regulatory logic. We release both feature atlases as interactive web platforms enabling exploration of more than 107,000 features across 30 layers of two leading single-cell foundation models.","manuscriptTitle":"Sparse autoencoders reveal organized biological knowledge but minimal regulatory logic in single-cell foundation models: a comparative atlas of Geneformer and scGPT","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-03-25 23:40:41","doi":"10.21203/rs.3.rs-9082479/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"92d83e76-9f15-4e45-af4b-643986eb08ac","owner":[],"postedDate":"March 25th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-05-11T07:46:08+00:00","versionOfRecord":[],"versionCreatedAt":"2026-03-25 23:40:41","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9082479","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9082479","identity":"rs-9082479","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}