Evaluation protocol choices dominate reported performance in gene regulatory network benchmarks for single-cell foundation models

Evaluation protocol choices dominate reported performance in gene regulatory network benchmarks for single-cell foundation models | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Evaluation protocol choices dominate reported performance in gene regulatory network benchmarks for single-cell foundation models Ihor Kendiukhov This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9283852/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 8 You are reading this latest preprint version Abstract Gene regulatory network (GRN) recovery claims from single-cell foundation models depend critically 6 on evaluation protocol choices that are rarely reported. We decompose the evaluation pipeline for scGPT 7 attention-derived networks into four dimensions—symbol mapping, gold-standard selection, candidate-8 set restriction, and label noise—and show that each shifts reported AUPR by one to three orders of 9 magnitude, dwarfing the signal attributable to the model itself. We introduce a universe-aware protocol 10 that scores predictions against all biologically plausible TF–target pairs (∼3.6×10 9), rather than restricted 11 candidate sets containing only known TF–target pairs (∼1.9 × 10 6). Under restricted evaluation, scGPT 12 attention achieves AUPR of 0.00738—a 295× inflation over the universe-aware AUPR of 2.5 × 10 −5 — 13 driven entirely by the mechanical increase in positive base rate. Under universe-aware evaluation, scGPT 14 attention falls below a random baseline, while tree-based methods (GENIE3, GRNBoost2) consistently 15 outperform attention-derived edges across three Tabula Sapiens tissues. These findings demonstrate that 16 published GRN recovery claims may reflect evaluation artifacts rather than learned biological structure. Health sciences/Biomarkers Biological sciences/Computational biology and bioinformatics evaluation bias gene regulatory network single-cell foundation model scGPT benchmark methodology universe-aware evaluation Full Text Additional Declarations No competing interests reported. Supplementary Files supplementaryinformation.pdf Cite Share Download PDF Status: Under Review Version 1 posted Reviewers agreed at journal 12 May, 2026 Reviewers agreed at journal 07 May, 2026 Reviewers agreed at journal 07 May, 2026 Reviewers invited by journal 07 May, 2026 Editor invited by journal 06 Apr, 2026 Editor assigned by journal 02 Apr, 2026 Submission checks completed at journal 02 Apr, 2026 First submitted to journal 31 Mar, 2026 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9283852","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":639070858,"identity":"c432be75-6090-4432-bd25-14680e1d9ea3","order_by":0,"name":"Ihor Kendiukhov","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA5klEQVRIiWNgGAWjYBACCRBmbACSzAyMD4AUDx8pWpgNQFrYiNfCwMAmASYJaZFs7z14m3eHTWIDO/u1yq85djJsDMwPH93Ao0Wa51yyNe+ZtMQGZp6y27LbkoEOYzM2zsGjRU4ix0yat+2wMQMzT9ptyW3MQC08bNJ4tci/QWgpltxWT1iLtAQPWIscAzP7McaP2w4T1iLZk2NsObctTY6NmYdZmnHbcR42ZgJ+kTh+xvDG2zYbHn7+4w8//txWbc/P3vzwMT4tcAB0jwEzD4jFTIxyCGB/wPiDeNWjYBSMglEwggAAJNw6fVf+p8MAAAAASUVORK5CYII=","orcid":"","institution":"University of Tübingen","correspondingAuthor":true,"prefix":"","firstName":"Ihor","middleName":"","lastName":"Kendiukhov","suffix":""}],"badges":[],"createdAt":"2026-03-31 19:38:27","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9283852/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9283852/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":109296408,"identity":"22507257-b669-4248-bef8-b2665288119b","added_by":"auto","created_at":"2026-05-15 08:46:52","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":695614,"visible":true,"origin":"","legend":"","description":"","filename":"evalbiasscirep.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9283852/v1_covered_40a38aad-8ad4-4fae-9b28-73000f6e9745.pdf"},{"id":109291347,"identity":"ec67f8fe-48f4-4d7e-ae75-c5b64f355b75","added_by":"auto","created_at":"2026-05-15 07:50:16","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":132494,"visible":true,"origin":"","legend":"","description":"","filename":"supplementaryinformation.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9283852/v1/bac55be9a50607fa681fa0e2.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Evaluation protocol choices dominate reported performance in gene regulatory network benchmarks for single-cell foundation models","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"evaluation bias, gene regulatory network, single-cell foundation model, scGPT, benchmark methodology, universe-aware evaluation","lastPublishedDoi":"10.21203/rs.3.rs-9283852/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9283852/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eGene regulatory network (GRN) recovery claims from single-cell foundation models depend critically 6 on evaluation protocol choices that are rarely reported. We decompose the evaluation pipeline for scGPT 7 attention-derived networks into four dimensions—symbol mapping, gold-standard selection, candidate-8 set restriction, and label noise—and show that each shifts reported AUPR by one to three orders of 9 magnitude, dwarfing the signal attributable to the model itself. We introduce a universe-aware protocol 10 that scores predictions against all biologically plausible TF–target pairs (∼3.6×10 9), rather than restricted 11 candidate sets containing only known TF–target pairs (∼1.9 × 10 6). Under restricted evaluation, scGPT 12 attention achieves AUPR of 0.00738—a 295× inflation over the universe-aware AUPR of 2.5 × 10 −5 — 13 driven entirely by the mechanical increase in positive base rate. Under universe-aware evaluation, scGPT 14 attention falls below a random baseline, while tree-based methods (GENIE3, GRNBoost2) consistently 15 outperform attention-derived edges across three Tabula Sapiens tissues. These findings demonstrate that 16 published GRN recovery claims may reflect evaluation artifacts rather than learned biological structure.\u003c/p\u003e","manuscriptTitle":"Evaluation protocol choices dominate reported performance in gene regulatory network benchmarks for single-cell foundation models","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-05-15 07:50:13","doi":"10.21203/rs.3.rs-9283852/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"reviewerAgreed","content":"122675671771402837039327331007697783484","date":"2026-05-12T13:49:12+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"280497296153267055850757752081523621448","date":"2026-05-07T16:57:41+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"242849469086745158443280177480839474944","date":"2026-05-07T15:35:25+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2026-05-07T07:49:27+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2026-04-06T12:44:47+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-04-02T04:41:05+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2026-04-02T04:40:18+00:00","index":"","fulltext":""},{"type":"submitted","content":"Scientific Reports","date":"2026-03-31T19:34:51+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"867c6459-0340-4460-8d1e-406517f05427","owner":[],"postedDate":"May 15th, 2026","published":true,"recentEditorialEvents":[{"type":"reviewerAgreed","content":"122675671771402837039327331007697783484","date":"2026-05-12T13:49:12+00:00","index":77,"fulltext":""},{"type":"reviewerAgreed","content":"280497296153267055850757752081523621448","date":"2026-05-07T16:57:41+00:00","index":74,"fulltext":""},{"type":"reviewerAgreed","content":"242849469086745158443280177480839474944","date":"2026-05-07T15:35:25+00:00","index":73,"fulltext":""},{"type":"reviewersInvited","content":"16","date":"2026-05-07T07:49:27+00:00","index":"","fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[{"id":68019205,"name":"Health sciences/Biomarkers"},{"id":68019206,"name":"Biological sciences/Computational biology and bioinformatics"}],"tags":[],"updatedAt":"2026-05-15T07:50:13+00:00","versionOfRecord":[],"versionCreatedAt":"2026-05-15 07:50:13","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9283852","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9283852","identity":"rs-9283852","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}