Benchmarking DNA Foundation Models for zero-shot variant effect prediction: the role of context, training, and architecture

Benchmarking DNA Foundation Models for zero-shot variant effect prediction: the role of context, training, and architecture | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Benchmarking DNA Foundation Models for zero-shot variant effect prediction: the role of context, training, and architecture Ilaria Alfisi, Francesca Ciapi, Marta Baragli, Alberto Magi This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6988910/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 9 You are reading this latest preprint version Abstract In this study, we systematically evaluate the performance of several DNA foundation models (NT, DNABERT, and HyenaDNA) in predicting the functional impact of genetic variants using Zero-shot scoring, a method that does not require task-specific fine-tuning. We assess the models’ sensitivity to sequence alterations introduced by Single Nucleotide Variants (SNVs), comparing their ability to capture both local and extended contextual effects. Using pathogenic, benign, and uncertain SNVs from Clin-Var, we show that large multi-species NT models outperform other architectures in detecting functional consequences, not only at the mutation site but also in adjacent regions. These models exhibit superior discriminative power across variant categories, especially when aggregating Zero-shot scores over multiple surrounding tokens. Conversely, models trained solely on human sequences, such as DNABERT and HyenaDNA, show limited contextual awareness and reduced ability to differentiate variant effects. Our findings highlight the critical importance of model size, training objective, and training data diversity in shaping model performance. Furthermore, we discuss current limitations in modeling long-range dependencies in genomic sequences and suggest that innovations in transformer architectures, such as sparse attention or memory-augmented models, may provide viable paths toward scalable, genome-wide variant effect prediction. Full Text Additional Declarations No competing interests reported. Supplementary Files LLMSupplementary.pdf Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 12 Nov, 2025 Reviews received at journal 20 Oct, 2025 Reviews received at journal 07 Oct, 2025 Reviewers agreed at journal 18 Aug, 2025 Reviewers agreed at journal 05 Aug, 2025 Reviewers invited by journal 31 Jul, 2025 Editor assigned by journal 30 Jun, 2025 Submission checks completed at journal 30 Jun, 2025 First submitted to journal 27 Jun, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6988910","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":494741007,"identity":"e148ab57-9542-41a4-b6e7-b4fe9c2cbac1","order_by":0,"name":"Ilaria Alfisi","email":"","orcid":"","institution":"University of Florence","correspondingAuthor":false,"prefix":"","firstName":"Ilaria","middleName":"","lastName":"Alfisi","suffix":""},{"id":494741008,"identity":"122cd30f-85a0-4903-8b23-28826e9b97fe","order_by":1,"name":"Francesca Ciapi","email":"","orcid":"","institution":"University of Florence","correspondingAuthor":false,"prefix":"","firstName":"Francesca","middleName":"","lastName":"Ciapi","suffix":""},{"id":494741010,"identity":"7b3d097d-b9df-4933-828a-cb010dd0b538","order_by":2,"name":"Marta Baragli","email":"","orcid":"","institution":"University of Florence","correspondingAuthor":false,"prefix":"","firstName":"Marta","middleName":"","lastName":"Baragli","suffix":""},{"id":494741012,"identity":"7094fa41-d391-4042-92a8-654adabeddd6","order_by":3,"name":"Alberto Magi","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA50lEQVRIie3RIQvCQBjG8fcwrLxqVRD9BIIrJ8LEr3IgzDLBJNoOBC3a/Tgbb7AMxWYwCIJpYTZt3qai6bYoeP/0hPtxGwdgMv1iFn62HwOgNVNDSwoJEc8drBVBUkRrvkm6MTmuI+VZMThPbw40F5s9defHGlrAKNaQCpX6dihc4KE3ouH8gljI+jBCXpWCgPueUISwl0UahO17SnaRoI4imbe01C0sJYeBTywPsdW/VKXrIj9EECy3CWHSDzWkvlkFV+k4db4bnOPbmHpYJoonGvJOvQi2XpvJHCDNOuU9aTKZTH/WAwytUNadeIxGAAAAAElFTkSuQmCC","orcid":"","institution":"University of Florence","correspondingAuthor":true,"prefix":"","firstName":"Alberto","middleName":"","lastName":"Magi","suffix":""}],"badges":[],"createdAt":"2025-06-27 07:23:14","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6988910/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6988910/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":88317356,"identity":"23cffaf0-4e4e-429e-a0ca-09bbfacbf2d2","added_by":"auto","created_at":"2025-08-05 08:14:19","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1420640,"visible":true,"origin":"","legend":"","description":"","filename":"LLM.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6988910/v1_covered_fe5d7508-6f1e-4c6c-9094-890910e1998a.pdf"},{"id":88316433,"identity":"97d74560-e86d-4e3d-8218-badf086cb65d","added_by":"auto","created_at":"2025-08-05 08:06:17","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":6257159,"visible":true,"origin":"","legend":"","description":"","filename":"LLMSupplementary.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6988910/v1/ebba60c86b80290878d62bd8.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Benchmarking DNA Foundation Models for zero-shot variant effect prediction: the role of context, training, and architecture","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"genome-biology","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"gbio","sideBox":"Learn more about [Genome Biology](https://genomebiology.biomedcentral.com/)","snPcode":"13059","submissionUrl":"https://submission.springernature.com/new-submission/13059/3","title":"Genome Biology","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"BMC/SO AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-6988910/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6988910/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"In this study, we systematically evaluate the performance of several DNA foundation models (NT, DNABERT, and HyenaDNA) in predicting the functional impact of genetic variants using Zero-shot scoring, a method that does not require task-specific fine-tuning. We assess the models’ sensitivity to sequence alterations introduced by Single Nucleotide Variants (SNVs), comparing their ability to capture both local and extended contextual effects. Using pathogenic, benign, and uncertain SNVs from Clin-Var, we show that large multi-species NT models outperform other architectures in detecting functional consequences, not only at the mutation site but also in adjacent regions. These models exhibit superior discriminative power across variant categories, especially when aggregating Zero-shot scores over multiple surrounding tokens. Conversely, models trained solely on human sequences, such as DNABERT and HyenaDNA, show limited contextual awareness and reduced ability to differentiate variant effects. Our findings highlight the critical importance of model size, training objective, and training data diversity in shaping model performance. Furthermore, we discuss current limitations in modeling long-range dependencies in genomic sequences and suggest that innovations in transformer architectures, such as sparse attention or memory-augmented models, may provide viable paths toward scalable, genome-wide variant effect prediction.","manuscriptTitle":"Benchmarking DNA Foundation Models for zero-shot variant effect prediction: the role of context, training, and architecture","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-08-05 07:58:12","doi":"10.21203/rs.3.rs-6988910/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-11-12T14:47:15+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-10-20T22:16:14+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-10-07T17:54:14+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"179160858909998529862899304547158077054","date":"2025-08-18T13:44:21+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"222964437963881141849512434694271067527","date":"2025-08-06T01:09:10+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-07-31T16:18:01+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-06-30T13:40:15+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-06-30T06:26:50+00:00","index":"","fulltext":""},{"type":"submitted","content":"Genome Biology","date":"2025-06-27T07:10:47+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"genome-biology","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"gbio","sideBox":"Learn more about [Genome Biology](https://genomebiology.biomedcentral.com/)","snPcode":"13059","submissionUrl":"https://submission.springernature.com/new-submission/13059/3","title":"Genome Biology","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"BMC/SO AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"c7677ee3-4d17-4b7d-b964-f71e58c1f23c","owner":[],"postedDate":"August 5th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2026-04-22T13:39:35+00:00","versionOfRecord":[],"versionCreatedAt":"2025-08-05 07:58:12","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-6988910","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6988910","identity":"rs-6988910","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}