Auditing frontier general-purpose large language models in biomedical tasks: reasoning gains, extraction limits, and benchmark reliability

Auditing frontier general-purpose large language models in biomedical tasks: reasoning gains, extraction limits, and benchmark reliability | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Auditing frontier general-purpose large language models in biomedical tasks: reasoning gains, extraction limits, and benchmark reliability Yu Hou, Zaifu Zhan, Min Zeng, Yifan Wu, Shuang Zhou, Xiaoyi Chen, and 3 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8605899/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract As large language models approach clinical deployment, their deployment-relevant reliability and the validity of the benchmarks used to assess it remain insufficiently examined. Here, we present a unified, reproducible, and human-centric audit of frontier general-purpose language models using representative biomedical text-mining tasks and nine biomedical question-answering benchmarks spanning reasoning-intensive, extraction-oriented, and multimodal settings. We observe consistent gains in clinical reasoning and multimodal biomedical QA; however, limitations in format-constrained tasks such as span-level extraction and evidence-dense summarization pose challenges for integration into structured clinical workflows, despite narrowing gaps with supervised systems. Blinded expert adjudication confirms more coherent and clinically plausible reasoning and further reveals that a substantial fraction of apparent errors arises from outdated or ambiguous benchmark annotations, suggesting that current benchmarks may misestimate model capability and potentially misguide deployment decisions. Cost-normalized analyses demonstrate that recent frontier models achieve higher accuracy at substantially lower cost per correct answer, reshaping practical deployment trade-offs for scalable digital medicine systems. Together, these findings suggest that general-purpose language models are approaching deployment-relevant reliability; however, safe and effective clinical use will require hybrid architectures, external grounding, and human-in-the-loop evaluation and expert oversight. Biological sciences/Computational biology and bioinformatics Health sciences/Health care Physical sciences/Mathematics and computing Health sciences/Medical research Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8605899","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":592872113,"identity":"70d402e4-9a38-48b4-aadd-e049b4724293","order_by":0,"name":"Yu Hou","email":"","orcid":"","institution":"University of Minnesota","correspondingAuthor":false,"prefix":"","firstName":"Yu","middleName":"","lastName":"Hou","suffix":""},{"id":592872114,"identity":"be47c250-3d8d-48ed-acdf-50d1dcb6670c","order_by":1,"name":"Zaifu Zhan","email":"","orcid":"","institution":"University of Minnesota","correspondingAuthor":false,"prefix":"","firstName":"Zaifu","middleName":"","lastName":"Zhan","suffix":""},{"id":592872115,"identity":"550a4a35-bdc2-4ce9-b7fe-774aa5e5ab1d","order_by":2,"name":"Min Zeng","email":"","orcid":"","institution":"University of Minnesota","correspondingAuthor":false,"prefix":"","firstName":"Min","middleName":"","lastName":"Zeng","suffix":""},{"id":592872116,"identity":"0ea4ee67-8d94-42c4-9e50-7d7ae4fdc85f","order_by":3,"name":"Yifan Wu","email":"","orcid":"","institution":"University of Minnesota","correspondingAuthor":false,"prefix":"","firstName":"Yifan","middleName":"","lastName":"Wu","suffix":""},{"id":592872117,"identity":"33b9599f-4b80-49ed-85db-9904a5297903","order_by":4,"name":"Shuang Zhou","email":"","orcid":"","institution":"University of Minnesota","correspondingAuthor":false,"prefix":"","firstName":"Shuang","middleName":"","lastName":"Zhou","suffix":""},{"id":592872118,"identity":"d14e2f86-077f-49aa-b8dc-09df5b6ced20","order_by":5,"name":"Xiaoyi Chen","email":"","orcid":"","institution":"University of Minnesota","correspondingAuthor":false,"prefix":"","firstName":"Xiaoyi","middleName":"","lastName":"Chen","suffix":""},{"id":592872119,"identity":"96fc5b74-16f5-44b0-8de6-60ebf9a2e2b4","order_by":6,"name":"Huixue Zhou","email":"","orcid":"","institution":"University of Minnesota","correspondingAuthor":false,"prefix":"","firstName":"Huixue","middleName":"","lastName":"Zhou","suffix":""},{"id":592872120,"identity":"4abe7840-452a-4e67-b2d1-cfaaedb160a7","order_by":7,"name":"Meijia Song","email":"","orcid":"","institution":"University of Minnesota","correspondingAuthor":false,"prefix":"","firstName":"Meijia","middleName":"","lastName":"Song","suffix":""},{"id":592872121,"identity":"80163e03-7e9b-4750-9ef6-d5906b06e595","order_by":8,"name":"Rui Zhang","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAuUlEQVRIiWNgGAWjYFACxgZmEMUP4TGToEWygXgtUGUGB4jVYj4juflzQc0du803stMeMFRYJzYQ0iJzI7HBeMaxZ8nbbuRuN2A4k05Yi4REYkMyD9vhZLMbudskGNsOE6flMM+/w8nGM0Ba/hGnpbGZt+2wnYEESEsDMVp4HjYz8/YdTpA483abRMKxdGPCWtjTH3/m+XbYnr8daMuHGmtZglpgAOKeBGKVg4A9KYpHwSgYBaNghAEANgI+i397l8IAAAAASUVORK5CYII=","orcid":"","institution":"University of Minnesota","correspondingAuthor":true,"prefix":"","firstName":"Rui","middleName":"","lastName":"Zhang","suffix":""}],"badges":[],"createdAt":"2026-01-15 01:53:19","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8605899/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8605899/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":107026327,"identity":"5ef240ff-560b-4df0-bdd0-2c174eb376be","added_by":"auto","created_at":"2026-04-16 01:25:09","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2209934,"visible":true,"origin":"","legend":"","description":"","filename":"npjDMv1.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8605899/v1_covered_516708cf-eebf-46d5-8000-ad8542ae9abc.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Auditing frontier general-purpose large language models in biomedical tasks: reasoning gains, extraction limits, and benchmark reliability","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-8605899/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8605899/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eAs large language models approach clinical deployment, their deployment-relevant reliability and the validity of the benchmarks used to assess it remain insufficiently examined. Here, we present a unified, reproducible, and human-centric audit of frontier general-purpose language models using representative biomedical text-mining tasks and nine biomedical question-answering benchmarks spanning reasoning-intensive, extraction-oriented, and multimodal settings. We observe consistent gains in clinical reasoning and multimodal biomedical QA; however, limitations in format-constrained tasks such as span-level extraction and evidence-dense summarization pose challenges for integration into structured clinical workflows, despite narrowing gaps with supervised systems. Blinded expert adjudication confirms more coherent and clinically plausible reasoning and further reveals that a substantial fraction of apparent errors arises from outdated or ambiguous benchmark annotations, suggesting that current benchmarks may misestimate model capability and potentially misguide deployment decisions. Cost-normalized analyses demonstrate that recent frontier models achieve higher accuracy at substantially lower cost per correct answer, reshaping practical deployment trade-offs for scalable digital medicine systems. Together, these findings suggest that general-purpose language models are approaching deployment-relevant reliability; however, safe and effective clinical use will require hybrid architectures, external grounding, and human-in-the-loop evaluation and expert oversight.\u003c/p\u003e","manuscriptTitle":"Auditing frontier general-purpose large language models in biomedical tasks: reasoning gains, extraction limits, and benchmark reliability","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-02-18 17:14:16","doi":"10.21203/rs.3.rs-8605899/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"81f46f01-2597-41bf-90dc-d06982845d57","owner":[],"postedDate":"February 18th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":63085521,"name":"Biological sciences/Computational biology and bioinformatics"},{"id":63085522,"name":"Health sciences/Health care"},{"id":63085523,"name":"Physical sciences/Mathematics and computing"},{"id":63085524,"name":"Health sciences/Medical research"}],"tags":[],"updatedAt":"2026-04-22T04:53:34+00:00","versionOfRecord":[],"versionCreatedAt":"2026-02-18 17:14:16","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8605899","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8605899","identity":"rs-8605899","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}