Learning the Phenotype of Medical Hallucinations

Learning the Phenotype of Medical Hallucinations | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Learning the Phenotype of Medical Hallucinations Carlos Garcia Fernandez, Luis Felipe, Monique Shotande, Muntasir Zitu, and 5 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7475667/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract The clinical deployment of powerful general-purpose large language models (LLMs) is fundamentally limited by their propensity for unreliable generation ("hallucination"), posing a significant safety risk in high-stakes domains. Here, we introduce CHECK, a model-agnostic safety layer that learns a hallucination phenotype from first principles of information theory to proactively suppress unsafe outputs. Rather than relying on content-based heuristics, CHECK integrates structured, open clinical knowledge with an independent classifier that estimates hallucination probability from distributional signals, such as the uncertainty (entropy) of individual token predictions and the divergence (Kullback–Leibler divergence) of probability distributions across an ensemble of language models. In a high-stakes setting of physician question and answer (QA) about pivotal oncology trials, where misinformation can have severe consequences, CHECK reduced hallucinations in a state-of-the-art open model (Llama-3-70B) from a clinically untenable 31% to just 0.3%. This capability to discern hallucination generalized across diverse medical tasks (reasoning and education, patient and physician dialogue, radiology report summarization) and a suite of leading open-weight and commercial models (e.g., GPT-4o, DeepSeek-R1-Distill-Llama-70B, GPT-o3, GPT-5), achieving detection AUCs of 0.95–0.98. As a practical application, we demonstrate that the hallucination probability signal from CHECK can guide an iterative refinement process, improving GPT-4o’s USMLE pass rate by five percentage points to a new state-of-the-art of 92.1% while significantly reducing compute. By systematically driving hallucination risk below accepted clinical error thresholds, CHECK provides a scalable and plug-and-play solution to enable the reliable deployment of LLMs in medicine and other mission-critical domains. Health sciences/Health care Physical sciences/Mathematics and computing/Computational science Full Text Additional Declarations There is NO Competing Interest. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7475667","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":531936997,"identity":"b314b115-a9ff-458c-8dcb-13acb654e7e3","order_by":0,"name":"Carlos Garcia Fernandez","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABZElEQVRIie2Rv0rDQBjALwjNEsl6pVJf4ZOArag4+CJ3FNIlSiEgHWo5KaSDFddIRV8hfYMrgXRJRyGhIhWhLg7pliGIFy22anAWzI/j7rv77sd3fxDKyfmbSHwxMtT8iEC0vbIqszSuZjlLxf9UGrpW7KUJwL8ogqUSudSxSTr5oajd8ZRHyelmpeueRbz5cKz2O4PHGFwNis93TtzAYs85WVGwX4fhlTXauvZpB3PfNPG9ZwKGerlSMk7CnjgY9sfOigJIR+468yQbS6w0twhlgbGNAXa1nb6hB4pQAB99UdQZcpXEOxBKJx6+EnobGJWYwBp1Ql8PkwwFiypKoUWFYuEhI9QRVRCHQxHI3iSjCg5mSNyF12yFWlXuEToIdLPIIH1kozDZAKx8u4t6qUvTKGnv2/LoKeAtQm+C2mAeJ+lXjmbhSxp0L1aVBW7WfyEF3vvMHGpnrsrT7N05OTk5/4w3AvONodNXsLUAAAAASUVORK5CYII=","orcid":"https://orcid.org/0000-0003-2226-3904","institution":"Moffitt Cancer Center","correspondingAuthor":true,"prefix":"","firstName":"Carlos","middleName":"Garcia","lastName":"Fernandez","suffix":""},{"id":531936998,"identity":"94f06fb0-3008-4c23-8f17-777ca325a181","order_by":1,"name":"Luis Felipe","email":"","orcid":"","institution":"Moffitt Cancer Center","correspondingAuthor":false,"prefix":"","firstName":"Luis","middleName":"","lastName":"Felipe","suffix":""},{"id":531936999,"identity":"c9829658-e998-4a84-a852-991513626039","order_by":2,"name":"Monique Shotande","email":"","orcid":"","institution":"Moffitt Cancer Center","correspondingAuthor":false,"prefix":"","firstName":"Monique","middleName":"","lastName":"Shotande","suffix":""},{"id":531937000,"identity":"bb3937d4-3ae7-40d3-a005-36f46e76d75c","order_by":3,"name":"Muntasir Zitu","email":"","orcid":"","institution":"Moffitt Cancer Center","correspondingAuthor":false,"prefix":"","firstName":"Muntasir","middleName":"","lastName":"Zitu","suffix":""},{"id":531937001,"identity":"c26345bf-f508-4cbc-83fb-b068d410fc37","order_by":4,"name":"Elier Delgado","email":"","orcid":"","institution":"Inova Montreal","correspondingAuthor":false,"prefix":"","firstName":"Elier","middleName":"","lastName":"Delgado","suffix":""},{"id":531937002,"identity":"fe062ccc-4e27-4dc5-8b67-0cebd8fc07d2","order_by":5,"name":"Ghulam Rasool","email":"","orcid":"https://orcid.org/0000-0001-8551-0090","institution":"Moffitt Cancer Center","correspondingAuthor":false,"prefix":"","firstName":"Ghulam","middleName":"","lastName":"Rasool","suffix":""},{"id":531937003,"identity":"e2290b8c-a6ed-45a8-87af-05cba03f5dea","order_by":6,"name":"Issam El Naqa","email":"","orcid":"https://orcid.org/0000-0001-6023-1132","institution":"Moffitt Cancer Center","correspondingAuthor":false,"prefix":"","firstName":"Issam","middleName":"El","lastName":"Naqa","suffix":""},{"id":531937004,"identity":"c2519554-2e50-418d-874e-b01609d91e62","order_by":7,"name":"Vivek Rudrapatna","email":"","orcid":"https://orcid.org/0000-0003-1789-3004","institution":"University of California, San Francisco","correspondingAuthor":false,"prefix":"","firstName":"Vivek","middleName":"","lastName":"Rudrapatna","suffix":""},{"id":531937005,"identity":"c882930e-f931-4106-a590-3a1a301fde46","order_by":8,"name":"Gilmer Valdes","email":"","orcid":"","institution":"Moffitt Cancer Center","correspondingAuthor":false,"prefix":"","firstName":"Gilmer","middleName":"","lastName":"Valdes","suffix":""}],"badges":[],"createdAt":"2025-08-28 02:55:31","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7475667/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7475667/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":94013818,"identity":"6cbb88ef-51dd-4128-8bed-8c83b86f6e85","added_by":"auto","created_at":"2025-10-21 10:41:04","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":5843101,"visible":true,"origin":"","legend":"","description":"","filename":"naturemedicine.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7475667/v1/1d33ef263682e6b6b31059f0.pdf"},{"id":94013816,"identity":"6bc741d0-275c-4553-9971-06adfa7e0540","added_by":"auto","created_at":"2025-10-21 10:41:03","extension":"json","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":9912,"visible":true,"origin":"","legend":"","description":"","filename":"NCOMMS2583786T.json","url":"https://assets-eu.researchsquare.com/files/rs-7475667/v1/c6370a438cae54a44d112073.json"},{"id":94013817,"identity":"38b6c5d7-73dc-4eeb-a58d-c874b982724a","added_by":"auto","created_at":"2025-10-21 10:41:04","extension":"pdf","order_by":2,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":1667998,"visible":true,"origin":"","legend":"","description":"","filename":"nrreportingsummaryfinal.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7475667/v1/99e284314ffda80a80188445.pdf"},{"id":94988711,"identity":"c1314320-c509-45e8-8fb8-6922e9cfb91a","added_by":"auto","created_at":"2025-11-03 07:10:32","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":3804061,"visible":true,"origin":"","legend":"Article File","description":"","filename":"naturemedicine.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7475667/v1_covered_027300a1-1dc6-4405-95c9-d1a081c55c0e.pdf"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"Learning the Phenotype of Medical Hallucinations","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-7475667/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7475667/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"The clinical deployment of powerful general-purpose large language models (LLMs) is fundamentally limited by their propensity for unreliable generation (\"hallucination\"), posing a significant safety risk in high-stakes domains. Here, we introduce CHECK, a model-agnostic safety layer that learns a hallucination phenotype from first principles of information theory to proactively suppress unsafe outputs. Rather than relying on content-based heuristics, CHECK integrates structured, open clinical knowledge with an independent classifier that estimates hallucination probability from distributional signals, such as the uncertainty (entropy) of individual token predictions and the divergence (Kullback–Leibler divergence) of probability distributions across an ensemble of language models. In a high-stakes setting of physician question and answer (QA) about pivotal oncology trials, where misinformation can have severe consequences, CHECK reduced hallucinations in a state-of-the-art open model (Llama-3-70B) from a clinically untenable 31\\% to just 0.3\\%. This capability to discern hallucination generalized across diverse medical tasks (reasoning and education, patient and physician dialogue, radiology report summarization) and a suite of leading open-weight and commercial models (e.g., GPT-4o, DeepSeek-R1-Distill-Llama-70B, GPT-o3, GPT-5), achieving detection AUCs of 0.95–0.98. As a practical application, we demonstrate that the hallucination probability signal from CHECK can guide an iterative refinement process, improving GPT-4o’s USMLE pass rate by five percentage points to a new state-of-the-art of 92.1\\% while significantly reducing compute. By systematically driving hallucination risk below accepted clinical error thresholds, CHECK provides a scalable and plug-and-play solution to enable the reliable deployment of LLMs in medicine and other mission-critical domains.","manuscriptTitle":"Learning the Phenotype of Medical Hallucinations","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-10-21 10:40:59","doi":"10.21203/rs.3.rs-7475667/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"5cff6e19-3fdb-4026-aa7e-8fe11d2e579b","owner":[],"postedDate":"October 21st, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":56548883,"name":"Health sciences/Health care"},{"id":56548884,"name":"Physical sciences/Mathematics and computing/Computational science"}],"tags":[],"updatedAt":"2025-11-03T01:45:20+00:00","versionOfRecord":[],"versionCreatedAt":"2025-10-21 10:40:59","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7475667","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7475667","identity":"rs-7475667","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}