Enhancing Substance Use Detection in Clinical Notes with Large Language Models

Enhancing Substance Use Detection in Clinical Notes with Large Language Models | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Enhancing Substance Use Detection in Clinical Notes with Large Language Models Fabrice Harel-Canada, Anabel Salimian, Brandon Moghanian, Sarah Clingan, and 10 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6615981/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Identifying substance use behaviors in electronic health records (EHRs) is challenging because critical details are often buried in unstructured notes that use varied terminology and negation, requiring careful contextual interpretation to distinguish relevant use from historical mentions or denials. Using MIMIC-III/IV discharge summaries, we created a large, annotated drug detection dataset to tackle this problem and support future systemic substance use surveillance. We then investigated the performance of multiple large language models (LLMs) for detecting eight substance use categories within this data. Evaluating models in zero-shot, few-shot, and fine-tuning configurations, we found that a fine-tuned model, Llama-DrugDetector-70B, outperformed others. It achieved near-perfect F1-scores (>=0.95) for most individual substances and strong scores for more complex tasks like prescription opioid misuse (F1=0.815) and polysubstance use (F1=0.917). These findings demonstrate that LLMs significantly enhance detection, showing promise for clinical decision support and research, although further work on scalability is warranted. Health sciences/Medical research/Epidemiology Health sciences/Health care/Public health/Epidemiology NLP natural language processing substance use drug use people who inject drugs Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6615981","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":456192643,"identity":"5528a863-4d02-47d8-bed3-e955d0f65312","order_by":0,"name":"Fabrice Harel-Canada","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA6UlEQVRIiWNgGAWjYBAC9gYeBoYEEGJvAAvIgAgJfFp4DsC08ByACBCnhQGkRSKBWC3sZ499eFBjk8c/8/HjD4xtdjz8DcwHb/Pg08KTlzwj4VhascTtNDMJxrZkHokDbMnW+LTYM+QYMyQ2HE7cIJ3DxsDYBnTnAR4zaby28L8BafmfuEHyDPMHkBb5A/zf8GuRANtyIHGDBA+DBEiLwQEeNgJa3iUzJBxLTpxxBuiXhHPJPIaH2Ywt5+B1WO5hxh81don97Ycff/hQZicnd7z54Y03eLSgggQQwUy08lEwCkbBKBgFuAAAx0lFkj+DuRsAAAAASUVORK5CYII=","orcid":"","institution":"University of California, Los Angeles","correspondingAuthor":true,"prefix":"","firstName":"Fabrice","middleName":"","lastName":"Harel-Canada","suffix":""},{"id":456192644,"identity":"ddfea63d-ea6a-44dc-8d9b-421db9da0fc6","order_by":1,"name":"Anabel Salimian","email":"","orcid":"","institution":"University of California, Los Angeles","correspondingAuthor":false,"prefix":"","firstName":"Anabel","middleName":"","lastName":"Salimian","suffix":""},{"id":456192645,"identity":"9a29952d-4aed-4ad8-a1c9-81fdd83dffef","order_by":2,"name":"Brandon Moghanian","email":"","orcid":"","institution":"University of California, Los Angeles","correspondingAuthor":false,"prefix":"","firstName":"Brandon","middleName":"","lastName":"Moghanian","suffix":""},{"id":456192646,"identity":"32276b0e-0c38-4f48-8453-8b4fe87b976d","order_by":3,"name":"Sarah Clingan","email":"","orcid":"","institution":"University of California, Los Angeles","correspondingAuthor":false,"prefix":"","firstName":"Sarah","middleName":"","lastName":"Clingan","suffix":""},{"id":456192647,"identity":"b1bf9147-9044-4228-b814-264170ad5205","order_by":4,"name":"Allan Nguyen","email":"","orcid":"","institution":"University of California, Los Angeles","correspondingAuthor":false,"prefix":"","firstName":"Allan","middleName":"","lastName":"Nguyen","suffix":""},{"id":456192648,"identity":"8b4d42c8-22bd-479a-935f-0c7a549f703b","order_by":5,"name":"Tucker Avra","email":"","orcid":"","institution":"University of California, Los Angeles","correspondingAuthor":false,"prefix":"","firstName":"Tucker","middleName":"","lastName":"Avra","suffix":""},{"id":456192649,"identity":"87e8520b-6c2c-4cbe-b60b-05e77a8c7ac2","order_by":6,"name":"Michelle Poimboeuf","email":"","orcid":"","institution":"University of California, Los Angeles","correspondingAuthor":false,"prefix":"","firstName":"Michelle","middleName":"","lastName":"Poimboeuf","suffix":""},{"id":456192650,"identity":"1cc871a9-61f0-4c8b-a926-ae410c87d9dc","order_by":7,"name":"Ruby Romero","email":"","orcid":"","institution":"University of California, Los Angeles","correspondingAuthor":false,"prefix":"","firstName":"Ruby","middleName":"","lastName":"Romero","suffix":""},{"id":456192651,"identity":"d523588c-3029-4ebd-8a49-1abf180ff4eb","order_by":8,"name":"Arthur Funnell","email":"","orcid":"","institution":"University of California, Los Angeles","correspondingAuthor":false,"prefix":"","firstName":"Arthur","middleName":"","lastName":"Funnell","suffix":""},{"id":456192652,"identity":"4bc9724f-f409-4cfb-bbf6-51e991b04622","order_by":9,"name":"Panayiotis Petousis","email":"","orcid":"","institution":"University of California, Los Angeles","correspondingAuthor":false,"prefix":"","firstName":"Panayiotis","middleName":"","lastName":"Petousis","suffix":""},{"id":456192653,"identity":"d23b0a20-403a-40a3-9b38-9f6405fa622e","order_by":10,"name":"Michael Shin","email":"","orcid":"","institution":"University of California, Los Angeles","correspondingAuthor":false,"prefix":"","firstName":"Michael","middleName":"","lastName":"Shin","suffix":""},{"id":456192654,"identity":"2ea71fba-1afa-40ad-8498-47925be773cc","order_by":11,"name":"Nanyun Peng","email":"","orcid":"","institution":"University of California, Los Angeles","correspondingAuthor":false,"prefix":"","firstName":"Nanyun","middleName":"","lastName":"Peng","suffix":""},{"id":456192655,"identity":"699c3868-38fc-42a9-a5dc-f3c26674487b","order_by":12,"name":"Chelsea Shover","email":"","orcid":"","institution":"University of California, Los Angeles","correspondingAuthor":false,"prefix":"","firstName":"Chelsea","middleName":"","lastName":"Shover","suffix":""},{"id":456192656,"identity":"be97100e-38f9-4ed1-b92d-3ddd60526b7e","order_by":13,"name":"David Goodman-Meza","email":"","orcid":"","institution":"UNSW Sydney","correspondingAuthor":false,"prefix":"","firstName":"David","middleName":"","lastName":"Goodman-Meza","suffix":""}],"badges":[],"createdAt":"2025-05-08 02:53:14","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6615981/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6615981/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":82801156,"identity":"1474d15d-51d6-43d3-86f9-831486d2c3f4","added_by":"auto","created_at":"2025-05-15 11:20:41","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1215089,"visible":true,"origin":"","legend":"","description":"","filename":"SubstanceAbuseClassificationsFinal.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6615981/v1_covered_65beba78-ef6a-4d95-94ba-1a8353706b54.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Enhancing Substance Use Detection in Clinical Notes with Large Language Models","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"NLP, natural language processing, substance use, drug use, people who inject drugs","lastPublishedDoi":"10.21203/rs.3.rs-6615981/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6615981/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Identifying substance use behaviors in electronic health records (EHRs) is challenging because critical details are often buried in unstructured notes that use varied terminology and negation, requiring careful contextual interpretation to distinguish relevant use from historical mentions or denials. Using MIMIC-III/IV discharge summaries, we created a large, annotated drug detection dataset to tackle this problem and support future systemic substance use surveillance. We then investigated the performance of multiple large language models (LLMs) for detecting eight substance use categories within this data. Evaluating models in zero-shot, few-shot, and fine-tuning configurations, we found that a fine-tuned model, Llama-DrugDetector-70B, outperformed others. It achieved near-perfect F1-scores (\u003e=0.95) for most individual substances and strong scores for more complex tasks like prescription opioid misuse (F1=0.815) and polysubstance use (F1=0.917). These findings demonstrate that LLMs significantly enhance detection, showing promise for clinical decision support and research, although further work on scalability is warranted. ","manuscriptTitle":"Enhancing Substance Use Detection in Clinical Notes with Large Language Models","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-05-15 11:12:33","doi":"10.21203/rs.3.rs-6615981/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"8d60b6cd-fdc8-43e3-8778-b84cb9190087","owner":[],"postedDate":"May 15th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":48491649,"name":"Health sciences/Medical research/Epidemiology"},{"id":48491650,"name":"Health sciences/Health care/Public health/Epidemiology"}],"tags":[],"updatedAt":"2025-05-15T11:12:33+00:00","versionOfRecord":[],"versionCreatedAt":"2025-05-15 11:12:33","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-6615981","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6615981","identity":"rs-6615981","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}