Evaluating a Customised Large Language Model (DELSTAR) and its Ability to Address Medication-Related Questions Associated with Delirium: A Quantitative Exploratory Study

Evaluating a Customised Large Language Model (DELSTAR) and its Ability to Address Medication-Related Questions Associated with Delirium: A Quantitative Exploratory Study | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Evaluating a Customised Large Language Model (DELSTAR) and its Ability to Address Medication-Related Questions Associated with Delirium: A Quantitative Exploratory Study Katharina Teresa Spagl, Edward William Watson, Adam Jatowt, Anita Elaine Weidmann This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5707431/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 10 Apr, 2025 Read the published version in International Journal of Clinical Pharmacy → Version 1 posted 6 You are reading this latest preprint version Abstract Background : A customised large language model (LLM) could serve as a next-generation clinical pharmacy research assistant to prevent medication-associated delirium. Comprehensive evaluation strategies are still missing. Aim : This quantitative exploratory study aims to develop an approach to comprehensively assess the domain-specific customised delirium LLM (DELSTAR) ability, quality and performance to accurately address complex clinical and practice research questions on delirium that typically require extensive literature searches and meta-analyses. Method : DELSTAR, focused on delirium-associated medications, was implemented as a 'Custom GPT' for quality assessment and as a Python-based software pipeline for performance testing on closed and leading open models. Quality metrics included statement accuracy and data credibility; performance metrics covered F1-Score, sensitivity/specificity, precision, AUC, and AUC-ROC curves. Results : DELSTAR demonstrated more precise information compared to information retrieved by traditional systematic literature reviews (SLRs) (p<0.05) and accessed Application Programmer Interfaces (API), private databases, and high-quality sources despite mainly relying on less reliable internet sources. GPT-3.5 and GPT-4o emerged as the most reliable foundation models. In Dataset 2, GPT-4o (F1-Score: 0.687) and Llama3-70b (F1-Score: 0.655) performed best, while in Dataset 3, GPT-3.5 (F1-Score: 0.708) and GPT-4o (F1-Score: 0.665) led. None consistently met desired threshold values across all metrics. Conclusion : DELSTAR demonstrated potential as a clinical pharmacy research assistant, surpassing traditional SLRs in quality. Improvements are needed in high-quality data use, citation, and performance optimisation. GPT-4o, GPT-3.5, and Llama3-70b were the most suitable foundation models, but fine-tuning DELSTAR is essential to enhance sensitivity, especially critical in pharmaceutical contexts. intelligence artificial intelligence machine clinical pharmacy information systems delirium drug prescribing patient safety Full Text Supplementary Files AppendixALLMDELSTAR.pdf Cite Share Download PDF Status: Published Journal Publication published 10 Apr, 2025 Read the published version in International Journal of Clinical Pharmacy → Version 1 posted Editorial decision: Minor revisions 23 Feb, 2025 Reviewers agreed at journal 09 Jan, 2025 Reviewers invited by journal 08 Jan, 2025 Editor invited by journal 05 Jan, 2025 Editor assigned by journal 25 Dec, 2024 First submitted to journal 24 Dec, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5707431","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":399571901,"identity":"b2aa53a2-5c7c-4620-8b42-8600853a304f","order_by":0,"name":"Katharina Teresa Spagl","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABG0lEQVRIiWNgGAWjYBACCQYGNggFARZyQILxAANDAh4tzChaJIwZGJgZiNGC4Cc2ENIi2X7+2IOfeyzy+BvYnz348UsivV8i/8CBDxVpDPzt3Vj1SfMksxv2PJMoljjAY27Y2yeRO3NGMsPBGWdyGCTOnN2ATYscQzKbBM8BoHsO8LBJ8PZI5G64kcxwmLetgsEAyMaqhf8xm+QfoJb5B9ifSf7tkUg3IKRFWiKZTRpky4YDDGbSPD8kEqBacnBqkZzx2ExaBqhl42Eec2PZBgnDmT2PDYB+SePB5ReJ84nPJN8cqEucd7z92cM3f2zk+dkTHz74UJEsx9/ei1ULAjADkwFjG4LPg185BADj9A8x6kbBKBgFo2CkAQARvWE92CMmcQAAAABJRU5ErkJggg==","orcid":"https://orcid.org/0009-0005-7426-3684","institution":"University of Innsbruck: Universitat Innsbruck","correspondingAuthor":true,"prefix":"","firstName":"Katharina","middleName":"Teresa","lastName":"Spagl","suffix":""},{"id":399571902,"identity":"4a2ac0b8-4d13-4b37-93d0-35ef3892b7bd","order_by":1,"name":"Edward William Watson","email":"","orcid":"","institution":"University of Innsbruck: Universitat Innsbruck","correspondingAuthor":false,"prefix":"","firstName":"Edward","middleName":"William","lastName":"Watson","suffix":""},{"id":399571903,"identity":"93d57b30-3a5e-46e6-a49a-fb5f2d10df39","order_by":2,"name":"Adam Jatowt","email":"","orcid":"","institution":"University of Innsbruck: Universitat Innsbruck","correspondingAuthor":false,"prefix":"","firstName":"Adam","middleName":"","lastName":"Jatowt","suffix":""},{"id":399571904,"identity":"bed1f458-364b-4f1d-ada7-89647259742f","order_by":3,"name":"Anita Elaine Weidmann","email":"","orcid":"https://orcid.org/0000-0003-3670-2357","institution":"University of Innsbruck: Universitat Innsbruck","correspondingAuthor":false,"prefix":"","firstName":"Anita","middleName":"Elaine","lastName":"Weidmann","suffix":""}],"badges":[],"createdAt":"2024-12-24 16:43:32","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5707431/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5707431/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1007/s11096-025-01900-8","type":"published","date":"2025-04-10T16:05:05+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":80558995,"identity":"a2091b7c-e2e9-47e1-9dae-772907cb4e78","added_by":"auto","created_at":"2025-04-14 16:17:26","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":558404,"visible":true,"origin":"","legend":"","description":"","filename":"AbstractMainTextFiguresTablesLLMDELSTAR.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5707431/v1_covered_c4f2f724-aa7f-4a77-a91c-babf40e6cd7f.pdf"},{"id":73438089,"identity":"5abb993b-999e-4b69-bdf3-875d0ba63806","added_by":"auto","created_at":"2025-01-10 02:55:01","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":27491,"visible":true,"origin":"","legend":"","description":"","filename":"AppendixALLMDELSTAR.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5707431/v1/1c58a864d5933262322cb072.pdf"}],"financialInterests":"","formattedTitle":"Evaluating a Customised Large Language Model (DELSTAR) and its Ability to Address Medication-Related Questions Associated with Delirium: A Quantitative Exploratory Study","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"international-journal-of-clinical-pharmacy","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"ijcp","sideBox":"Learn more about [International Journal of Clinical Pharmacy](https://www.springer.com/journal/11096)","snPcode":"11096","submissionUrl":"https://submission.nature.com/new-submission/11096/3","title":"International Journal of Clinical Pharmacy","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"intelligence, artificial, intelligence, machine, clinical pharmacy information systems, delirium, drug prescribing, patient safety","lastPublishedDoi":"10.21203/rs.3.rs-5707431/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5707431/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003eBackground\u003c/strong\u003e: A customised large language model (LLM) could serve as a next-generation clinical pharmacy research assistant to prevent medication-associated delirium. Comprehensive evaluation strategies are still missing.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAim\u003c/strong\u003e: This quantitative exploratory study aims to develop an approach to comprehensively assess the domain-specific customised delirium LLM (DELSTAR) ability, quality and performance to accurately address complex clinical and practice research questions on delirium that typically require extensive literature searches and meta-analyses.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMethod\u003c/strong\u003e: DELSTAR, focused on delirium-associated medications, was implemented as a 'Custom GPT' for quality assessment and as a Python-based software pipeline for performance testing on closed and leading open models. Quality metrics included statement accuracy and data credibility; performance metrics covered F1-Score, sensitivity/specificity, precision, AUC, and AUC-ROC curves.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResults\u003c/strong\u003e: DELSTAR demonstrated more precise information compared to information retrieved by traditional systematic literature reviews (SLRs) (p\u0026lt;0.05) and accessed Application Programmer Interfaces (API), private databases, and high-quality sources despite mainly relying on less reliable internet sources. GPT-3.5 and GPT-4o emerged as the most reliable foundation models. In Dataset 2, GPT-4o (F1-Score: 0.687) and Llama3-70b (F1-Score: 0.655) performed best, while in Dataset 3, GPT-3.5 (F1-Score: 0.708) and GPT-4o (F1-Score: 0.665) led. None consistently met desired threshold values across all metrics.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConclusion\u003c/strong\u003e: DELSTAR demonstrated potential as a clinical pharmacy research assistant, surpassing traditional SLRs in quality. Improvements are needed in high-quality data use, citation, and performance optimisation. GPT-4o, GPT-3.5, and Llama3-70b were the most suitable foundation models, but fine-tuning DELSTAR is essential to enhance sensitivity, especially critical in pharmaceutical contexts.\u003c/p\u003e","manuscriptTitle":"Evaluating a Customised Large Language Model (DELSTAR) and its Ability to Address Medication-Related Questions Associated with Delirium: A Quantitative Exploratory Study","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-01-10 02:54:57","doi":"10.21203/rs.3.rs-5707431/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Minor revisions","date":"2025-02-23T06:09:26+00:00","index":"","fulltext":""},{"type":"reviewerAgreed","content":"","date":"2025-01-09T10:21:54+00:00","index":0,"fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-01-08T17:56:11+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"International Journal of Clinical Pharmacy","date":"2025-01-05T12:57:32+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-12-26T03:55:28+00:00","index":"","fulltext":""},{"type":"submitted","content":"International Journal of Clinical Pharmacy","date":"2024-12-24T11:41:38+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"international-journal-of-clinical-pharmacy","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"ijcp","sideBox":"Learn more about [International Journal of Clinical Pharmacy](https://www.springer.com/journal/11096)","snPcode":"11096","submissionUrl":"https://submission.nature.com/new-submission/11096/3","title":"International Journal of Clinical Pharmacy","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"87a6e3ae-67f1-4273-8236-ce3d5e1cece7","owner":[],"postedDate":"January 10th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2025-04-14T16:14:19+00:00","versionOfRecord":{"articleIdentity":"rs-5707431","link":"https://doi.org/10.1007/s11096-025-01900-8","journal":{"identity":"international-journal-of-clinical-pharmacy","isVorOnly":false,"title":"International Journal of Clinical Pharmacy"},"publishedOn":"2025-04-10 16:05:05","publishedOnDateReadable":"April 10th, 2025"},"versionCreatedAt":"2025-01-10 02:54:57","video":"","vorDoi":"10.1007/s11096-025-01900-8","vorDoiUrl":"https://doi.org/10.1007/s11096-025-01900-8","workflowStages":[]},"version":"v1","identity":"rs-5707431","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5707431","identity":"rs-5707431","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}