Evaluation of ChatGPT and Gemini Large Language Models for Pharmacometrics with NONMEM

Evaluation of ChatGPT and Gemini Large Language Models for Pharmacometrics with NONMEM | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Evaluation of ChatGPT and Gemini Large Language Models for Pharmacometrics with NONMEM Euibeom Shin, Yifan Yu, Robert R. Bies, Murali Ramanathan This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4189234/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 24 Apr, 2024 Read the published version in Journal of Pharmacokinetics and Pharmacodynamics → Version 1 posted 7 You are reading this latest preprint version Abstract Purpose To assess the ChatGPT 4.0 (ChatGPT) and Gemini Ultra 1.0 (Gemini) large language models on tasks relevant to NONMEM coding in pharmacometrics and clinical pharmacology settings. Methods ChatGPT and Gemini performance on tasks mimicking real-world applications of NONMEM was assessed. The tasks ranged from providing a curriculum for learning NONMEM and an overview of NONMEM code structure to generating code. Prompts to elicit NONMEM code for a linear pharmacokinetic (PK) model with oral administration and a more complex one-compartment model with two parallel first-order absorption mechanisms were investigated. The prompts for all tasks were presented in lay language. The code was carefully reviewed for errors by two experienced NONMEM experts, and the revisions needed to run the code successfully were identified. Results ChatGPT and Gemini provided useful NONMEM curriculum structures combining foundational knowledge with advanced concepts (e.g., covariate modeling and Bayesian approaches) and practical skills, including NONMEM code structure and syntax. Large language models (LLMs) provided an informative summary of the NONMEM control stream structure and outlined the key NM-TRAN records needed. ChatGPT and Gemini were able to generate applicable code blocks for the NONMEM control stream from the lay language prompts for the three coding tasks. The control streams contained focal structural and NONMEM syntax errors that required revision before they could be executed without errors and warnings. Conclusions LLMs may be useful in pharmacometrics for efficiently generating an initial coding template for modeling projects. However, the output can contain errors that require correction. Pharmacometrics ChatGPT Pharmacokinetics Drug Development Artificial intelligence Generative AI Modeling Nonlinear mixed effects NONMEM Full Text Additional Declarations No competing interests reported. Supplementary Files NONMEMGPTSupplementaryFile.docx Cite Share Download PDF Status: Published Journal Publication published 24 Apr, 2024 Read the published version in Journal of Pharmacokinetics and Pharmacodynamics → Version 1 posted Editorial decision: Revision requested 08 Apr, 2024 Reviews received at journal 08 Apr, 2024 Reviewers agreed at journal 01 Apr, 2024 Reviewers invited by journal 01 Apr, 2024 Submission checks completed at journal 29 Mar, 2024 Editor assigned by journal 29 Mar, 2024 First submitted to journal 29 Mar, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4189234","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":286251762,"identity":"06a8756d-03b2-4bab-9fbf-d1dfa592dca6","order_by":0,"name":"Euibeom Shin","email":"","orcid":"","institution":"University at Buffalo, The State University of New York","correspondingAuthor":false,"prefix":"","firstName":"Euibeom","middleName":"","lastName":"Shin","suffix":""},{"id":286251763,"identity":"a4be0dea-ddfc-4c3f-9f06-ab27924701d5","order_by":1,"name":"Yifan Yu","email":"","orcid":"","institution":"University at Buffalo, The State University of New York","correspondingAuthor":false,"prefix":"","firstName":"Yifan","middleName":"","lastName":"Yu","suffix":""},{"id":286251764,"identity":"64286257-3fba-482a-b344-35bfb197b7df","order_by":2,"name":"Robert R. Bies","email":"","orcid":"","institution":"University at Buffalo, The State University of New York","correspondingAuthor":false,"prefix":"","firstName":"Robert","middleName":"R.","lastName":"Bies","suffix":""},{"id":286251765,"identity":"c32c1628-55b7-482e-8a9d-1ddbb7fbcdd3","order_by":3,"name":"Murali Ramanathan","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAx0lEQVRIie3PsQrCMBCA4ZNAxs7pWxwIUkH0Va4E4iQ4dkqdXH0VH6Ei2CXoGnAwLs4ZOziYOoqQujnkH0KG+7gEIJX6w7AJh8OZBnZw4ABqaCJk2g8QKglcIlC4R8niTeBYbgaTiV3dPa0vEk9s5KkCyizFyHIsCK86ECbIgM7jREFPJN52jSi34V8xglaxjvBc7sOWbijhYUvTEz5wi3nwglDKPJCCjKDcuAhpFbP+OddZeJj11azO2siWz8Rv46lUKpX63gu2iUzBJbcALgAAAABJRU5ErkJggg==","orcid":"","institution":"University at Buffalo, The State University of New York","correspondingAuthor":true,"prefix":"","firstName":"Murali","middleName":"","lastName":"Ramanathan","suffix":""}],"badges":[],"createdAt":"2024-03-29 17:44:19","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4189234/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4189234/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1007/s10928-024-09921-y","type":"published","date":"2024-04-24T23:27:32+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":55695037,"identity":"61e56598-fca6-431b-af75-c9b857999129","added_by":"auto","created_at":"2024-05-02 00:56:07","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1278223,"visible":true,"origin":"","legend":"","description":"","filename":"NONMEMGPTMain.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4189234/v1_covered_0f33e974-d048-4f3a-b43d-7475ccae12b0.pdf"},{"id":53995744,"identity":"fbe57da0-9450-4b7c-92b4-16f98ec313ed","added_by":"auto","created_at":"2024-04-03 07:12:56","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":10860733,"visible":true,"origin":"","legend":"","description":"","filename":"NONMEMGPTSupplementaryFile.docx","url":"https://assets-eu.researchsquare.com/files/rs-4189234/v1/b6af2e1385aa7b71bb240102.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Evaluation of ChatGPT and Gemini Large Language Models for Pharmacometrics with NONMEM","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"journal-of-pharmacokinetics-and-pharmacodynamics","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"jopa","sideBox":"Learn more about [Journal of Pharmacokinetics and Pharmacodynamics](http://link.springer.com/journal/10928)","snPcode":"10928","submissionUrl":"https://submission.nature.com/new-submission/10928/3","title":"Journal of Pharmacokinetics and Pharmacodynamics","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Pharmacometrics, ChatGPT, Pharmacokinetics, Drug Development, Artificial intelligence, Generative AI, Modeling, Nonlinear mixed effects, NONMEM","lastPublishedDoi":"10.21203/rs.3.rs-4189234/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4189234/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003ePurpose\u003c/h2\u003e \u003cp\u003eTo assess the ChatGPT 4.0 (ChatGPT) and Gemini Ultra 1.0 (Gemini) large language models on tasks relevant to NONMEM coding in pharmacometrics and clinical pharmacology settings.\u003c/p\u003e\u003ch2\u003eMethods\u003c/h2\u003e \u003cp\u003eChatGPT and Gemini performance on tasks mimicking real-world applications of NONMEM was assessed. The tasks ranged from providing a curriculum for learning NONMEM and an overview of NONMEM code structure to generating code. Prompts to elicit NONMEM code for a linear pharmacokinetic (PK) model with oral administration and a more complex one-compartment model with two parallel first-order absorption mechanisms were investigated. The prompts for all tasks were presented in lay language. The code was carefully reviewed for errors by two experienced NONMEM experts, and the revisions needed to run the code successfully were identified.\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e \u003cp\u003eChatGPT and Gemini provided useful NONMEM curriculum structures combining foundational knowledge with advanced concepts (e.g., covariate modeling and Bayesian approaches) and practical skills, including NONMEM code structure and syntax. Large language models (LLMs) provided an informative summary of the NONMEM control stream structure and outlined the key NM-TRAN records needed. ChatGPT and Gemini were able to generate applicable code blocks for the NONMEM control stream from the lay language prompts for the three coding tasks. The control streams contained focal structural and NONMEM syntax errors that required revision before they could be executed without errors and warnings.\u003c/p\u003e\u003ch2\u003eConclusions\u003c/h2\u003e \u003cp\u003eLLMs may be useful in pharmacometrics for efficiently generating an initial coding template for modeling projects. However, the output can contain errors that require correction.\u003c/p\u003e","manuscriptTitle":"Evaluation of ChatGPT and Gemini Large Language Models for Pharmacometrics with NONMEM","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-04-03 07:12:51","doi":"10.21203/rs.3.rs-4189234/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2024-04-08T19:27:49+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-04-08T19:21:45+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"7432e0c1-5bfd-4e78-80c7-59444d3a21eb","date":"2024-04-01T16:02:31+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-04-01T15:54:44+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-03-30T03:37:14+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-03-30T03:37:14+00:00","index":"","fulltext":""},{"type":"submitted","content":"Journal of Pharmacokinetics and Pharmacodynamics","date":"2024-03-29T17:42:09+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"journal-of-pharmacokinetics-and-pharmacodynamics","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"jopa","sideBox":"Learn more about [Journal of Pharmacokinetics and Pharmacodynamics](http://link.springer.com/journal/10928)","snPcode":"10928","submissionUrl":"https://submission.nature.com/new-submission/10928/3","title":"Journal of Pharmacokinetics and Pharmacodynamics","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"cb446adb-8019-4de2-bfd9-68fe1fa784bf","owner":[],"postedDate":"April 3rd, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2024-05-01T23:27:32+00:00","versionOfRecord":{"articleIdentity":"rs-4189234","link":"https://doi.org/10.1007/s10928-024-09921-y","journal":{"identity":"journal-of-pharmacokinetics-and-pharmacodynamics","isVorOnly":false,"title":"Journal of Pharmacokinetics and Pharmacodynamics"},"publishedOn":"2024-04-24 23:27:32","publishedOnDateReadable":"April 24th, 2024"},"versionCreatedAt":"2024-04-03 07:12:51","video":"","vorDoi":"10.1007/s10928-024-09921-y","vorDoiUrl":"https://doi.org/10.1007/s10928-024-09921-y","workflowStages":[]},"version":"v1","identity":"rs-4189234","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4189234","identity":"rs-4189234","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}