Can Large Language Models Replace Coding Specialists? Evaluating GPT Performance in Medical Coding Tasks

Can Large Language Models Replace Coding Specialists? Evaluating GPT Performance in Medical Coding Tasks | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Can Large Language Models Replace Coding Specialists? Evaluating GPT Performance in Medical Coding Tasks Yeli Feng This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5750190/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Purpose: Large language Models (LLM), GPT in particular, have demonstrated near human-level performance in medical domain, from summarizing clinical notes and passing medical licensing examinations, to predictive tasks such as disease diagnoses and treatment recommendations. However, currently there is little research on their efficacy for medical coding, a pivotal component in health informatics, clinical trials, and reimbursement management. This study proposes a prompt framework and investigates its effectiveness in medical coding tasks. Methods: First, a medical coding prompt framework is proposed. This framework aims to improve the performance of complex coding tasks by leveraging state-of-the-art (SOTA) prompt techniques including meta prompt, multi-shot learning, and dynamic in-context learning to extract task specific knowledge. This framework is implemented with a combination of commercial GPT-4o and open-source LLM. Then its effectiveness is evaluated with three different coding tasks. Finally, ablation studies are presented to validate and analyze the contribution of each module in the proposed prompt framework. Results: On the MIMIC-IV dataset, the prediction accuracy is 68.1% over the 30 most frequent MS-DRG codes. The result is comparable to SOTA 69.4% that fine-tunes the open-source LLaMA model, to the best of our knowledge. And the top-5 accuracy is 90.0%. The clinical trial criteria coding task results in a macro F1 score of 68.4 on the CHIP-CTC test dataset in Chinese, close to 70.9, the best supervised model training method in comparison. For the less complex semantic coding task, our method results in a macro F1 score of 79.7 on the CHIP-STS test dataset in Chinese, which is not competitive with most supervised model training methods in comparison. Conclusion: This study demonstrates that for complex medical coding tasks, carefully designed prompt-based learning can achieve similar performance as SOTA supervised model training approaches. Currently, it can be very helpful assistants, but it does not replace human coding specialists. With the rapid advancement of LLM, their potential to reliably automate complex medical coding in the near future cannot be underestimated. many-shot learning in-context learning meta prompt GPT-4o medical coding clinical trials diagnostic-related groups Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5750190","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":398958836,"identity":"8d072746-dafa-4ff5-9bfe-2f8c8668d87a","order_by":0,"name":"Yeli Feng","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABJElEQVRIie2QsUrEQBBA9zjQZrm0k8bzE9YmXBH8D7tZAqmiBgSxOnJYpBB7xeJ+4Q7Beo8F08R+IQcqh9YBQdLp7HGIQoytxb5iZhj2MTPLmMPxL+lnioWUtzMKKXgMKD8x1aH0SIkpc/tIgJ9ZBbFbYWsF0Crsb0UUk4liGB4ND1Yvt6kYgajOF7VslszLL7BVKRe0GMYne1UcVFcCQCzvI0B8ZVA+zFoVIzPVa7Sc32BQcQFjYRJBiqa7DtuVx2c75UPOr4t3q4Awx29Np2Lsj6GSU+DBRkm2Oqf4JS2GGMkZT07Xt/gmDkYYa/7bLYNC67rGfTnNi7sqPRvDwEQrU4d6x8svW5VdRQHthlT0vzoEb3tuGWbfiv6PjsPhcDg2fAL1IHP2bx0ixgAAAABJRU5ErkJggg==","orcid":"","institution":"Amplify Health Asia","correspondingAuthor":true,"prefix":"","firstName":"Yeli","middleName":"","lastName":"Feng","suffix":""}],"badges":[],"createdAt":"2025-01-02 08:38:28","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5750190/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5750190/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":73709073,"identity":"d6af66f0-5e31-4d05-ae8c-f0ed8d1aaba9","added_by":"auto","created_at":"2025-01-13 20:01:24","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1078351,"visible":true,"origin":"","legend":"","description":"","filename":"JHIRsubmission20250107.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5750190/v1_covered_181fa1ca-a7ab-4121-abc3-a3b4e5c687bd.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Can Large Language Models Replace Coding Specialists? Evaluating GPT Performance in Medical Coding Tasks","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"many-shot learning, in-context learning, meta prompt, GPT-4o, medical coding, clinical trials, diagnostic-related groups","lastPublishedDoi":"10.21203/rs.3.rs-5750190/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5750190/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003ePurpose: \u003c/strong\u003eLarge language Models (LLM), GPT in particular, have demonstrated near human-level performance in medical domain, from summarizing clinical notes and passing medical licensing examinations, to predictive tasks such as disease diagnoses and treatment recommendations. However, currently there is little research on their efficacy for medical coding, a pivotal component in health informatics, clinical trials, and reimbursement management. This study proposes a prompt framework and investigates its effectiveness in medical coding tasks.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMethods: \u003c/strong\u003eFirst, a medical coding prompt framework is proposed. This framework aims to improve the performance of complex coding tasks by leveraging state-of-the-art (SOTA) prompt techniques including meta prompt, multi-shot learning, and dynamic in-context learning to extract task specific knowledge. This framework is implemented with a combination of commercial GPT-4o and open-source LLM. Then its effectiveness is evaluated with three different coding tasks. Finally, ablation studies are presented to validate and analyze the contribution of each module in the proposed prompt framework.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResults: \u003c/strong\u003eOn the MIMIC-IV dataset, the prediction accuracy is 68.1% over the 30 most frequent MS-DRG codes. The result is comparable to SOTA 69.4% that fine-tunes the open-source LLaMA model, to the best of our knowledge. And the top-5 accuracy is 90.0%. The clinical trial criteria coding task results in a macro F1 score of 68.4 on the CHIP-CTC test dataset in Chinese, close to 70.9, the best supervised model training method in comparison. For the less complex semantic coding task, our method results in a macro F1 score of 79.7 on the CHIP-STS test dataset in Chinese, which is not competitive with most supervised model training methods in comparison.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConclusion: \u003c/strong\u003eThis study demonstrates that for complex medical coding tasks, carefully designed prompt-based learning can achieve similar performance as SOTA supervised model training approaches. Currently, it can be very helpful assistants, but it does not replace human coding specialists. With the rapid advancement of LLM, their potential to reliably automate complex medical coding in the near future cannot be underestimated.\u003c/p\u003e","manuscriptTitle":"Can Large Language Models Replace Coding Specialists? Evaluating GPT Performance in Medical Coding Tasks","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-01-08 04:41:48","doi":"10.21203/rs.3.rs-5750190/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"6d05944d-4dfd-4a98-8a30-7d45747da357","owner":[],"postedDate":"January 8th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-01-13T19:53:15+00:00","versionOfRecord":[],"versionCreatedAt":"2025-01-08 04:41:48","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5750190","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5750190","identity":"rs-5750190","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}