A Comparative Investigation of Zero-shot Prompting and Fine-tuning for Clinical Note Summarization

A Comparative Investigation of Zero-shot Prompting and Fine-tuning for Clinical Note Summarization | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article A Comparative Investigation of Zero-shot Prompting and Fine-tuning for Clinical Note Summarization Abir Naskar, Jane S Hocking, Patty Chondros, Douglas Boyle, Mike Conway This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9008079/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 11 You are reading this latest preprint version Abstract Background This study aims to use large language models (LLMs) to develop and evaluate techniques for automated discharge summary generation, with the long term goal of reducing clinician documentation burden, improving workflow efficiency, and enhancing the accuracy and completeness of patient records. Methods We used structured and unstructured inputs from MIMIC-IV—including chief complaints, ICD-9/10 codes, radiology reports, and non-target discharge summary sections, to generate two targets: BHC and DI. Reference sections were obtained from the Discharge Me! shared task. We evaluated instruction-tuned LLMs (Phi-4, LLaMA-3.1-8B, Mistral-7B, Gemma-2-9B, and Gemma-3-12B) under supervised fine-tuning and zero-shot prompting under full and reduced-input settings. Performance was measured using eight metrics (BLEU-4, ROUGE-1/2/L, BERTScore, METEOR, AlignScore, and MEDCON), with additional analyses of input truncation and output length effects. Results Fine-tuned models with full input outperformed others across all evaluation settings. Gemma-2 achieved the highest overall score (0.307), closely followed by LLaMA-3.1 (0.306). zero-shot models performed substantially worse than their fine-tuned counterparts, with the highest zero-shot score (0.21) obtained by Gemma-3 using both full and truncated inputs. Truncating the input reduced the average context length by approximately 50% while yielding competitive performance, resulting in less than a 2% degradation under fine-tuning and nearly identical performance in the zero-shot setting. Analysis of generation length revealed that performance declined beyond a certain character threshold. Conclusion Fine-tuning large language models with full input outperformed other approaches. Input truncation reduced context length and computational cost with minimal impact on generation quality. We observed occasional generation artifacts, such as repeated phrases in fine-tuned outputs. Restricting our analysis to instruction-tuned models (<= 14B parameters), we observed competitive performance across experimental settings under comparable hyperparameter configurations. Discharge Summary Generation Finetuning LLMs Comprehensive Uniqueness Score (CUS) Full Text Additional Declarations Competing interest reported. One of the co-authors, Dr. Mike Conway, serves as a Senior Editorial Board Member of the journal. To avoid any potential conflict of interest, we respectfully request that the manuscript be handled by an independent editor. Supplementary Files supplementary.pdf Cite Share Download PDF Status: Under Review Version 1 posted Reviewers agreed at journal 18 May, 2026 Reviewers agreed at journal 18 May, 2026 Reviews received at journal 20 Apr, 2026 Reviewers agreed at journal 14 Apr, 2026 Reviewers agreed at journal 08 Apr, 2026 Reviewers agreed at journal 26 Mar, 2026 Reviewers invited by journal 11 Mar, 2026 Editor invited by journal 05 Mar, 2026 Editor assigned by journal 03 Mar, 2026 Submission checks completed at journal 03 Mar, 2026 First submitted to journal 02 Mar, 2026 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9008079","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":604434068,"identity":"7691e356-a295-4d37-8fb3-df03cac51582","order_by":0,"name":"Abir Naskar","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABKklEQVRIie3QMUvDQBTA8ReEuFzNmqAkX+GFQKGTX+WKYJYUCoXi4FAInkvq3OLQr5Dg4lgJZCrOVxqkImTqYBDEQitexFJKQrs63H96b/jdgwOQyf5pSg/ALAZ9u7fHYqB7iQPqL8G/HQ+TZm9LivYQ7d5/ywfXL+5oFNg8f1wBzvyb8B1T0I49hCUrET1N6kaYdFphQpzGcIKA6RObDjADI1ig0i8T4LSuzFXaClVSP60xQXiTzQjGYhBXamVicfcjn39T12KCrDdkJci5IMq6TIqnjIhRCokgyoZAcUX38Kjiis29rjG8o3aYXHYafeYQQ5BpgBnRJ1k7PnsuEZO7D3nwSS3LjyO+ZKZ5wt2Mf12lpnZ7Eb0uupUfvRPZGcaHgUwmk8kq+gEVw3P53jWbTQAAAABJRU5ErkJggg==","orcid":"","institution":"University of Melbourne","correspondingAuthor":true,"prefix":"","firstName":"Abir","middleName":"","lastName":"Naskar","suffix":""},{"id":604434069,"identity":"34ad5a5d-1192-4227-903a-ecb2d5bf60db","order_by":1,"name":"Jane S Hocking","email":"","orcid":"","institution":"University of Melbourne","correspondingAuthor":false,"prefix":"","firstName":"Jane","middleName":"S","lastName":"Hocking","suffix":""},{"id":604434070,"identity":"ad48b168-457b-4be3-a791-5470623e3d4a","order_by":2,"name":"Patty Chondros","email":"","orcid":"","institution":"University of Melbourne","correspondingAuthor":false,"prefix":"","firstName":"Patty","middleName":"","lastName":"Chondros","suffix":""},{"id":604434071,"identity":"6b503f89-4a09-41f1-b01a-1d2c516000cc","order_by":3,"name":"Douglas Boyle","email":"","orcid":"","institution":"University of Melbourne","correspondingAuthor":false,"prefix":"","firstName":"Douglas","middleName":"","lastName":"Boyle","suffix":""},{"id":604434072,"identity":"ce297dda-6ad6-4528-98ff-1e6a5c0770f0","order_by":4,"name":"Mike Conway","email":"","orcid":"","institution":"University of Melbourne","correspondingAuthor":false,"prefix":"","firstName":"Mike","middleName":"","lastName":"Conway","suffix":""}],"badges":[],"createdAt":"2026-03-02 09:10:42","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9008079/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9008079/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":104781188,"identity":"d0997e52-962c-457f-b5a2-428a40adf6be","added_by":"auto","created_at":"2026-03-17 07:55:05","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1825077,"visible":true,"origin":"","legend":"","description":"","filename":"latex.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9008079/v1_covered_420488c5-ef69-4281-9379-1b916baab139.pdf"},{"id":104536529,"identity":"2bc421b9-7ee2-4a19-bc09-0fce91c46cc8","added_by":"auto","created_at":"2026-03-13 04:02:30","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":2062049,"visible":true,"origin":"","legend":"","description":"","filename":"supplementary.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9008079/v1/345c9b95d9575f897c44c831.pdf"}],"financialInterests":"Competing interest reported. One of the co-authors, Dr. Mike Conway, serves as a Senior Editorial Board Member of the journal. To avoid any potential conflict of interest, we respectfully request that the manuscript be handled by an independent editor.","formattedTitle":"A Comparative Investigation of Zero-shot Prompting and Fine-tuning for Clinical Note Summarization","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"bmc-medical-informatics-and-decision-making","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"midm","sideBox":"Learn more about [BMC Medical Informatics and Decision Making](http://bmcmedinformdecismak.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/midm/default.aspx","title":"BMC Medical Informatics and Decision Making","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Discharge Summary Generation, Finetuning LLMs, Comprehensive Uniqueness Score (CUS)","lastPublishedDoi":"10.21203/rs.3.rs-9008079/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9008079/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"Background\nThis study aims to use large language models (LLMs) to develop and evaluate techniques for automated discharge summary generation, with the long term goal of reducing clinician documentation burden, improving workflow efficiency, and enhancing the accuracy and completeness of patient records. \n\nMethods\nWe used structured and unstructured inputs from MIMIC-IV—including chief complaints, ICD-9/10 codes, radiology reports, and non-target discharge summary sections, to generate two targets: BHC and DI. Reference sections were obtained from the Discharge Me! shared task. We evaluated instruction-tuned LLMs (Phi-4, LLaMA-3.1-8B, Mistral-7B, Gemma-2-9B, and Gemma-3-12B) under supervised fine-tuning and zero-shot prompting under full and reduced-input settings. Performance was measured using eight metrics (BLEU-4, ROUGE-1/2/L, BERTScore, METEOR, AlignScore, and MEDCON), with additional analyses of input truncation and output length effects.\n\nResults\nFine-tuned models with full input outperformed others across all evaluation settings. Gemma-2 achieved the highest overall score (0.307), closely followed by LLaMA-3.1 (0.306). zero-shot models performed substantially worse than their fine-tuned counterparts, with the highest zero-shot score (0.21) obtained by Gemma-3 using both full and truncated inputs. Truncating the input reduced the average context length by approximately 50% while yielding competitive performance, resulting in less than a 2% degradation under fine-tuning and nearly identical performance in the zero-shot setting. Analysis of generation length revealed that performance declined beyond a certain character threshold.\n\nConclusion\nFine-tuning large language models with full input outperformed other approaches. Input truncation reduced context length and computational cost with minimal impact on generation quality. We observed occasional generation artifacts, such as repeated phrases in fine-tuned outputs. Restricting our analysis to instruction-tuned models (\u003c= 14B parameters), we observed competitive performance across experimental settings under comparable hyperparameter configurations.","manuscriptTitle":"A Comparative Investigation of Zero-shot Prompting and Fine-tuning for Clinical Note Summarization","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-03-13 04:02:25","doi":"10.21203/rs.3.rs-9008079/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"reviewerAgreed","content":"94631405724349774503578016909923022899","date":"2026-05-18T14:35:20+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"219445817202557105084713483356597735066","date":"2026-05-18T08:57:09+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2026-04-20T06:39:46+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"23208941312961071802706230837126756391","date":"2026-04-14T08:04:56+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"44478542888716540709627845118476827933","date":"2026-04-08T08:06:12+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"41504057816126050759426555694542193942","date":"2026-03-26T08:34:46+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2026-03-11T11:27:27+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2026-03-05T07:52:36+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-03-04T04:46:08+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2026-03-04T04:45:32+00:00","index":"","fulltext":""},{"type":"submitted","content":"BMC Medical Informatics and Decision Making","date":"2026-03-02T09:00:57+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"bmc-medical-informatics-and-decision-making","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"midm","sideBox":"Learn more about [BMC Medical Informatics and Decision Making](http://bmcmedinformdecismak.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/midm/default.aspx","title":"BMC Medical Informatics and Decision Making","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"0ffa72cf-289d-453b-a6d1-c7a7554c9d95","owner":[],"postedDate":"March 13th, 2026","published":true,"recentEditorialEvents":[{"type":"reviewerAgreed","content":"94631405724349774503578016909923022899","date":"2026-05-18T14:35:20+00:00","index":72,"fulltext":""},{"type":"reviewerAgreed","content":"219445817202557105084713483356597735066","date":"2026-05-18T08:57:09+00:00","index":71,"fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2026-03-13T04:02:25+00:00","versionOfRecord":[],"versionCreatedAt":"2026-03-13 04:02:25","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9008079","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9008079","identity":"rs-9008079","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}