Comparative analysis of privacy-preserving open-source LLMs regarding extraction of diagnosticinformation from clinical CMR imaging reports

Comparative analysis of privacy-preserving open-source LLMs regarding extraction of diagnosticinformation from clinical CMR imaging reports | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Comparative analysis of privacy-preserving open-source LLMs regarding extraction of diagnosticinformation from clinical CMR imaging reports Sina Amirrajab, Volker Vehof, Michael Bietenbeck, Ali Yilmaz This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6776028/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 15 You are reading this latest preprint version Abstract We investigated the utilization of privacy-preserving, locally-deployed, open-source Large Language Models (LLMs) to extractdiagnostic information from free-text cardiovascular magnetic resonance (CMR) reports. We evaluated nine open-sourceLLMs on their ability to identify diagnoses and classify patients into various cardiac diagnostic categories based on descriptivefindings in 109 clinical CMR reports. Performance was quantified using standard classification metrics including accuracy,precision, recall, and F1 score. We also employed confusion matrices to examine patterns of misclassification across models.Most open-source LLMs demonstrated exceptional performance in classifying reports into different diagnostic categories.Google’s Gemma 2 model achieved the highest average F1 score of 0.98, followed by Qwen2.5:32B and DeepseekR1-32Bwith F1 scores of 0.96 and 0.95, respectively. All other evaluated models attained average scores above 0.93, with Mistraland DeepseekR1-7B being the only exceptions. The top four LLMs outperformed our board-certified cardiologist (F1 scoreof 0.94) across all evaluation metrics in analyzing CMR reports. Our findings demonstrate the feasibility of implementingopen-source, privacy-preserving LLMs in clinical settings for automated analysis of imaging reports, enabling accurate, fastand resource-efficient diagnostic categorization. Physical sciences/Engineering/Biomedical engineering Physical sciences/Mathematics and computing/Computational science Physical sciences/Mathematics and computing/Computer science Physical sciences/Mathematics and computing/Scientific data Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 15 Apr, 2026 Reviews received at journal 29 Oct, 2025 Reviews received at journal 28 Oct, 2025 Reviews received at journal 27 Oct, 2025 Reviews received at journal 13 Oct, 2025 Reviewers agreed at journal 12 Oct, 2025 Reviewers agreed at journal 12 Oct, 2025 Reviewers agreed at journal 12 Oct, 2025 Reviewers agreed at journal 12 Oct, 2025 Reviewers agreed at journal 10 Oct, 2025 Reviewers invited by journal 10 Oct, 2025 Editor assigned by journal 07 Oct, 2025 Editor invited by journal 09 Jun, 2025 Submission checks completed at journal 06 Jun, 2025 First submitted to journal 06 Jun, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6776028","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":532282575,"identity":"00a4d789-0e82-45fe-9bea-baae166447d6","order_by":0,"name":"Sina Amirrajab","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABKElEQVRIie2QMUvDQBTHLwTS5eJ8kLT5Ci8cJA6iX6UhkKlDIKtgoHBO6iYJAb9DltKxEnAKdRVuCTgLAUFajGLOIlS5UkfB+w33hvf/3Xt3CCkUfxF9U7A4QEuPENlqwm+U6EuB3co3tLTarzjn5m2LTme2X1RNvJ7fB/l17T7GXTfyU71sJApUByFBdxzbywjci5oHhTWhNGdA7YWRyMaAjoEgg2OCEYDJeHBjTTzLTCHIEPaIRHGmmK7Qu1AGrfvGlkLxX3EHZ73ir2SPqfqrNCYUDNRkC7GYp2MDxqSfIjOgVw6Dy08lpsM6pHkRJZbJqJvpRiJd7KqmD+0LP+kXm7lP8+NhxsPyGXcjhwymZSsbIxhvivHjf/Rd+a1Isz+jUCgU/5EPrNFY/f64VuMAAAAASUVORK5CYII=","orcid":"","institution":"Maastricht University","correspondingAuthor":true,"prefix":"","firstName":"Sina","middleName":"","lastName":"Amirrajab","suffix":""},{"id":532282576,"identity":"3e33056b-11c5-4c86-b303-cdb9b21fc2e9","order_by":1,"name":"Volker Vehof","email":"","orcid":"","institution":"University Hospital Münster","correspondingAuthor":false,"prefix":"","firstName":"Volker","middleName":"","lastName":"Vehof","suffix":""},{"id":532282577,"identity":"945e38e8-762a-4fd2-9a1e-93f4f68309a4","order_by":2,"name":"Michael Bietenbeck","email":"","orcid":"","institution":"University Hospital Münster","correspondingAuthor":false,"prefix":"","firstName":"Michael","middleName":"","lastName":"Bietenbeck","suffix":""},{"id":532282578,"identity":"7b6cf177-b322-485f-ac08-528ebc33ccf8","order_by":3,"name":"Ali Yilmaz","email":"","orcid":"","institution":"University Hospital Münster","correspondingAuthor":false,"prefix":"","firstName":"Ali","middleName":"","lastName":"Yilmaz","suffix":""}],"badges":[],"createdAt":"2025-05-29 11:23:12","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6776028/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6776028/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":94402545,"identity":"47fb0f84-efb9-4860-97db-3fa128a8658e","added_by":"auto","created_at":"2025-10-27 13:59:43","extension":"json","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":6054,"visible":true,"origin":"","legend":"","description":"","filename":"f4da1145e68f4962a920400d285cc5f5.json","url":"https://assets-eu.researchsquare.com/files/rs-6776028/v1/c5aa8257765b69d1da1a7269.json"},{"id":94402475,"identity":"109f181c-6ee0-4baf-8548-a5fdea66c8ff","added_by":"auto","created_at":"2025-10-27 13:59:42","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":49783,"visible":true,"origin":"","legend":"","description":"","filename":"ResponsetoEditorialComments.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6776028/v1/4038b326aea0dfc5a56c318e.pdf"},{"id":94402138,"identity":"8b9a3e66-c491-4df6-bcb0-a2a1f92a3075","added_by":"auto","created_at":"2025-10-27 13:59:32","extension":"pdf","order_by":2,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":563112,"visible":true,"origin":"","legend":"","description":"","filename":"czxbdfgwgvrjqnwhskrkvpkpdxnkkcwt.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6776028/v1/ecb4faa4b0b4278cab4efa47.pdf"},{"id":94402526,"identity":"7c94abf5-cc1d-476f-bf90-72c33bc337cb","added_by":"auto","created_at":"2025-10-27 13:59:43","extension":"pdf","order_by":3,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":481901,"visible":true,"origin":"","legend":"","description":"","filename":"figurs.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6776028/v1/45736f7f1131b440f8ed8370.pdf"},{"id":94401151,"identity":"237a7a68-7ef5-46ae-ac2c-d1ec20aaa3fd","added_by":"auto","created_at":"2025-10-27 13:58:53","extension":"ldf","order_by":4,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":49107,"visible":true,"origin":"","legend":"","description":"","filename":"jabbrvltwaall.ldf","url":"https://assets-eu.researchsquare.com/files/rs-6776028/v1/96a7b5585b8065d12cacbe3b.ldf"},{"id":94402406,"identity":"edb73175-d083-4fc7-bf42-c10c5c59a774","added_by":"auto","created_at":"2025-10-27 13:59:38","extension":"ldf","order_by":5,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":268657,"visible":true,"origin":"","legend":"","description":"","filename":"jabbrvltwaen.ldf","url":"https://assets-eu.researchsquare.com/files/rs-6776028/v1/e1f11309d5b59baeec6b1554.ldf"},{"id":94402265,"identity":"4b89d3e4-5c9d-473e-ad37-5670c5791cd2","added_by":"auto","created_at":"2025-10-27 13:59:34","extension":"sty","order_by":6,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":15480,"visible":true,"origin":"","legend":"","description":"","filename":"jabbrv.sty","url":"https://assets-eu.researchsquare.com/files/rs-6776028/v1/1fc88655f71a61d77d4544eb.sty"},{"id":94401153,"identity":"ad9a7224-06e6-4af6-ab5c-385d6ff2484e","added_by":"auto","created_at":"2025-10-27 13:58:53","extension":"bbl","order_by":7,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":5072,"visible":true,"origin":"","legend":"","description":"","filename":"main.bbl","url":"https://assets-eu.researchsquare.com/files/rs-6776028/v1/85f3085b5045ec8a35da119c.bbl"},{"id":94401611,"identity":"0a1c5e9d-fa32-4d3d-ab6e-a12490aeccb5","added_by":"auto","created_at":"2025-10-27 13:59:13","extension":"bst","order_by":8,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":36153,"visible":true,"origin":"","legend":"","description":"","filename":"naturemagdoi.bst","url":"https://assets-eu.researchsquare.com/files/rs-6776028/v1/1da966522a775d003def7a21.bst"},{"id":94401893,"identity":"6430d73c-c1f7-4eb2-b96b-7c1a23d230b9","added_by":"auto","created_at":"2025-10-27 13:59:23","extension":"jpg","order_by":9,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":379752,"visible":true,"origin":"","legend":"","description":"","filename":"stream.jpg","url":"https://assets-eu.researchsquare.com/files/rs-6776028/v1/74fbbf60ac410b240ef190b1.jpg"},{"id":94400852,"identity":"c566cc85-57a4-412d-a39e-ed59fad608b4","added_by":"auto","created_at":"2025-10-27 13:58:31","extension":"cls","order_by":10,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":5824,"visible":true,"origin":"","legend":"","description":"","filename":"wlscirep.cls","url":"https://assets-eu.researchsquare.com/files/rs-6776028/v1/915c89bced4cc7f9f418de30.cls"},{"id":94402725,"identity":"922658e8-5287-4f00-ad1a-79b591e18546","added_by":"auto","created_at":"2025-10-27 13:59:54","extension":"xml","order_by":11,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":23756,"visible":true,"origin":"","legend":"","description":"","filename":"f4da1145e68f4962a920400d285cc5f51structuring.xml","url":"https://assets-eu.researchsquare.com/files/rs-6776028/v1/e7cc8d10383358286febe805.xml"},{"id":94490705,"identity":"2a60cc6b-3efb-401d-8312-abf1edb4f58d","added_by":"auto","created_at":"2025-10-27 17:14:01","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":657801,"visible":true,"origin":"","legend":"","description":"","filename":"czxbdfgwgvrjqnwhskrkvpkpdxnkkcwt.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6776028/v1_covered_9c198dbc-a45d-4a89-9224-8f8e9b3c40a4.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Comparative analysis of privacy-preserving open-source LLMs regarding extraction of diagnosticinformation from clinical CMR imaging reports","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-6776028/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6776028/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"We investigated the utilization of privacy-preserving, locally-deployed, open-source Large Language Models (LLMs) to extractdiagnostic information from free-text cardiovascular magnetic resonance (CMR) reports. We evaluated nine open-sourceLLMs on their ability to identify diagnoses and classify patients into various cardiac diagnostic categories based on descriptivefindings in 109 clinical CMR reports. Performance was quantified using standard classification metrics including accuracy,precision, recall, and F1 score. We also employed confusion matrices to examine patterns of misclassification across models.Most open-source LLMs demonstrated exceptional performance in classifying reports into different diagnostic categories.Google’s Gemma 2 model achieved the highest average F1 score of 0.98, followed by Qwen2.5:32B and DeepseekR1-32Bwith F1 scores of 0.96 and 0.95, respectively. All other evaluated models attained average scores above 0.93, with Mistraland DeepseekR1-7B being the only exceptions. The top four LLMs outperformed our board-certified cardiologist (F1 scoreof 0.94) across all evaluation metrics in analyzing CMR reports. Our findings demonstrate the feasibility of implementingopen-source, privacy-preserving LLMs in clinical settings for automated analysis of imaging reports, enabling accurate, fastand resource-efficient diagnostic categorization.","manuscriptTitle":"Comparative analysis of privacy-preserving open-source LLMs regarding extraction of diagnosticinformation from clinical CMR imaging reports","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-10-26 05:41:44","doi":"10.21203/rs.3.rs-6776028/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2026-04-15T18:29:13+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-10-29T09:22:37+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-10-28T11:27:55+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-10-27T14:39:32+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-10-13T15:06:45+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"130216608584659939079072128021651814109","date":"2025-10-12T23:04:54+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"290024569270549918299749979249311482734","date":"2025-10-12T13:18:59+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"56482243615847260182029540130284400827","date":"2025-10-12T13:04:50+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"319604684697053811221684113228376804329","date":"2025-10-12T08:54:21+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"47249006667440176674010249925928119842","date":"2025-10-10T16:15:48+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-10-10T11:31:55+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-10-07T14:25:43+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2025-06-09T17:37:20+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-06-06T07:11:48+00:00","index":"","fulltext":""},{"type":"submitted","content":"Scientific Reports","date":"2025-06-06T07:09:12+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"3f2d1838-97a3-41ee-af76-0befe0e60809","owner":[],"postedDate":"October 26th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[{"id":56579127,"name":"Physical sciences/Engineering/Biomedical engineering"},{"id":56579128,"name":"Physical sciences/Mathematics and computing/Computational science"},{"id":56579129,"name":"Physical sciences/Mathematics and computing/Computer science"},{"id":56579130,"name":"Physical sciences/Mathematics and computing/Scientific data"}],"tags":[],"updatedAt":"2026-04-23T05:54:59+00:00","versionOfRecord":[],"versionCreatedAt":"2025-10-26 05:41:44","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-6776028","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6776028","identity":"rs-6776028","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}