Automated ACR TI-RADS Classification of Thyroid Nodules from Narrative Ultrasound Reports Using a Fine-Tuned Open-Source Language Model: A Reproducible and Low-Resource Framework

Automated ACR TI-RADS Classification of Thyroid Nodules from Narrative Ultrasound Reports Using a Fine-Tuned Open-Source Language Model: A Reproducible and Low-Resource Framework | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Automated ACR TI-RADS Classification of Thyroid Nodules from Narrative Ultrasound Reports Using a Fine-Tuned Open-Source Language Model: A Reproducible and Low-Resource Framework Miao Yu, Sijia Huang, Muyang Li, Likuan Zhang, Heng Zhang, Qiao Xu, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7864505/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Background: Manual ACR TI-RADS classification from narrative ultrasound reports is a key component of thyroid nodule risk stratification but is laborious and subject to inter-observer variability. While Large Language Models (LLMs) offer potential solutions, existing approaches often rely on proprietary models or require extensive computational resources, limiting widespread adoption. This study aimed to develop and validate a reproducible, low-resource framework using a fine-tuned open-source LLM to automate this task. Methods: This retrospective study utilized a dataset of 1,850 de-identified thyroid ultrasound reports from a primary single center. The reports were annotated by radiologists to establish a ground truth. An open-source 7-billion parameter model (Qwen1.5-7B) was fine-tuned on a training set (n=1,480) using Low-Rank Adaptation (LoRA) on a single consumer-grade GPU. The model's performance was evaluated on a hold-out internal test set (n=370) and a separate external validation set (n=210) from another institution. Results: On the internal test set, the fine-tuned model achieved an overall accuracy of 93.0% and a macro-averaged F1-score of 0.950. On the external validation set, it maintained robust performance with an accuracy of 88.6% and a macro F1-score of 0.891, demonstrating strong generalizability. It significantly outperformed both a zero-shot LLM baseline and a traditional machine learning model (TF-IDF with SVM) on both datasets. Conclusions: Fine-tuning an accessible, open-source language model on local, consumer-grade hardware is an effective and resource-efficient strategy for automating ACR TI-RADS classification from narrative reports. This approach offers a practical and generalizable blueprint for healthcare institutions to develop bespoke AI tools, potentially enhancing workflow efficiency and diagnostic consistency while preserving data privacy. thyroid nodule ACR TI-RADS large language model natural language processing automated classification ultrasound Full Text Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7864505","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":539372558,"identity":"7cd3c5f4-d7c4-439b-8dda-17726b336d89","order_by":0,"name":"Miao Yu","email":"","orcid":"","institution":"The First Affiliated Hospital of University of Science and Technology of China","correspondingAuthor":false,"prefix":"","firstName":"Miao","middleName":"","lastName":"Yu","suffix":""},{"id":539372559,"identity":"fed40ee3-a437-412e-9723-503c97f03867","order_by":1,"name":"Sijia Huang","email":"","orcid":"","institution":"The First Affiliated Hospital of University of Science and Technology of China (Department of Clinical Pathology)","correspondingAuthor":false,"prefix":"","firstName":"Sijia","middleName":"","lastName":"Huang","suffix":""},{"id":539372560,"identity":"a344ef17-9923-4f2e-b41e-54dc5d78e67c","order_by":2,"name":"Muyang Li","email":"","orcid":"","institution":"The First Affiliated Hospital of University of Science and Technology of China","correspondingAuthor":false,"prefix":"","firstName":"Muyang","middleName":"","lastName":"Li","suffix":""},{"id":539372561,"identity":"19741ee0-26ac-46ae-8584-8ee2f2955f96","order_by":3,"name":"Likuan Zhang","email":"","orcid":"","institution":"Anqing No. 116 Hospital (Department of Oncology)","correspondingAuthor":false,"prefix":"","firstName":"Likuan","middleName":"","lastName":"Zhang","suffix":""},{"id":539372562,"identity":"47c4aeff-1cd4-4e80-b791-cf010a3cf72a","order_by":4,"name":"Heng Zhang","email":"","orcid":"","institution":"The First Affiliated Hospital of University of Science and Technology of China","correspondingAuthor":false,"prefix":"","firstName":"Heng","middleName":"","lastName":"Zhang","suffix":""},{"id":539372563,"identity":"ed8acc09-90d0-47a0-80d2-f28585545a66","order_by":5,"name":"Qiao Xu","email":"","orcid":"","institution":"The First Affiliated Hospital of University of Science and Technology of China","correspondingAuthor":false,"prefix":"","firstName":"Qiao","middleName":"","lastName":"Xu","suffix":""},{"id":539372564,"identity":"f57b807b-4dd7-4ce3-b69e-f96a1edefc81","order_by":6,"name":"Zikang Wang","email":"","orcid":"","institution":"The First Affiliated Hospital of University of Science and Technology of China","correspondingAuthor":false,"prefix":"","firstName":"Zikang","middleName":"","lastName":"Wang","suffix":""},{"id":539372565,"identity":"6e40a3b0-87b4-4dde-94c7-f10ae4a9d894","order_by":7,"name":"Jian Gao","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA/UlEQVRIiWNgGAWjYJACgwQQwcDcwMBQwcBDihZGoJYzRGqB6ANpYWwjQqW8++EDBQ9q7sib8x9sk+adVydjzn6A8cPHHNxaDM+kJRgkHHtmuHNGIlDLtsM8lj0JzJIzt+HR0pBjYJDAdphxww1GkJYDPAYHEtiYefFp6X8D1PLvsP2G8yCHzanjMTj/AL8WeQmgLYlthxM3HAA5rIGZx+AGAVsMJJ4lGCT2HU7ecCOx2XLOscNALQ+b8fpFvj/5mOGPb4dtN5w/fPDGm5o6e4PzyQc/fMRnywEGNgMom0UKEo+gOMUD5BsYmB9A2cwff+BVOwpGwSgYBSMVAADmZ1gKkQoJUAAAAABJRU5ErkJggg==","orcid":"","institution":"The First Affiliated Hospital of University of Science and Technology of China","correspondingAuthor":true,"prefix":"","firstName":"Jian","middleName":"","lastName":"Gao","suffix":""}],"badges":[],"createdAt":"2025-10-15 06:38:27","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7864505/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7864505/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":95081928,"identity":"2cc4ec96-b3c3-41bb-baca-810b1ffb575b","added_by":"auto","created_at":"2025-11-04 06:28:50","extension":"json","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":9452,"visible":true,"origin":"","legend":"","description":"","filename":"b6d0b56731be4e43a0e562ac03e746f6.json","url":"https://assets-eu.researchsquare.com/files/rs-7864505/v1/e5f2309d23efa1e07ceb11ab.json"},{"id":95081929,"identity":"bbaba344-5b33-4714-8d7f-431756499825","added_by":"auto","created_at":"2025-11-04 06:28:50","extension":"xml","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":60326,"visible":true,"origin":"","legend":"","description":"","filename":"b6d0b56731be4e43a0e562ac03e746f61enriched.xml","url":"https://assets-eu.researchsquare.com/files/rs-7864505/v1/2192135945ea81a003924e76.xml"},{"id":95081930,"identity":"dde7acb1-3f61-4663-92f5-e2170519b1e8","added_by":"auto","created_at":"2025-11-04 06:28:50","extension":"pdf","order_by":3,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":88974,"visible":true,"origin":"","legend":"","description":"","filename":"coverletter3.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7864505/v1/0dd7e065cf7d679baefacfb7.pdf"},{"id":95081932,"identity":"7e66f0f6-cfbf-4789-aba1-ad545ddc8943","added_by":"auto","created_at":"2025-11-04 06:28:50","extension":"eps","order_by":4,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":2890,"visible":true,"origin":"","legend":"","description":"","filename":"empty.eps","url":"https://assets-eu.researchsquare.com/files/rs-7864505/v1/958815c5a103ef9ffeb5e25d.eps"},{"id":95223382,"identity":"616b9160-e0af-4538-b722-faddd33eec3b","added_by":"auto","created_at":"2025-11-05 16:22:10","extension":"eps","order_by":5,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":91593,"visible":true,"origin":"","legend":"","description":"","filename":"fig.eps","url":"https://assets-eu.researchsquare.com/files/rs-7864505/v1/886b1d8992fb6685280b56f9.eps"},{"id":95081931,"identity":"23eabc2f-34d5-4e06-8743-0faa30d814ab","added_by":"auto","created_at":"2025-11-04 06:28:50","extension":"bbl","order_by":6,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":17769,"visible":true,"origin":"","legend":"","description":"","filename":"main.bbl","url":"https://assets-eu.researchsquare.com/files/rs-7864505/v1/491cdb331abb7931d9886ad1.bbl"},{"id":95223456,"identity":"e3aa5be1-a0fa-4424-b9be-165a23715a27","added_by":"auto","created_at":"2025-11-05 16:22:17","extension":"bst","order_by":8,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":29828,"visible":true,"origin":"","legend":"","description":"","filename":"snaps.bst","url":"https://assets-eu.researchsquare.com/files/rs-7864505/v1/fbdbacac5fc59e01a4cd3beb.bst"},{"id":95081945,"identity":"4fe23cf3-03e4-4a19-8beb-c9ea28c9edc4","added_by":"auto","created_at":"2025-11-04 06:28:50","extension":"bst","order_by":9,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":35515,"visible":true,"origin":"","legend":"","description":"","filename":"snbasic.bst","url":"https://assets-eu.researchsquare.com/files/rs-7864505/v1/beef112ace4a940b04ac9152.bst"},{"id":95223503,"identity":"ab9301de-8176-4fd2-a8b2-8751354098fe","added_by":"auto","created_at":"2025-11-05 16:22:22","extension":"bst","order_by":10,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":33968,"visible":true,"origin":"","legend":"","description":"","filename":"snchicago.bst","url":"https://assets-eu.researchsquare.com/files/rs-7864505/v1/ce5e2dbe98e1818c6c6522e0.bst"},{"id":95081947,"identity":"6af76011-55d9-4afe-8735-42c8037884b2","added_by":"auto","created_at":"2025-11-04 06:28:50","extension":"cls","order_by":11,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":55857,"visible":true,"origin":"","legend":"","description":"","filename":"snjnl.cls","url":"https://assets-eu.researchsquare.com/files/rs-7864505/v1/c8b17917cd7ba6111f8f718c.cls"},{"id":95081942,"identity":"5caec01e-8bc5-440c-9e40-391e274b15df","added_by":"auto","created_at":"2025-11-04 06:28:50","extension":"bst","order_by":12,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":64023,"visible":true,"origin":"","legend":"","description":"","filename":"snmathphysay.bst","url":"https://assets-eu.researchsquare.com/files/rs-7864505/v1/2641c2afa49c5b01abb75b4a.bst"},{"id":95223499,"identity":"1655dba8-d051-43fb-8da8-840d23463b0e","added_by":"auto","created_at":"2025-11-05 16:22:22","extension":"bst","order_by":13,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":64166,"visible":true,"origin":"","legend":"","description":"","filename":"snmathphysnum.bst","url":"https://assets-eu.researchsquare.com/files/rs-7864505/v1/d43cda5b4db1546772be7ef6.bst"},{"id":95081952,"identity":"573bb4cd-2291-4edb-bdc5-50c31e9974db","added_by":"auto","created_at":"2025-11-04 06:28:50","extension":"bst","order_by":14,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":37333,"visible":true,"origin":"","legend":"","description":"","filename":"snnature.bst","url":"https://assets-eu.researchsquare.com/files/rs-7864505/v1/3acdbb687fde2cc6b929b58e.bst"},{"id":95223520,"identity":"fdeeb896-7dc1-43b5-b6ec-aa30bb05f6cb","added_by":"auto","created_at":"2025-11-05 16:22:24","extension":"bst","order_by":15,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":39951,"visible":true,"origin":"","legend":"","description":"","filename":"snvancouveray.bst","url":"https://assets-eu.researchsquare.com/files/rs-7864505/v1/21f04577e8a2cd0482ffba6f.bst"},{"id":95081949,"identity":"6ff8ab7c-c852-4b62-b86a-74a7162bdb49","added_by":"auto","created_at":"2025-11-04 06:28:50","extension":"bst","order_by":16,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":40758,"visible":true,"origin":"","legend":"","description":"","filename":"snvancouvernum.bst","url":"https://assets-eu.researchsquare.com/files/rs-7864505/v1/b8e50dc572a8e2ef8f9dacbe.bst"},{"id":95081954,"identity":"0648923d-435a-4675-b466-af9ba9e49a34","added_by":"auto","created_at":"2025-11-04 06:28:51","extension":"pdf","order_by":17,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":430996,"visible":true,"origin":"","legend":"","description":"","filename":"zxjkfjhkrbfqpmmkgrhtdhmgzhzfdhpt.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7864505/v1/b9e56c8a442cbfce76c2e8c2.pdf"},{"id":95081948,"identity":"9aa55e2e-5a2b-460d-b15c-1603e9dfc6b8","added_by":"auto","created_at":"2025-11-04 06:28:50","extension":"xml","order_by":18,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":65859,"visible":true,"origin":"","legend":"","description":"","filename":"b6d0b56731be4e43a0e562ac03e746f61structuring.xml","url":"https://assets-eu.researchsquare.com/files/rs-7864505/v1/68fd5aaf434b2c77ad65f39d.xml"},{"id":95081950,"identity":"c256e0c8-4602-498b-8b81-581e7ba752da","added_by":"auto","created_at":"2025-11-04 06:28:50","extension":"html","order_by":19,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":69013,"visible":true,"origin":"","legend":"","description":"","filename":"earlyproof.html","url":"https://assets-eu.researchsquare.com/files/rs-7864505/v1/00e0b500d77a4e7831549e4a.html"},{"id":97898941,"identity":"fc983ed7-439b-4084-ae87-37b1c80ca720","added_by":"auto","created_at":"2025-12-10 15:40:15","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":345596,"visible":true,"origin":"","legend":"","description":"","filename":"zxjkfjhkrbfqpmmkgrhtdhmgzhzfdhpt.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7864505/v1_covered_1164f5ed-a053-455f-997f-04ee6cbfab0f.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Automated ACR TI-RADS Classification of Thyroid Nodules from Narrative Ultrasound Reports Using a Fine-Tuned Open-Source Language Model: A Reproducible and Low-Resource Framework","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"thyroid nodule, ACR TI-RADS, large language model, natural language processing, automated classification, ultrasound","lastPublishedDoi":"10.21203/rs.3.rs-7864505/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7864505/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eBackground:\u003c/h2\u003e\u003cp\u003eManual ACR TI-RADS classification from narrative ultrasound reports is a key component of thyroid nodule risk stratification but is laborious and subject to inter-observer variability. While Large Language Models (LLMs) offer potential solutions, existing approaches often rely on proprietary models or require extensive computational resources, limiting widespread adoption. This study aimed to develop and validate a reproducible, low-resource framework using a fine-tuned open-source LLM to automate this task.\u003c/p\u003e\u003ch2\u003eMethods:\u003c/h2\u003e\u003cp\u003eThis retrospective study utilized a dataset of 1,850 de-identified thyroid ultrasound reports from a primary single center. The reports were annotated by radiologists to establish a ground truth. An open-source 7-billion parameter model (Qwen1.5-7B) was fine-tuned on a training set (n=1,480) using Low-Rank Adaptation (LoRA) on a single consumer-grade GPU. The model's performance was evaluated on a hold-out internal test set (n=370) and a separate external validation set (n=210) from another institution.\u003c/p\u003e\u003ch2\u003eResults:\u003c/h2\u003e\u003cp\u003eOn the internal test set, the fine-tuned model achieved an overall accuracy of 93.0% and a macro-averaged F1-score of 0.950. On the external validation set, it maintained robust performance with an accuracy of 88.6% and a macro F1-score of 0.891, demonstrating strong generalizability. It significantly outperformed both a zero-shot LLM baseline and a traditional machine learning model (TF-IDF with SVM) on both datasets.\u003c/p\u003e\u003ch2\u003eConclusions:\u003c/h2\u003e\u003cp\u003eFine-tuning an accessible, open-source language model on local, consumer-grade hardware is an effective and resource-efficient strategy for automating ACR TI-RADS classification from narrative reports. This approach offers a practical and generalizable blueprint for healthcare institutions to develop bespoke AI tools, potentially enhancing workflow efficiency and diagnostic consistency while preserving data privacy.\u003c/p\u003e","manuscriptTitle":"Automated ACR TI-RADS Classification of Thyroid Nodules from Narrative Ultrasound Reports Using a Fine-Tuned Open-Source Language Model: A Reproducible and Low-Resource Framework","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-11-04 06:28:46","doi":"10.21203/rs.3.rs-7864505/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"3d11f323-06f4-403c-bf70-135a43e6fc50","owner":[],"postedDate":"November 4th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-12-09T17:53:31+00:00","versionOfRecord":[],"versionCreatedAt":"2025-11-04 06:28:46","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7864505","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7864505","identity":"rs-7864505","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}