Implementing a Resource-Light and Low-Code Large Language Model System for Information Extraction from Mammography Reports: A Case Study

Background Large Language Models (LLMs) have been successfully used to extract structured data from free-text radiology reports. Most of current studies were conducted with private models accessed via Application Programming Interface (API). We aimed to evaluate the feasibility of using open-source LLMs, deployed on limited local hardware resources for extraction of structured information from free-text mammography reports, according to a Common Data Elements (CDE)-based framework.

Seventy-nine CDEs were defined by an interdisciplinary expert panel, reflecting real-world reporting practice. Sixty-one reports were classified by two independent researchers with 1533 classifications assigned to establish ground truth. Five different open-source LLMs deployable on a single GPU were used for data extraction using the general-classifier Python package. Extractions were performed for two different prompt approaches with classification metrics calculated overall and on subgroups. Additional analyses were conducted using thresholds for the relative probability of classifications.

High inter-rater agreement was observed between manual classifiers (Cohen’s Kappa 0.83). Using default prompts, the LLMs achieved accuracies of 59.23–72.86%. Adapting prompts to better explain classification tasks improved performance for all models, with accuracies of 64.71–85.32%. Setting certainty thresholds further improved accuracies to >90% but reduced the coverage rate to <50%.

Locally deployed open-source LLMs can effectively extract information from mammography reports with good accuracy, addressing data privacy concerns while maintaining compatibility with limited computational resources. Prompt engineering substantially increases performance, highlighting the importance of optimization in clinical applications. Using a CDE-based framework provides clear semantics and structure, facilitating interoperability and consistent data extraction. Competing Interest Statement Dr. Cihoric is a technical lead for the SmartOncology project and medical advisor for Wemedoo AG, Steinhausen AG, Switzerland. The authors declare no other conflicts of interest. Funding Statement This study was supported by Innosuisse grant 59228.1/120.503 "SMARAGD". Author Declarations I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained. Yes The details of the IRB/oversight body that provided approval or exemption for the research described are given below: An ethics approval for the study was granted by the Ethics Committee of the Canton of Bern (BASEC number 2022-01621), in alignment with the principles outlined in the Declaration of Helsinki. I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals. Yes I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance). Yes I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable. Yes Data Availability The source code for the project as well as the .JSON files containing the relevant data about the data structure with classification topics, categories and prompts are provided at GitHub (https://github.com/Smart-Radiology-Goes-Digital-SMARAGD/low-code-LLM-CDE-extraction-mammography). Information regarding the used anonymized mammography reports can be obtained from the authors upon reasonable request. List of abbreviations - ACR - American College of Radiology - AI - Artificial Intelligence - API - Application Programming Interface - CDE - Common Data Element - IRA - Inter-rater agreement - IT - information technology - LLM - Large Language Model - LoRA - Low-Rank Adaptation - NER - Named entity recognition - NIH - National Institutes of Health - NLP - Natural Language Processing - RSNA - Radiological Society of North America - RWD - Real-world Data