Combining Clinician Expertise with Prompt Engineering enhances Small Language Models Reliability for Cancer Entity Recognition in Electronic Health Records

ABSTRACT Real-world data (RWD), largely stored in unstructured electronic health records (EHRs), are critical for understanding complex diseases like cancer. However, extracting structured information from these narratives is challenging due to linguistic variability, semantic complexity, and privacy concerns. This study evaluates the performance of four locally deployable and small language models (SLMs), LLaMA, Mistral, BioMistral, and MedLLaMA, for information extraction (IE) from Italian EHRs within the APOLLO 11 trial on non-small cell lung cancer (NSCLC). We examined three prompting strategies (zero-shot, few-shot, and annotated few-shot) across English and Italian, involving clinicians with varying expertise to assess prompt design’s impact on accuracy. Results show that general-purpose models (e.g., LLaMA 3.1 8B) outperform biomedical models in most tasks, particularly in extracting binary features. Multiclass variables such as TNM staging, PD-L1, and ECOG were more difficult due to implicit language and lack of standardization. Few-shot prompting and native-language inputs significantly improved performance and reduced hallucinations. Clinical expertise enhanced consistency in annotation, particularly among students using annotated examples. The study confirms that privacy-preserving SLMs can be deployed locally for efficient and secure cancer data extraction. Findings highlight the need for hybrid systems combining SLMs with expert input and underline the importance of aligning clinical documentation practices with SLM capabilities. This is the first study to benchmark SLMs on Italian EHRs and investigate the role of clinical expertise in prompt engineering, offering valuable insights for the future integration of SLMs into real-world clinical workflows. Competing Interest Statement All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: no support from any organisation for the submitted work; LM declares conference grants from Sanofi, Daiichi Sankyo, LEOPharma; honoraria from Novartis, MSD, Elma Research. VM declares speaker honorarium from Novartis. AF declares speaker honorarium from Novartis. L.Provenzano declares speaker honorarium from Novartis, Pfizer, Gilead. AS declares speaker honorarium from Novartis, MSD, BMS, Immunocore, Pierre Fabre. GC declares speaker honorarium from Takeda, conference grants from Amgen, AstraZeneca. TB declares conference grants from MSD, Sanofi, Pfizer, Lilly; honoraria from MSD. MO declares consulting/advisory role for AstraZeneca, BMS, MSD, Pfizer, JeJ; honoraria from AstraZeneca, BMS, MSD; conference grants from Eli Lilly, JeJ. M.Brambilla declares conference grants from Eli Lilly, JeJ, LeoPharma, honoraria from BMS, AstraZeneca. CP declares personal fees from Italfarmaco, AstraZeneca, BMS, Merck Sharp and Dohme, and Janssen; and institutional funding from Novartis. JNK declares consulting services for Panakeia, AstraZeneca, MultiplexDx, Mindpeak, Owkin, DoMore Diagnostics, and Bioptimus. Furthermore, he holds shares in StratifAI, Synagen, Tremont AI, and Ignition Labs, has received an institutional research grant from GSK, and has received honoraria from AstraZeneca, Bayer, Daiichi Sankyo, Eisai, Janssen, Merck, MSD, BMS, Roche, Pfizer, and Fresenius. ALGP declares co-founder and shareholder of two startup companies, Agade srl and AllyArm srl; speaker honorarium from Novartis. FdB declares a patent for PCT/ IB2020/055956 pending and a patent for IT201900009954 pending; honoraria from, or consultant role for, Roche, EMD Serono, NMS Nerviano Medical Science, Sanofi, MSD, Novartis, Incyte, BMS, Menarini Healthcare Research & Pharmacoepidemiology, Merck Group, Pfizer, Servier, AMGEN, Incyte, outside the submitted work. GLR declares consultant role for Roche, Novartis, BMS, MSD, AstraZeneca, Takeda, Amgen, Sanofi, Italfarmaco, Pfizer; payment or honoraria for lectures, presenta- tions, speakers bureaus, manuscript writing or educational events from Roche, Novartis, BMS, MSD, AstraZeneca, Takeda, Amgen, Sanofi; support for attending meetings and/or travel from Roche, BMS, MSD; data safety monitoring board or advisory board for Roche, Novartis, BMS, MSD, AstraZeneca, Sanofi; has acted as principal investigator in spon- sored clinical trials for Roche, Novartis, BMS, MSD, AstraZeneca, GSK, Amgen, Sanofi, outside the submitted work. AP declares consulting/advisory role for BMS, AstraZeneca, Novartis; travel, accommodations, or other expenses paid or reimbursed by Roche, Italfarmaco; principal investigator of Spectrum Pharmaceuticals; personal fees from Roche, AstraZeneca and BMS, outside the submitted work. FC, VP, GL, L.Passos, JA, ICW, FW, GM, RR, PA, TC, SN, SR, PA, MDP, AR, GS, MMP, CC, CG, RS, M.Borracino., CB, RMDM, CA, ADD, GDL, MG, PB declare they have no financial or non-financial interests to disclose. No other relationships or activities that could appear to have influenced the submitted work. Clinical Trial NCT05550961 Clinical Protocols https://clinicaltrials.gov/study/NCT05550961 Funding Statement This study was funded by Fondazione IRCCS Istituto Nazionale dei Tumori di Milano, 5 per 1000 Ministry of Health funds under the project DWH 2.0: maintenance, integration, harmonization, and diffusion; and 5 per 1000 Ministry of University and Research funds and Fondazione IRCSS Istituto Nazionale dei Tumori, through its call for the Valorisation of Institutional Research Program BRI 2021. Author Declarations I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained. Yes The details of the IRB/oversight body that provided approval or exemption for the research described are given below: Ethics committee/IRB of Fondazione IRCCS Istituto Nazionale dei Tumori di Milano gave ethical approval for this work I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals. Yes I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance). Yes I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable. Yes Data Availability All data produced in the present study are not readily available because of patients' privacy protection. Requests to access the datasets should be directed to the corresponding author