Scalable information extraction from free text electronic health records using large language models

Scalable information extraction from free text electronic health records using large language models | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Scalable information extraction from free text electronic health records using large language models Bowen Gu, Vivian Shao, Ziqian Liao, Valentina Carducci, Santiago Romero-Brufau, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4714451/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 28 Jan, 2025 Read the published version in BMC Medical Research Methodology → Version 1 posted 12 You are reading this latest preprint version Abstract Background A vast amount of potentially useful information such as description of patient symptoms, family, and social history is recorded as free-text notes in electronic health records (EHRs) but is difficult to reliably extract at scale, limiting their utility in research. This study aims to assess whether an “out of the box” implementation of open-source large language models (LLMs) without any fine-tuning can accurately extract social determinants of health (SDoH) data from free-text clinical notes. Methods We conducted a cross-sectional study using EHR data from the Mass General Brigham (MGB) system, analyzing free-text notes for SDoH information. We selected a random sample of 200 patients and manually labeled nine SDoH aspects. Eight advanced open-source LLMs were evaluated against a baseline pattern-matching model. Two human reviewers provided the manual labels, achieving 93% inter-annotator agreement. LLM performance was assessed using accuracy metrics for overall, mentioned, and non-mentioned SDoH, and macro F1 scores. Results LLMs outperformed the baseline pattern-matching approach, particularly for explicitly mentioned SDoH, achieving up to 40% higher Accuracy mentioned . openchat_3.5 was the best-performing model, surpassing the baseline in overall accuracy across all nine SDoH aspects. The refined pipeline with prompt engineering reduced hallucinations and improved accuracy. Conclusions Open-source LLMs are effective and scalable tools for extracting SDoH from unstructured EHRs, surpassing traditional pattern-matching methods. Further refinement and domain-specific training could enhance their utility in clinical research and predictive analytics, improving healthcare outcomes and addressing health disparities. Social Determinants of Health (SDoH) Electronic Health Records (EHR) Natural Language Processing (NLP) Large Language Models (LLMs) Clinical Information Extraction Figures Figure 1 Figure 2 Figure 3 Figure 4 BACKGROUND A vast amount of potentially useful information such as description of patient symptoms, family and social history, is not recorded as structured fields but found in free-text notes from electronic health records (EHRs). Substantial feature engineering is needed to directly extract and analyze the relevant information contained within free-text notes, which is time and resource intensive. [ 1 – 3 ] Therefore, free-text notes have historically been underutilized for research activities. [ 4 ] Various Natural Language Processing (NLP) models have seen developed to extract clinical concepts from EHR notes automatically. [ 5 – 6 ] Traditional rule-based NLP approaches are often constrained by their lack of generalizability across different datasets and settings. [ 7 – 8 ] Machine learning-based NLP models rely heavily on the costly step of manual data annotation for model training, which makes them harder to scale across diverse concepts, particularly in clinical environments where data annotation requires specialized knowledge, cost and confidentiality considerations. [ 9 – 11 ] In contrast, Large Language Models (LLMs), advanced deep learning models that are pre-trained on large volumes of text, provide a scalable alternative for the task of clinical information extraction. [ 12 – 14 ] With their capabilities of zero-shot and few-shot learning, LLMs can extract target clinical information from EHR notes without the need for complex rules creation or extensive data annotation. [ 15 – 17 ] Social determinants of health (SDoH) of patients are the economic and social conditions that influence individual and group differences in health status. [ 18 – 20 ] They are important for assessing and addressing health disparities, and are critical for clinical interventions and research. [ 21 – 23 ] For instance, quantifying the role SDoH in predisposing patients to adverse health outcomes is often of interest to facilitate development of educational and counseling interventions. [ 24 – 25 ] In epidemiologic investigations, various domains of SDoH are also frequently considered important confounding variables that require adjustment. [ 26 – 28 ] While structured EHRs typically lack explicit recording of SDoH information, it is frequently available in free-text form in notes. [ 29 – 31 ] As variables of universal interest, we selected SDoH as a representative use case where scalable extraction methods like LLMs can be of high value. In the current study, we aimed to assess an “out of the box” implementation of 8 advanced open-source LLMs without any fine-tuning for their capabilities on extracting 9 SDoH from free-text EHRs, and compare their performance with the traditional approach of using basic pattern matching. METHODS Data Source We used EHR data from the Mass General Brigham (MGB) system, which is the largest healthcare system in the state of Massachusetts. The sampling frame included a total of 1.2 million individuals with deterministically linked EHRs from MGB to insurance claims data from Medicare and Medicaid for the period of 2007–2020. For this study, we used free text associated with the patients’ social history documentation from progress notes using regular expression matching. Since the patients’ social documentation is added incrementally over time, we only used the most recent social documentation for each patient. SDoH Questions and Manual Labels To identify the SDoH that are frequently reported in social documentation, we first performed a manual review of a random sample of 200 patients’ social documentation and summarized 9 aspects of the patient’s SDoH that appeared in > 5% of the notes. This list included marital status, number of children, employment status, educational status, lifestyle factors (use of tobacco, alcohol, illicit drugs, exercise), and cohabitation status. Of the random sample of the 200 reviewed patients, we split the first 100 patients’ social documentation as the validation set to inform prompt engineering (described below) and the remaining 100 patients’ social documentation as the test set to evaluate performance. Following the classic evaluation framework of LLM evaluation, we converted the SDoH extraction as a question-answer problem. That is, for each of the 9 SDoH characteristics, given the EHR notes, we designed a question and candidate options along with the note text as the LLM input, and let the LLM to select the option from the candidates. The SDoH questions, together with their distribution of candidate options in the validation and the test set, are shown in Supplementary Table S1 . For each of the 200 patients, two human reviewers (B.G and V.S) manually labeled the 9 SDoH aspects to one of the quantified choices according to the labeling criteria documentation ( Supplementary : Annotation Guide section). Each reviewer labeled the 200 patients independently. The inter-annotator agreement was calculated based on the total number of the 1800 questions (9 SdoH questions * 200 patients) that the two annotators agreed before discussion among all the 1800 questions, which was 93%. For the inconsistently annotated cases, the two annotators discussed them in detail and reached a consensus. New criteria were also added to the labeling criteria documentation that addressed the causes of these inconsistencies. Experiment Settings We selected 8 well performing open source LLMs on the LLM leaderboard hosted by Hugging Face. [ 32 – 41 ] All LLMs used in this study are publicly available. The details of the LLMs and the links to the models can be found in Supplementary Table S2 . A copy of the model weights went through the AWQ quantization process to generate the quantized model weights. [ 42 – 43 ] Quantization is a technique of reducing the model size for faster inference in resource limited settings. Rule-based Baseline Model To evaluate the comparative performance of LLMs against a common baseline, we designed a model that used pattern matching to extract the answers to the SDoH questions from the patients’ social documentation. The matching patterns were designed according to the labeling criteria. If a match was found in the patients’ social documentation, the output answer was guaranteed to be one of the choices of the SDoH question. If no match was found in the patients’ social documentation, the output answer was “Not mentioned”. To avoid mismatching (e.g. answer “No” was matched to “Not mentioned” since “No” was a substring of “Not mentioned”), we sorted the choices by their character lengths in descending order and matched the response from the longest choice to the shortest choice and stopped matching if the longer choice was matched. The specific patterns for each SDoH question are shown in Supplementary Table S3. Pipeline Workflow and prompt engineering We built two pipelines: a default pipeline and a refined pipeline. The default pipeline was designed to run the LLM to extract the SDoHs from the unstructured social documentation using the default prompt for all SDoH questions. Alternatively, the refined pipeline used different engineered prompts on 3 (Q2, Q6, and Q7) of the 9 SDoH questions that most LLMs struggled with in the validation set experiments. We ran both pipelines to compare the effectiveness of the refinement. An illustration of the two pipelines is shown in Fig. 1 . The “LLM Response Postprocessing” step included implementing a systematic code that uses pattern matching to map the model response to one of the choices of the SDoH questions. The “Auto-Grader” took the refined response from the “LLM Response Postprocessing” procedure and compared it against manual labels. When mapped model response matched exactly with the human label, then the auto-grader considered it as an accurate extraction. Otherwise, the auto-grader considered the LLM extraction inaccurate. The “Model Comparator” step combined the graded model responses from the “Auto-Grader” and provided the grading results in a single chart to formulate the final benchmark. Additional context for the “Default Prompt” and the “Engineered Prompt” is shown in Supplementary Table S4 , which summarizes 4 types of prompts: default prompts (not including the default secondary prompts), premise prompts, special prompts, and secondary prompts. The default pipeline only used the default prompts while the refined pipeline used all 4 types of prompts. LLM performance evaluation To evaluate the model performance, we used three metrics in the test set: Accuracy overall , Accuracy mentioned , and Accuracy non−mentioned , which corresponds to the overall accuracy for extractions, the accuracy when a note contained mention of the specific SDoH, and the accuracy when a note did not contain mention of the specific SDoH, respectively. The three accuracies are defined as follows, where the Accuracy overall is a weighted average of the Accuracy mentioned and the Accuracy non−mentioned , with the weights dependent on the missingness of the SDoH aspects in the text: $$\:{Accuracy}_{overall}\:=\:\frac{Total\:\#\:ofaccurate\:extractions}{Total\:\#\:of\:questions}\:$$ $$\:{Accuracy}_{mentioned}\:=\:\frac{Total\:\#\:of\:accurate\:extractions\:where\:the\:ground\:truth\:is\:NOT\:labeled\:as\:"not\:mentioned"}{Total\:\#\:of\:questions\:where\:the\:ground\:truth\:is\:NOT\:labeled\:as\:"not\:mentioned"}$$ $$\:{Accuracy}_{non-mentioned}=\:\frac{Total\:\#\:of\:accurate\:extractions\:where\:the\:ground\:truth\:is\:labeled\:as\:"not\:mentioned"}{Total\:\#\:of\:questions\:where\:the\:ground\:truth\:is\:labeled\:as\:"not\:mentioned"}$$ We calculated all three metrics on all 9 SDoH questions. To calculate the confidence interval for these accuracies, we used the Jackknife resampling technique to generate samples for each model accuracy on every question. [ 44 ] We then calculated the 95% CI of each accuracy using the samples generated. We assumed t-distribution since our sample size was small (100 per model, per question, per accuracy). For questions that did not meet the premise (e.g. The human label is “not mentioned” for this patient on this SDoH question when trying to calculate Accuracy mentioned ), we marked them as not applicable for calculations. To evaluate the performance difference between the LLM and the baseline, we also used the Jackknife resampling to generate samples for each model on each question on each accuracy. After that, we performed a two-sided Welch's t-test on the accuracy differences between the LLM and the baseline using the samples generated from the LLM and the baseline, assuming t-distribution. Additionally, when the post-processing procedure could not map the LLM response to one of the predefined choices, we defined the response to be invalid and reported the proportion of invalid responses for all models. We further reported the F1 score, calculated as a harmonic mean of precision and recall, using the macro averaging method considering all the SDoH questions as multi-class classification problems. When calculating F1 scores for LLM responses, we combined invalid responses with the ‘not mentioned’ category for each question as a default since invalid responses from LLMs lacked a corresponding manual label. RESULTS LLM Accuracy To compare the performance between the LLMs and the baseline, we averaged the model Accuracy overall across the 9 SDoH questions for each LLM and the baseline and did this for the three accuracies defined above (Fig. 2 (a) and (b)) . The baseline model achieved a 77.33% average Accuracy overall , which was mainly attributable to its high average Accuracy non−mentioned . Among all the LLMs, openchat_3.5 was the only LLM that had a significantly better average Accuracy overall than the baseline, it was also the only LLM that outperformed the baseline on each of the 9 SDoH questions on the average Accuracy overall . On the other hand, the two Llama-2 models had the worst performance on the average Accuracy overall and were significantly lower than the baseline. The remaining LLMs had comparable average Accuracy overall than the baseline, with the zephyr-7b models performing slightly worse. The baseline, which only had limited patterns to match, had an average Accuracy mentioned of 39.02%; while the majority of the LLMs achieved a higher average Accuracy mentioned . Openchat_3.5 achieved over 40% higher of the average Accuracy mentioned than the baseline, which shows its superior capability in extracting information when it is contained in free text. On the other hand, Llama-2-13b-chat was the only LLM that had a significantly worse average Accuracy mentioned compared to the baseline. The Accuracy non−mentioned is the accuracy that indicates the model’s capability of not producing false information (i.e avoiding “hallucination”). [ 45 – 47 ] The baseline, which reported its responses using pattern matching, had a very slim probability of producing false responses unless the information recorded in the text was internally contradicting. On the other hand, the LLMs had a higher probability of producing false information. We found that the baseline achieved an average Accuracy non−mentioned of 96.66%. This was reasonable as the default response from the baseline is “Not mentioned”. The openchat_3.5 model was the best model in terms of minimizing hallucination. On the contrary, the two models that had poor average Accuracy non−mentioned were the two Llama-2 models. Compared to the default pipeline, the refined pipeline helped reduce LLM hallucination, but in some instances, this came at a cost of reduced sensitivity to the SDoH features mentioned in the text. This is expected as we used the premise prompt in the refined pipeline, which aims at reducing model hallucinations, but this prompt can make the model more conservative in its responses. We observed that among all the 9 SDoH questions, Q2 (How many children does the patient have?) and Q7 (What is the patient’s employment status?) were the two questions that most LLMs (except the two Llama 2 models) had a better Accuracy overall than the baseline on both the default and the refined pipeline ( Supplementary Table S6-S7 ). This can be explained by Supplementary Table S8-S9 , which shows that the Accuracy mentioned for the baseline on Q2 and Q7 were very low compared to the LLMs. Besides Q2, and Q7, Supplementary Table S8-S9 indicates that the baseline model also had very poor Accuracy mentioned on SDoH questions Q3 (Does the patient currently use tobacco?) and Q5 (Does the patient currently use illicit drugs?). This is likely attributable to the fact that the unstructured social history documentation has many ways of expressing the answers on these questions, a simpler rule-based approach misses most of them. Supplementary Table S10-S11 indicates that the baseline had a much lower probability to produce false positive results compared to LLMs among all 9 SDoH questions. This eliminated the gains in accuracy by LLMs on the Accuracy mentioned and makes the advantage of most LLMs (except the openchat_3.5 model) slim or even negative on the Accuracy overall . Figure 2 (a). Average accuracies comparison of the LLMs over the baseline (Refined pipeline) Macro F1 Score We averaged the model macro F1 scores across the 9 SDoH questions for each LLM and the baseline (Fig. 3 ). The baseline had 0.53 average macro F1 for the default pipeline and 0.54 average macro F1 for the refined pipeline. Among all the models, openchat_3.5 was the only LLM that had a clear advantage over the baseline. The two Vicuna models and the WizardLM model had comparable performance as the baseline. The zephyr model and the two Llama 2 models had worse average macro F1 scores than the baseline. Invalid Responses Figure 4 shows the comparison of the invalid responses between the refined and the default pipeline. It indicates that the refined pipeline greatly reduced the number of invalid responses for each LLM. Some examples of the models’ invalid responses and the corresponding analysis are shown in Supplementary Table S12 . DISCUSSION In this work, we designed a pipeline that used various open-sourced LLMs to extract patients’ SDoH based on social history documentation in free text EHRs. Using prompt engineering and postprocessing, we demonstrated that some LLMs achieved better performance than the baseline approach of pattern matching on most of the SDoH questions, especially when the answer to the SDoH question was mentioned in the text. We also observed that quantization, which could make these models more applicable in resource limited settings, did not substantially compromise the performance on extraction tasks. Our study offers some practical learnings regarding using LLMs as scalable extraction tools from free text EHRs. First, thoughtful prompt engineering, as we conducted in our ‘refined’ pipeline, can substantially elevate model performance, and should be routinely considered. Second, manual review of erroneous model responses indicated that in almost all such instances, the text was either controversial or required strong deductive reasoning to infer the correct answer, which is a known challenge for LLMs. [ 48 – 50 ] We also found these cases had a high overlap with the cases reported by the human reviewers as ambiguous or controversial, and the reviewers needed to expand the labeling criteria to reach an agreement on such cases, which showed that such cases are also challenging to humans. Third, we documented clear and large differences between LLMs in this task of information extraction, which underscores the importance of evaluating multiple models when considering specific tasks. Our findings suggesting robust performance of “out of the box” implementation of LLMs without any fine-tuning are complementary to that of a prior report [ 16 ] , which demonstrated that when resources permit, fine-tuning LLMs for specific extraction tasks including SDOH information can be a viable alternative to further enhance performance. Our study has important implications. The use of LLM pipelines for generating structured SDoH data from unstructured EHR notes could facilitate numerous downstream applications. One significant area is predictive analytics, where structured SDoH information can be routinely considered alongside structured clinical information to improve risk prediction for various health outcomes. This capability is particularly valuable in managing chronic diseases, where social and environmental factors, such as income and community, play a crucial role. Additionally, structured SDoH data can enhance population health management by enabling healthcare providers to identify and address social disparities in health. In clinical research, the availability of structured SDoH data can refine the selection of study cohorts, improve confounding adjustment in epidemiologic studies, and improve recruitment efforts in clinical trials that aim to oversample socioeconomically disadvantaged patient populations. The strengths of this work include the automation of the SDoH extraction process with high accuracy. As LLMs avoid the need for model training, this approach is likely scalable and transportable across institutions. Limitations include the complexity and subjectivity involved in the prompt engineering and LLM response postprocessing steps. Further, as most open source LLMs are trained using non-clinical text data, specific training on free-text EHRs maybe needed for extraction of more complex clinical concepts with LLMs and our observations regarding model performance based on general SDoH concepts may not extrapolate to other clinical feature extraction tasks. CONCLUSIONS In conclusion, we demonstrated the feasibility of employing open weight LLMs to extract patients’ SDoH with high accuracy without any additional finetuning. LLMs can offer effective and efficient information extraction from EHR text. Abbreviations AWQ Activation-aware Weight Quantization EHR Electronic Health Records LLM Large Language Model MGB Mass General Brigham NLP Natural Language Processing SDoH Social Determinants of Health Declarations Ethics approval and consent to participate Not applicable. Consent for publication Not applicable. Availability of data and materials The data that support the findings of this study are available from Mass General Brigham (MGB) but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. Data are however available from the authors upon reasonable request and with permission of Mass General Brigham (MGB). Competing interests Dr. Desai reports serving as Principal Investigator on investigator-initiated grants to the Brigham and Women’s Hospital from Novartis, Vertex, and Bayer on unrelated projects. Other authors do not have any competing interests to disclose. Funding There is no source of funding for this study. Authors' contributions BG and VS processed the data, labeled the data, analyzed the data, and drafted the manuscript. ZL contributed the idea of the secondary prompt, which significantly improved the study results. VC and SB contributed the idea of the pipelines in this study. JY and RD proposed the idea of the study, supervised the study, and refined the manuscript. All authors read and approved the final manuscript. Acknowledgements Not applicable. References Petch J, Batt J, Murray J, Mamdani M. Extracting clinical features from dictated ambulatory consult notes using a commercially available natural language processing tool: pilot, retrospective, cross-sectional validation study. JMIR Med Inform. Ozery-Flato M, Yanover C, Gottlieb A et al. Fast and efficient feature engineering for multi-cohort analysis of EHR data. Stud Health Technol Inform. Soguero-Ruíz C, Hindberg K, Rojo-Álvarez J et al. Support vector feature selection for early detection of anastomosis leakage from bag-of-words in electronic health records. IEEE J Biomed Health Inform. Koleck TA, Dreisbach C, Bourne PE, Bakken S. Natural language processing of symptoms documented in free-text narratives of Electronic Health Records: A systematic review. J Am Med Inf Assoc. 2019;26(4):364–79. https://doi.org/10.1093/jamia/ocy173 . Reátegui Rojas R, Ratté S. Comparison of MetaMap and cTAKES for entity extraction in clinical notes. BMC Med Inform Decis Mak. Peng J, Zhao M, Havrilla JM et al. Natural language processing (NLP) tools in extracting biomedical concepts from research articles: a case study on autism spectrum disorder. BMC Med Inform Decis Mak. Dasgupta I, Grant E, Griffiths TL. Distinguishing rule- and exemplar-based generalization in learning systems. arXiv. Shi J, Hurdle JF. Trie-based rule processing for clinical NLP: A use-case study of n-trie, making the ConText algorithm more efficient and scalable. J Biomed Inform. Chen Y, Mani S, Xu H. Applying active learning to assertion classification of concepts in clinical text. J Biomed Inf 45(2):265–72. Spasic I, Nenadic G. Clinical text data in machine learning: systematic review. JMIR Med Inf. 8. Landolsi MY, Hlaoua L, Ben Romdhane L. Information extraction from Electronic Medical Documents: State of the art and Future Research Directions. Knowl Inf Syst. 2022;65(2):463–516. https://doi.org/10.1007/s10115-022-01779-1 . Yang X, Pournejatian NM, Shin H-C et al. GatorTron: A large clinical language model to unlock patient information from unstructured electronic health records. medRxiv. 2022.02.27.22271257. Meoni S, De la Clergerie E, Ryffel T. Large language models as instructors: A study on multilingual clinical entity extraction. In The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 178–90, Toronto, Canada. Association for Computational Linguistics. Guevara M, Chen S, Thomas S, et al. Large language models to identify social determinants of health in electronic health records. NPJ Digit Med. 2024;7:6. Agrawal M, Hegselmann S, Lang H et al. Large language models are few-shot clinical information extractors. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1998–2022, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. McInerney DJ, Young GS, van de Meent J-W, Wallace BC. CHiLL: Zero-shot custom interpretable feature extraction from clinical notes with large language models. arXiv:2302.12343. Singhal K, Azizi S, Tu T, et al. Large language models encode clinical knowledge. Nature. 2023;620:172–80. https://doi.org/10.1038/s41586-023-06291-2 . Marmot M. Social determinants of health inequalities. Lancet. 2005;365:1099–104. Brown A. The community impact of social determinants of health. MOJ Public Health. 2016;4(3):98–105. https://doi.org/10.15406/mojph.2016.04.00083 . Wypych-Ślusarska A, Głogowska-Ligus J, Słowiński J. Social determinants of health inequalities. Nurs Public Health. Williams DR, Costa M, Odunlami AO, Mohammed SA. Moving upstream: how interventions that address the social determinants of health can improve health and reduce disparities. J Public. Braveman P, Egerter S, Williams DR. The social determinants of health: Coming of age. Annu Rev Public Health. Hewitt AM. Addressing health disparities: Understanding place in the role of social determinant interventions. arXiv. Anderson E, Lippert SC, Newberry J, et al. Addressing social determinants of health from the emergency department through social emergency medicine. West J Emerg Med. 2016;17(3):487–9. Farthing H, Simko S, Eckembrecher FJ et al. Development and evaluation of electronic social needs assessment and resource connection tool in facilitating utilization of community services that address upstream health. J Student-Run Clin. Poundstone K, Strathdee S, Celentano D. The social epidemiology of human immunodeficiency virus/acquired immunodeficiency syndrome. Epidemiol Rev 26:22–35. Liang Y, Gong Y, Wen X et al. Social determinants of health and depression: A preliminary investigation from rural China. PLoS ONE. 7. Vineis P. Invited perspective: The mysterious case of social determinants of health. Environ Health Perspect. 130. Feller DJ, Oliver JB, Zucker J et al. Detecting social and behavioral determinants of health with structured and free-text clinical data. Appl Clin Inform. Wang MD, Pantell M, Gottlieb L, Adler-Milstein J. Documentation and review of social determinants of health data in the EHR: Measures and associated insights. J Am Med Inform Assoc. Yu Z, Yang X, Guo Y, Bian J, Wu Y. Assessing the documentation of social determinants of health for lung cancer patients in clinical narratives. Front Public Health. Beeching E, Fourrier C, Habib N et al. Open LLM Leaderboard Hugging Face. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard . Gao L, Tow J, Biderman S et al. A framework for few-shot language model evaluation. Zenodo (Version v0.0.1). https://doi.org/10.5281/zenodo.5371628 . Clark P, Cowhey I, Etzioni O et al. Think you have solved question answering? Try ARC, the AI2 Reasoning Challenge. arXiv. https://arxiv.org/abs/1803.05457 . Zellers R, Holtzman A, Bisk Y et al. HellaSwag: Can a machine really finish your sentence? arXiv. https://arxiv.org/abs/1905.07830 . Hendrycks D, Burns C, Basart S et al. Measuring massive multitask language understanding. arXiv. https://arxiv.org/abs/2009.03300 . Lin S, Hilton J, Evans O, TruthfulQA. Measuring how models mimic human falsehoods. arXiv. https://arxiv.org/abs/2109.07958 . Sakaguchi K, Le Bras R, Bhagavatula C, Choi Y. WINOGRANDE: An adversarial Winograd schema challenge at scale. arXiv. https://arxiv.org/abs/1907.10641 . Cobbe K, Kosaraju V, Bavarian M et al. Training verifiers to solve math word problems. arXiv. https://arxiv.org/abs/2110.14168 . Xu C, Sun Q, Zheng K et al. WizardLM: Empowering large language models to follow complex instructions. arXiv:2304.12244v2. Vicuna. An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality. LMSYS Org. https://lmsys.org/blog/2023-03-30-vicuna/ . Accessed April 17, 2024. Lin J, AWQ. Activation-aware weight quantization for LLM compression and acceleration. Retrieved from https://arxiv.org/abs/2306.00978 . AWQ. Activation-aware weight quantization for LLM compression and acceleration - GitHub. GitHub. https://github.com/mit-han-lab/llm-awq . Accessed April 11, 2024. McIntosh A. The jackknife estimation method. arXiv: Methodology. Huang L, Yu W, Feng Z, Feng X. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. Retrieved from https://arxiv.org/abs/2311.05232 . Ji Z, Yu T, Xu Y, et al. Towards mitigating hallucination in large language models via self-reflection. ArXiv. 2023. https://doi.org/10.48550/arXiv.2310.06271 . abs/2310.06271. McKenna N, Li T, Cheng L, Hosseini MJ, Johnson M, Steedman M. Sources of hallucination by large language models on inference tasks. ArXiv. 2023;2758–74. https://doi.org/10.48550/arXiv.2305.14552 . abs/2305.14552 . Seals SM, Shalin VL. Evaluating the deductive competence of large language models. Retrieved from https://arxiv.org/abs/2309.05452 . Chen K, Wang C, Yang K, et al. Gaining wisdom from setbacks: Aligning large language models via mistake analysis. ArXiv. 2023. https://doi.org/10.48550/arXiv.2310.10477 . abs/2310.10477. Liu R, Shah NB. ReviewerGPT? An exploratory study on using large language models for paper reviewing. ArXiv. 2023. https://doi.org/10.48550/arXiv.2306.00622 . abs/2306.00622 . Additional Declarations Competing interest reported. Dr. Desai reports serving as Principal Investigator on investigator-initiated grants to the Brigham and Women’s Hospital from Novartis, Vertex, and Bayer on unrelated projects. Other authors do not have any competing interests to disclose. Supplementary Files Supplimentary.docx Cite Share Download PDF Status: Published Journal Publication published 28 Jan, 2025 Read the published version in BMC Medical Research Methodology → Version 1 posted Editorial decision: Revision requested 21 Oct, 2024 Reviews received at journal 20 Oct, 2024 Reviews received at journal 22 Sep, 2024 Reviewers agreed at journal 22 Sep, 2024 Reviewers agreed at journal 20 Sep, 2024 Reviewers agreed at journal 24 Aug, 2024 Reviewers agreed at journal 20 Aug, 2024 Reviewers invited by journal 19 Jul, 2024 Editor invited by journal 10 Jul, 2024 Editor assigned by journal 10 Jul, 2024 Submission checks completed at journal 10 Jul, 2024 First submitted to journal 09 Jul, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4714451","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":331772017,"identity":"4651b93b-f412-4d16-93b6-872ac2c41680","order_by":0,"name":"Bowen Gu","email":"","orcid":"","institution":"Brigham and Women's Hospital","correspondingAuthor":false,"prefix":"","firstName":"Bowen","middleName":"","lastName":"Gu","suffix":""},{"id":331772025,"identity":"f7cd1fa3-231e-4583-b37a-5d931d37447d","order_by":1,"name":"Vivian Shao","email":"","orcid":"","institution":"Brigham and Women's Hospital","correspondingAuthor":false,"prefix":"","firstName":"Vivian","middleName":"","lastName":"Shao","suffix":""},{"id":331772027,"identity":"a083b5b0-2a5f-476c-b855-0ed5a6db9688","order_by":2,"name":"Ziqian Liao","email":"","orcid":"","institution":"Harvard T.H. Chan School of Public Health, Harvard University","correspondingAuthor":false,"prefix":"","firstName":"Ziqian","middleName":"","lastName":"Liao","suffix":""},{"id":331772028,"identity":"09ea0eb5-984d-4de8-8c8d-5ddf324442ab","order_by":3,"name":"Valentina Carducci","email":"","orcid":"","institution":"Mayo Clinic","correspondingAuthor":false,"prefix":"","firstName":"Valentina","middleName":"","lastName":"Carducci","suffix":""},{"id":331772029,"identity":"b660fb1e-ea1f-45ef-bcca-3b80a0ddb9f3","order_by":4,"name":"Santiago Romero-Brufau","email":"","orcid":"","institution":"Mayo Clinic","correspondingAuthor":false,"prefix":"","firstName":"Santiago","middleName":"","lastName":"Romero-Brufau","suffix":""},{"id":331772032,"identity":"4273c3d0-0cbb-45c5-b0f9-9c242d88be74","order_by":5,"name":"Jie Yang","email":"","orcid":"","institution":"Harvard Medical School","correspondingAuthor":false,"prefix":"","firstName":"Jie","middleName":"","lastName":"Yang","suffix":""},{"id":331772033,"identity":"11ac3fb4-a76e-4b04-b5b7-066107284321","order_by":6,"name":"Rishi J Desai","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA5klEQVRIiWNgGAWjYFCCBCA2sElgkGBgQ4jwENaSRrIWhsMkaDFvTz784UfB+TyD2w1sD37uOSxv3p7A+OBtG24tMmeepUn2GNwuNrhzgN2w59lhwzlnHjAbzsWjRUIix4yBx+B24rYbCWwSPAfSGGdIJLBJ8+LVkv/54x+Dc2Atkn8OpNkDtbD/xq8lh0Gax+AAWIs0zwGbRJAtzHi18Dwzk5YxSC62v5HYJi1zwCZ5Bs/DZsk55/BoYU9+/PHNH7s8yRnJxyTfHJCwncGefPDDmzLcWpAAYwM6YxSMglEwCkYBuQAAdUdR/t0LCJkAAAAASUVORK5CYII=","orcid":"","institution":"Harvard Medical School","correspondingAuthor":true,"prefix":"","firstName":"Rishi","middleName":"J","lastName":"Desai","suffix":""}],"badges":[],"createdAt":"2024-07-09 21:56:19","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4714451/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4714451/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1186/s12874-025-02470-z","type":"published","date":"2025-01-28T15:58:04+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":62190318,"identity":"a594b733-a48d-4378-b7c0-f0d50740e6f7","added_by":"auto","created_at":"2024-08-10 12:26:46","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":65175,"visible":true,"origin":"","legend":"\u003cp\u003eThe default and the refined pipeline\u003c/p\u003e","description":"","filename":"Figure1.png","url":"https://assets-eu.researchsquare.com/files/rs-4714451/v1/ad831e664893a0e1a4dd2e53.png"},{"id":62188932,"identity":"e2d1959a-09af-4305-a974-4cbce9156940","added_by":"auto","created_at":"2024-08-10 12:18:46","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":73320,"visible":true,"origin":"","legend":"\u003cp\u003e(a). Average accuracies comparison of the LLMs over the baseline (Refined pipeline)\u003c/p\u003e\n\u003cp\u003e(b). Average accuracies comparison of the LLMs over the baseline (Default pipeline)\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-4714451/v1/63e63097ea7fec3f8ef8a3cc.png"},{"id":62190866,"identity":"4f45cacd-421a-45f5-b410-90e77eb9cfa7","added_by":"auto","created_at":"2024-08-10 12:34:46","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":39396,"visible":true,"origin":"","legend":"\u003cp\u003eAverage Macro F1 score of LLMs\u003c/p\u003e","description":"","filename":"Figure3.png","url":"https://assets-eu.researchsquare.com/files/rs-4714451/v1/468cc6730d81df472480f635.png"},{"id":62188930,"identity":"8cd0f4e4-aff1-4b68-b6e1-e24338c44f31","added_by":"auto","created_at":"2024-08-10 12:18:46","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":36431,"visible":true,"origin":"","legend":"\u003cp\u003eInvalid response rate comparison between the default and the refined pipeline\u003c/p\u003e","description":"","filename":"Figure4.png","url":"https://assets-eu.researchsquare.com/files/rs-4714451/v1/b866f6a620bb4ee238d5df5f.png"},{"id":75352049,"identity":"02a9dbad-92f3-4443-b8f6-29d09389cc9f","added_by":"auto","created_at":"2025-02-03 16:12:58","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":950853,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4714451/v1/5c9a2caf-fe98-4dde-9efb-0f6c25e07392.pdf"},{"id":62188934,"identity":"b9cac67a-e0f1-45d4-91a0-6c60cb0764a3","added_by":"auto","created_at":"2024-08-10 12:18:46","extension":"docx","order_by":7,"title":"","display":"","copyAsset":false,"role":"supplement","size":106015,"visible":true,"origin":"","legend":"","description":"","filename":"Supplimentary.docx","url":"https://assets-eu.researchsquare.com/files/rs-4714451/v1/e1e52b9cfc7210730525849d.docx"}],"financialInterests":"Competing interest reported. Dr. Desai reports serving as Principal Investigator on investigator-initiated grants to the Brigham and Women’s Hospital from Novartis, Vertex, and Bayer on unrelated projects.\nOther authors do not have any competing interests to disclose.","formattedTitle":"Scalable information extraction from free text electronic health records using large language models","fulltext":[{"header":"BACKGROUND","content":"\u003cp\u003eA vast amount of potentially useful information such as description of patient symptoms, family and social history, is not recorded as structured fields but found in free-text notes from electronic health records (EHRs). Substantial feature engineering is needed to directly extract and analyze the relevant information contained within free-text notes, which is time and resource intensive.\u003csup\u003e[\u003cspan additionalcitationids=\"CR2\" citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]\u003c/sup\u003e Therefore, free-text notes have historically been underutilized for research activities.\u003csup\u003e[\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]\u003c/sup\u003e\u003c/p\u003e \u003cp\u003eVarious Natural Language Processing (NLP) models have seen developed to extract clinical concepts from EHR notes automatically.\u003csup\u003e[\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e]\u003c/sup\u003e Traditional rule-based NLP approaches are often constrained by their lack of generalizability across different datasets and settings.\u003csup\u003e[\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]\u003c/sup\u003e Machine learning-based NLP models rely heavily on the costly step of manual data annotation for model training, which makes them harder to scale across diverse concepts, particularly in clinical environments where data annotation requires specialized knowledge, cost and confidentiality considerations.\u003csup\u003e[\u003cspan additionalcitationids=\"CR10\" citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]\u003c/sup\u003e In contrast, Large Language Models (LLMs), advanced deep learning models that are pre-trained on large volumes of text, provide a scalable alternative for the task of clinical information extraction.\u003csup\u003e[\u003cspan additionalcitationids=\"CR13\" citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e]\u003c/sup\u003e With their capabilities of zero-shot and few-shot learning, LLMs can extract target clinical information from EHR notes without the need for complex rules creation or extensive data annotation.\u003csup\u003e[\u003cspan additionalcitationids=\"CR16\" citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]\u003c/sup\u003e\u003c/p\u003e \u003cp\u003eSocial determinants of health (SDoH) of patients are the economic and social conditions that influence individual and group differences in health status.\u003csup\u003e[\u003cspan additionalcitationids=\"CR19\" citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e]\u003c/sup\u003e They are important for assessing and addressing health disparities, and are critical for clinical interventions and research.\u003csup\u003e[\u003cspan additionalcitationids=\"CR22\" citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e]\u003c/sup\u003e For instance, quantifying the role SDoH in predisposing patients to adverse health outcomes is often of interest to facilitate development of educational and counseling interventions.\u003csup\u003e[\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e]\u003c/sup\u003e In epidemiologic investigations, various domains of SDoH are also frequently considered important confounding variables that require adjustment.\u003csup\u003e[\u003cspan additionalcitationids=\"CR27\" citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e]\u003c/sup\u003e While structured EHRs typically lack explicit recording of SDoH information, it is frequently available in free-text form in notes.\u003csup\u003e[\u003cspan additionalcitationids=\"CR30\" citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e]\u003c/sup\u003e As variables of universal interest, we selected SDoH as a representative use case where scalable extraction methods like LLMs can be of high value. In the current study, we aimed to assess an \u0026ldquo;out of the box\u0026rdquo; implementation of 8 advanced open-source LLMs without any fine-tuning for their capabilities on extracting 9 SDoH from free-text EHRs, and compare their performance with the traditional approach of using basic pattern matching.\u003c/p\u003e"},{"header":"METHODS","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003eData Source\u003c/h2\u003e \u003cp\u003eWe used EHR data from the Mass General Brigham (MGB) system, which is the largest healthcare system in the state of Massachusetts. The sampling frame included a total of 1.2\u0026nbsp;million individuals with deterministically linked EHRs from MGB to insurance claims data from Medicare and Medicaid for the period of 2007\u0026ndash;2020. For this study, we used free text associated with the patients\u0026rsquo; social history documentation from progress notes using regular expression matching. Since the patients\u0026rsquo; social documentation is added incrementally over time, we only used the most recent social documentation for each patient.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003eSDoH Questions and Manual Labels\u003c/h2\u003e \u003cp\u003eTo identify the SDoH that are frequently reported in social documentation, we first performed a manual review of a random sample of 200 patients\u0026rsquo; social documentation and summarized 9 aspects of the patient\u0026rsquo;s SDoH that appeared in \u0026gt;\u0026thinsp;5% of the notes. This list included marital status, number of children, employment status, educational status, lifestyle factors (use of tobacco, alcohol, illicit drugs, exercise), and cohabitation status. Of the random sample of the 200 reviewed patients, we split the first 100 patients\u0026rsquo; social documentation as the validation set to inform prompt engineering (described below) and the remaining 100 patients\u0026rsquo; social documentation as the test set to evaluate performance. Following the classic evaluation framework of LLM evaluation, we converted the SDoH extraction as a question-answer problem. That is, for each of the 9 SDoH characteristics, given the EHR notes, we designed a question and candidate options along with the note text as the LLM input, and let the LLM to select the option from the candidates. The SDoH questions, together with their distribution of candidate options in the validation and the test set, are shown in \u003cb\u003eSupplementary Table \u003cspan refid=\"MOESM1\" class=\"InternalRef\"\u003eS1\u003c/span\u003e\u003c/b\u003e.\u003c/p\u003e \u003cp\u003eFor each of the 200 patients, two human reviewers (B.G and V.S) manually labeled the 9 SDoH aspects to one of the quantified choices according to the labeling criteria documentation (\u003cb\u003eSupplementary\u003c/b\u003e: \u003cb\u003eAnnotation Guide\u003c/b\u003e section). Each reviewer labeled the 200 patients independently. The inter-annotator agreement was calculated based on the total number of the 1800 questions (9 SdoH questions * 200 patients) that the two annotators agreed before discussion among all the 1800 questions, which was 93%. For the inconsistently annotated cases, the two annotators discussed them in detail and reached a consensus. New criteria were also added to the labeling criteria documentation that addressed the causes of these inconsistencies.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003eExperiment Settings\u003c/h2\u003e \u003cp\u003eWe selected 8 well performing open source LLMs on the LLM leaderboard hosted by Hugging Face.\u003csup\u003e[\u003cspan additionalcitationids=\"CR33 CR34 CR35 CR36 CR37 CR38 CR39 CR40\" citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR41\" class=\"CitationRef\"\u003e41\u003c/span\u003e]\u003c/sup\u003e All LLMs used in this study are publicly available. The details of the LLMs and the links to the models can be found in \u003cb\u003eSupplementary Table S2\u003c/b\u003e. A copy of the model weights went through the AWQ quantization process to generate the quantized model weights.\u003csup\u003e[\u003cspan citationid=\"CR42\" class=\"CitationRef\"\u003e42\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR43\" class=\"CitationRef\"\u003e43\u003c/span\u003e]\u003c/sup\u003e Quantization is a technique of reducing the model size for faster inference in resource limited settings.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003eRule-based Baseline Model\u003c/h2\u003e \u003cp\u003eTo evaluate the comparative performance of LLMs against a common baseline, we designed a model that used pattern matching to extract the answers to the SDoH questions from the patients\u0026rsquo; social documentation. The matching patterns were designed according to the labeling criteria. If a match was found in the patients\u0026rsquo; social documentation, the output answer was guaranteed to be one of the choices of the SDoH question. If no match was found in the patients\u0026rsquo; social documentation, the output answer was \u0026ldquo;Not mentioned\u0026rdquo;. To avoid mismatching (e.g. answer \u0026ldquo;No\u0026rdquo; was matched to \u0026ldquo;Not mentioned\u0026rdquo; since \u0026ldquo;No\u0026rdquo; was a substring of \u0026ldquo;Not mentioned\u0026rdquo;), we sorted the choices by their character lengths in descending order and matched the response from the longest choice to the shortest choice and stopped matching if the longer choice was matched. The specific patterns for each SDoH question are shown in \u003cb\u003eSupplementary Table S3.\u003c/b\u003e\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003ePipeline Workflow and prompt engineering\u003c/h2\u003e \u003cp\u003eWe built two pipelines: a default pipeline and a refined pipeline. The default pipeline was designed to run the LLM to extract the SDoHs from the unstructured social documentation using the default prompt for all SDoH questions. Alternatively, the refined pipeline used different engineered prompts on 3 (Q2, Q6, and Q7) of the 9 SDoH questions that most LLMs struggled with in the validation set experiments. We ran both pipelines to compare the effectiveness of the refinement. An illustration of the two pipelines is shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e. The \u0026ldquo;LLM Response Postprocessing\u0026rdquo; step included implementing a systematic code that uses pattern matching to map the model response to one of the choices of the SDoH questions. The \u0026ldquo;Auto-Grader\u0026rdquo; took the refined response from the \u0026ldquo;LLM Response Postprocessing\u0026rdquo; procedure and compared it against manual labels. When mapped model response matched exactly with the human label, then the auto-grader considered it as an accurate extraction. Otherwise, the auto-grader considered the LLM extraction inaccurate. The \u0026ldquo;Model Comparator\u0026rdquo; step combined the graded model responses from the \u0026ldquo;Auto-Grader\u0026rdquo; and provided the grading results in a single chart to formulate the final benchmark.\u003c/p\u003e \u003cp\u003eAdditional context for the \u0026ldquo;Default Prompt\u0026rdquo; and the \u0026ldquo;Engineered Prompt\u0026rdquo; is shown in \u003cb\u003eSupplementary Table S4\u003c/b\u003e, which summarizes 4 types of prompts: default prompts (not including the default secondary prompts), premise prompts, special prompts, and secondary prompts. The default pipeline only used the default prompts while the refined pipeline used all 4 types of prompts.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003eLLM performance evaluation\u003c/h2\u003e \u003cp\u003eTo evaluate the model performance, we used three metrics in the test set: Accuracy\u003csub\u003eoverall\u003c/sub\u003e, Accuracy\u003csub\u003ementioned\u003c/sub\u003e, and Accuracy\u003csub\u003enon\u0026minus;mentioned\u003c/sub\u003e, which corresponds to the overall accuracy for extractions, the accuracy when a note contained mention of the specific SDoH, and the accuracy when a note did not contain mention of the specific SDoH, respectively. The three accuracies are defined as follows, where the Accuracy\u003csub\u003eoverall\u003c/sub\u003e is a weighted average of the Accuracy\u003csub\u003ementioned\u003c/sub\u003e and the Accuracy\u003csub\u003enon\u0026minus;mentioned\u003c/sub\u003e, with the weights dependent on the missingness of the SDoH aspects in the text:\u003cdiv id=\"Equa\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equa\" name=\"EquationSource\"\u003e\n$$\\:{Accuracy}_{overall}\\:=\\:\\frac{Total\\:\\#\\:ofaccurate\\:extractions}{Total\\:\\#\\:of\\:questions}\\:$$\u003c/div\u003e\u003c/div\u003e\u003cdiv id=\"Equb\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equb\" name=\"EquationSource\"\u003e\n$$\\:{Accuracy}_{mentioned}\\:=\\:\\frac{Total\\:\\#\\:of\\:accurate\\:extractions\\:where\\:the\\:ground\\:truth\\:is\\:NOT\\:labeled\\:as\\:\u0026quot;not\\:mentioned\u0026quot;}{Total\\:\\#\\:of\\:questions\\:where\\:the\\:ground\\:truth\\:is\\:NOT\\:labeled\\:as\\:\u0026quot;not\\:mentioned\u0026quot;}$$\u003c/div\u003e\u003c/div\u003e\u003cdiv id=\"Equc\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equc\" name=\"EquationSource\"\u003e\n$$\\:{Accuracy}_{non-mentioned}=\\:\\frac{Total\\:\\#\\:of\\:accurate\\:extractions\\:where\\:the\\:ground\\:truth\\:is\\:labeled\\:as\\:\u0026quot;not\\:mentioned\u0026quot;}{Total\\:\\#\\:of\\:questions\\:where\\:the\\:ground\\:truth\\:is\\:labeled\\:as\\:\u0026quot;not\\:mentioned\u0026quot;}$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eWe calculated all three metrics on all 9 SDoH questions. To calculate the confidence interval for these accuracies, we used the Jackknife resampling technique to generate samples for each model accuracy on every question.\u003csup\u003e[\u003cspan citationid=\"CR44\" class=\"CitationRef\"\u003e44\u003c/span\u003e]\u003c/sup\u003e We then calculated the 95% CI of each accuracy using the samples generated. We assumed t-distribution since our sample size was small (100 per model, per question, per accuracy). For questions that did not meet the premise (e.g. The human label is \u0026ldquo;not mentioned\u0026rdquo; for this patient on this SDoH question when trying to calculate Accuracy\u003csub\u003ementioned\u003c/sub\u003e), we marked them as not applicable for calculations. To evaluate the performance difference between the LLM and the baseline, we also used the Jackknife resampling to generate samples for each model on each question on each accuracy. After that, we performed a two-sided Welch's t-test on the accuracy differences between the LLM and the baseline using the samples generated from the LLM and the baseline, assuming t-distribution.\u003c/p\u003e \u003cp\u003eAdditionally, when the post-processing procedure could not map the LLM response to one of the predefined choices, we defined the response to be invalid and reported the proportion of invalid responses for all models. We further reported the F1 score, calculated as a harmonic mean of precision and recall, using the macro averaging method considering all the SDoH questions as multi-class classification problems. When calculating F1 scores for LLM responses, we combined invalid responses with the \u0026lsquo;not mentioned\u0026rsquo; category for each question as a default since invalid responses from LLMs lacked a corresponding manual label.\u003c/p\u003e \u003c/div\u003e"},{"header":"RESULTS","content":"\u003cdiv id=\"Sec10\" class=\"Section2\"\u003e \u003ch2\u003eLLM Accuracy\u003c/h2\u003e \u003cp\u003eTo compare the performance between the LLMs and the baseline, we averaged the model Accuracy\u003csub\u003eoverall\u003c/sub\u003e across the 9 SDoH questions for each LLM and the baseline and did this for the three accuracies defined above (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e \u003cb\u003e(a)\u003c/b\u003e and \u003cb\u003e(b))\u003c/b\u003e. The baseline model achieved a 77.33% average Accuracy\u003csub\u003eoverall\u003c/sub\u003e, which was mainly attributable to its high average Accuracy\u003csub\u003enon\u0026minus;mentioned\u003c/sub\u003e. Among all the LLMs, openchat_3.5 was the only LLM that had a significantly better average Accuracy\u003csub\u003eoverall\u003c/sub\u003e than the baseline, it was also the only LLM that outperformed the baseline on each of the 9 SDoH questions on the average Accuracy\u003csub\u003eoverall\u003c/sub\u003e. On the other hand, the two Llama-2 models had the worst performance on the average Accuracy\u003csub\u003eoverall\u003c/sub\u003e and were significantly lower than the baseline. The remaining LLMs had comparable average Accuracy\u003csub\u003eoverall\u003c/sub\u003e than the baseline, with the zephyr-7b models performing slightly worse. The baseline, which only had limited patterns to match, had an average Accuracy\u003csub\u003ementioned\u003c/sub\u003e of 39.02%; while the majority of the LLMs achieved a higher average Accuracy\u003csub\u003ementioned\u003c/sub\u003e. Openchat_3.5 achieved over 40% higher of the average Accuracy\u003csub\u003ementioned\u003c/sub\u003e than the baseline, which shows its superior capability in extracting information when it is contained in free text. On the other hand, Llama-2-13b-chat was the only LLM that had a significantly worse average Accuracy\u003csub\u003ementioned\u003c/sub\u003e compared to the baseline.\u003c/p\u003e \u003cp\u003eThe Accuracy\u003csub\u003enon\u0026minus;mentioned\u003c/sub\u003e is the accuracy that indicates the model\u0026rsquo;s capability of not producing false information (i.e avoiding \u0026ldquo;hallucination\u0026rdquo;).\u003csup\u003e[\u003cspan additionalcitationids=\"CR46\" citationid=\"CR45\" class=\"CitationRef\"\u003e45\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR47\" class=\"CitationRef\"\u003e47\u003c/span\u003e]\u003c/sup\u003e The baseline, which reported its responses using pattern matching, had a very slim probability of producing false responses unless the information recorded in the text was internally contradicting. On the other hand, the LLMs had a higher probability of producing false information. We found that the baseline achieved an average Accuracy\u003csub\u003enon\u0026minus;mentioned\u003c/sub\u003e of 96.66%. This was reasonable as the default response from the baseline is \u0026ldquo;Not mentioned\u0026rdquo;. The openchat_3.5 model was the best model in terms of minimizing hallucination. On the contrary, the two models that had poor average Accuracy\u003csub\u003enon\u0026minus;mentioned\u003c/sub\u003e were the two Llama-2 models.\u003c/p\u003e \u003cp\u003eCompared to the default pipeline, the refined pipeline helped reduce LLM hallucination, but in some instances, this came at a cost of reduced sensitivity to the SDoH features mentioned in the text. This is expected as we used the premise prompt in the refined pipeline, which aims at reducing model hallucinations, but this prompt can make the model more conservative in its responses.\u003c/p\u003e \u003cp\u003eWe observed that among all the 9 SDoH questions, Q2 (How many children does the patient have?) and Q7 (What is the patient\u0026rsquo;s employment status?) were the two questions that most LLMs (except the two Llama 2 models) had a better Accuracy\u003csub\u003eoverall\u003c/sub\u003e than the baseline on both the default and the refined pipeline (\u003cb\u003eSupplementary Table S6-S7\u003c/b\u003e). This can be explained by \u003cb\u003eSupplementary Table S8-S9\u003c/b\u003e, which shows that the Accuracy\u003csub\u003ementioned\u003c/sub\u003e for the baseline on Q2 and Q7 were very low compared to the LLMs. Besides Q2, and Q7, \u003cb\u003eSupplementary Table S8-S9\u003c/b\u003e indicates that the baseline model also had very poor Accuracy\u003csub\u003ementioned\u003c/sub\u003e on SDoH questions Q3 (Does the patient currently use tobacco?) and Q5 (Does the patient currently use illicit drugs?). This is likely attributable to the fact that the unstructured social history documentation has many ways of expressing the answers on these questions, a simpler rule-based approach misses most of them. \u003cb\u003eSupplementary Table S10-S11\u003c/b\u003e indicates that the baseline had a much lower probability to produce false positive results compared to LLMs among all 9 SDoH questions. This eliminated the gains in accuracy by LLMs on the Accuracy\u003csub\u003ementioned\u003c/sub\u003e and makes the advantage of most LLMs (except the openchat_3.5 model) slim or even negative on the Accuracy\u003csub\u003eoverall\u003c/sub\u003e.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eFigure \u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e (a). Average accuracies comparison of the LLMs over the baseline (Refined pipeline)\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003eMacro F1 Score\u003c/h2\u003e \u003cp\u003eWe averaged the model macro F1 scores across the 9 SDoH questions for each LLM and the baseline (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e\u003cb\u003e).\u003c/b\u003e The baseline had 0.53 average macro F1 for the default pipeline and 0.54 average macro F1 for the refined pipeline. Among all the models, openchat_3.5 was the only LLM that had a clear advantage over the baseline. The two Vicuna models and the WizardLM model had comparable performance as the baseline. The zephyr model and the two Llama 2 models had worse average macro F1 scores than the baseline.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003eInvalid Responses\u003c/h2\u003e \u003cp\u003eFigure \u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e shows the comparison of the invalid responses between the refined and the default pipeline. It indicates that the refined pipeline greatly reduced the number of invalid responses for each LLM. Some examples of the models\u0026rsquo; invalid responses and the corresponding analysis are shown in \u003cb\u003eSupplementary Table S12\u003c/b\u003e.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"DISCUSSION","content":"\u003cp\u003eIn this work, we designed a pipeline that used various open-sourced LLMs to extract patients\u0026rsquo; SDoH based on social history documentation in free text EHRs. Using prompt engineering and postprocessing, we demonstrated that some LLMs achieved better performance than the baseline approach of pattern matching on most of the SDoH questions, especially when the answer to the SDoH question was mentioned in the text. We also observed that quantization, which could make these models more applicable in resource limited settings, did not substantially compromise the performance on extraction tasks.\u003c/p\u003e \u003cp\u003eOur study offers some practical learnings regarding using LLMs as scalable extraction tools from free text EHRs. First, thoughtful prompt engineering, as we conducted in our \u0026lsquo;refined\u0026rsquo; pipeline, can substantially elevate model performance, and should be routinely considered. Second, manual review of erroneous model responses indicated that in almost all such instances, the text was either controversial or required strong deductive reasoning to infer the correct answer, which is a known challenge for LLMs.\u003csup\u003e[\u003cspan additionalcitationids=\"CR49\" citationid=\"CR48\" class=\"CitationRef\"\u003e48\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR50\" class=\"CitationRef\"\u003e50\u003c/span\u003e]\u003c/sup\u003e We also found these cases had a high overlap with the cases reported by the human reviewers as ambiguous or controversial, and the reviewers needed to expand the labeling criteria to reach an agreement on such cases, which showed that such cases are also challenging to humans. Third, we documented clear and large differences between LLMs in this task of information extraction, which underscores the importance of evaluating multiple models when considering specific tasks. Our findings suggesting robust performance of \u0026ldquo;out of the box\u0026rdquo; implementation of LLMs without any fine-tuning are complementary to that of a prior report\u003csup\u003e[\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e]\u003c/sup\u003e, which demonstrated that when resources permit, fine-tuning LLMs for specific extraction tasks including SDOH information can be a viable alternative to further enhance performance.\u003c/p\u003e \u003cp\u003eOur study has important implications. The use of LLM pipelines for generating structured SDoH data from unstructured EHR notes could facilitate numerous downstream applications. One significant area is predictive analytics, where structured SDoH information can be routinely considered alongside structured clinical information to improve risk prediction for various health outcomes. This capability is particularly valuable in managing chronic diseases, where social and environmental factors, such as income and community, play a crucial role. Additionally, structured SDoH data can enhance population health management by enabling healthcare providers to identify and address social disparities in health. In clinical research, the availability of structured SDoH data can refine the selection of study cohorts, improve confounding adjustment in epidemiologic studies, and improve recruitment efforts in clinical trials that aim to oversample socioeconomically disadvantaged patient populations.\u003c/p\u003e \u003cp\u003eThe strengths of this work include the automation of the SDoH extraction process with high accuracy. As LLMs avoid the need for model training, this approach is likely scalable and transportable across institutions. Limitations include the complexity and subjectivity involved in the prompt engineering and LLM response postprocessing steps. Further, as most open source LLMs are trained using non-clinical text data, specific training on free-text EHRs maybe needed for extraction of more complex clinical concepts with LLMs and our observations regarding model performance based on general SDoH concepts may not extrapolate to other clinical feature extraction tasks.\u003c/p\u003e"},{"header":"CONCLUSIONS","content":"\u003cp\u003eIn conclusion, we demonstrated the feasibility of employing open weight LLMs to extract patients\u0026rsquo; SDoH with high accuracy without any additional finetuning. LLMs can offer effective and efficient information extraction from EHR text.\u003c/p\u003e"},{"header":"Abbreviations","content":"\u003cdiv class=\"DefinitionList\"\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eAWQ\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eActivation-aware Weight Quantization\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eEHR\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eElectronic Health Records\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eLLM\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eLarge Language Model\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eMGB\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eMass General Brigham\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eNLP\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eNatural Language Processing\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eSDoH\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eSocial Determinants of Health\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003c/div\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eEthics approval and consent to participate\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNot applicable.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConsent for publication\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNot applicable.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAvailability of data and materials\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe data that support the findings of this study are available from Mass General Brigham (MGB) but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. Data are however available from the authors upon reasonable request and with permission of Mass General Brigham (MGB).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting interests\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eDr. Desai reports serving as Principal Investigator on investigator-initiated grants to the Brigham and Women\u0026rsquo;s Hospital from Novartis, Vertex, and Bayer on unrelated projects.\u003c/p\u003e\n\u003cp\u003eOther authors do not have any competing interests to disclose.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThere\u0026nbsp;is\u0026nbsp;no source of funding for this study.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthors\u0026apos; contributions\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eBG and VS processed the data, labeled the data, analyzed the data, and drafted the manuscript. ZL contributed the idea of the secondary prompt, which significantly improved the study results. VC and SB contributed the idea of the pipelines in this study. JY and RD proposed the idea of the study, supervised the study, and refined the manuscript. All authors read and approved the final manuscript.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAcknowledgements\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNot applicable.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003ePetch J, Batt J, Murray J, Mamdani M. Extracting clinical features from dictated ambulatory consult notes using a commercially available natural language processing tool: pilot, retrospective, cross-sectional validation study. JMIR Med Inform.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eOzery-Flato M, Yanover C, Gottlieb A et al. Fast and efficient feature engineering for multi-cohort analysis of EHR data. Stud Health Technol Inform.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSoguero-Ru\u0026iacute;z C, Hindberg K, Rojo-\u0026Aacute;lvarez J et al. Support vector feature selection for early detection of anastomosis leakage from bag-of-words in electronic health records. IEEE J Biomed Health Inform.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKoleck TA, Dreisbach C, Bourne PE, Bakken S. Natural language processing of symptoms documented in free-text narratives of Electronic Health Records: A systematic review. J Am Med Inf Assoc. 2019;26(4):364\u0026ndash;79. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1093/jamia/ocy173\u003c/span\u003e\u003cspan address=\"10.1093/jamia/ocy173\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRe\u0026aacute;tegui Rojas R, Ratt\u0026eacute; S. Comparison of MetaMap and cTAKES for entity extraction in clinical notes. BMC Med Inform Decis Mak.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePeng J, Zhao M, Havrilla JM et al. Natural language processing (NLP) tools in extracting biomedical concepts from research articles: a case study on autism spectrum disorder. BMC Med Inform Decis Mak.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDasgupta I, Grant E, Griffiths TL. Distinguishing rule- and exemplar-based generalization in learning systems. arXiv.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eShi J, Hurdle JF. Trie-based rule processing for clinical NLP: A use-case study of n-trie, making the ConText algorithm more efficient and scalable. J Biomed Inform.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen Y, Mani S, Xu H. Applying active learning to assertion classification of concepts in clinical text. J Biomed Inf 45(2):265\u0026ndash;72.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSpasic I, Nenadic G. Clinical text data in machine learning: systematic review. JMIR Med Inf. 8.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLandolsi MY, Hlaoua L, Ben Romdhane L. Information extraction from Electronic Medical Documents: State of the art and Future Research Directions. Knowl Inf Syst. 2022;65(2):463\u0026ndash;516. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1007/s10115-022-01779-1\u003c/span\u003e\u003cspan address=\"10.1007/s10115-022-01779-1\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYang X, Pournejatian NM, Shin H-C et al. GatorTron: A large clinical language model to unlock patient information from unstructured electronic health records. medRxiv. 2022.02.27.22271257.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMeoni S, De la Clergerie E, Ryffel T. Large language models as instructors: A study on multilingual clinical entity extraction. In The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 178\u0026ndash;90, Toronto, Canada. Association for Computational Linguistics.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGuevara M, Chen S, Thomas S, et al. Large language models to identify social determinants of health in electronic health records. NPJ Digit Med. 2024;7:6.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAgrawal M, Hegselmann S, Lang H et al. Large language models are few-shot clinical information extractors. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1998\u0026ndash;2022, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMcInerney DJ, Young GS, van de Meent J-W, Wallace BC. CHiLL: Zero-shot custom interpretable feature extraction from clinical notes with large language models. arXiv:2302.12343.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSinghal K, Azizi S, Tu T, et al. Large language models encode clinical knowledge. Nature. 2023;620:172\u0026ndash;80. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1038/s41586-023-06291-2\u003c/span\u003e\u003cspan address=\"10.1038/s41586-023-06291-2\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMarmot M. Social determinants of health inequalities. Lancet. 2005;365:1099\u0026ndash;104.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBrown A. The community impact of social determinants of health. MOJ Public Health. 2016;4(3):98\u0026ndash;105. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.15406/mojph.2016.04.00083\u003c/span\u003e\u003cspan address=\"10.15406/mojph.2016.04.00083\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWypych-Ślusarska A, Głogowska-Ligus J, Słowiński J. Social determinants of health inequalities. Nurs Public Health.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWilliams DR, Costa M, Odunlami AO, Mohammed SA. Moving upstream: how interventions that address the social determinants of health can improve health and reduce disparities. J Public.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBraveman P, Egerter S, Williams DR. The social determinants of health: Coming of age. Annu Rev Public Health.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHewitt AM. Addressing health disparities: Understanding place in the role of social determinant interventions. arXiv.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAnderson E, Lippert SC, Newberry J, et al. Addressing social determinants of health from the emergency department through social emergency medicine. West J Emerg Med. 2016;17(3):487\u0026ndash;9.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFarthing H, Simko S, Eckembrecher FJ et al. Development and evaluation of electronic social needs assessment and resource connection tool in facilitating utilization of community services that address upstream health. J Student-Run Clin.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePoundstone K, Strathdee S, Celentano D. The social epidemiology of human immunodeficiency virus/acquired immunodeficiency syndrome. Epidemiol Rev 26:22\u0026ndash;35.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLiang Y, Gong Y, Wen X et al. Social determinants of health and depression: A preliminary investigation from rural China. PLoS ONE. 7.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eVineis P. Invited perspective: The mysterious case of social determinants of health. Environ Health Perspect. 130.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFeller DJ, Oliver JB, Zucker J et al. Detecting social and behavioral determinants of health with structured and free-text clinical data. Appl Clin Inform.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang MD, Pantell M, Gottlieb L, Adler-Milstein J. Documentation and review of social determinants of health data in the EHR: Measures and associated insights. J Am Med Inform Assoc.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYu Z, Yang X, Guo Y, Bian J, Wu Y. Assessing the documentation of social determinants of health for lung cancer patients in clinical narratives. Front Public Health.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBeeching E, Fourrier C, Habib N et al. Open LLM Leaderboard Hugging Face. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard\u003c/span\u003e\u003cspan address=\"https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGao L, Tow J, Biderman S et al. A framework for few-shot language model evaluation. Zenodo (Version v0.0.1). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.5281/zenodo.5371628\u003c/span\u003e\u003cspan address=\"10.5281/zenodo.5371628\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eClark P, Cowhey I, Etzioni O et al. Think you have solved question answering? Try ARC, the AI2 Reasoning Challenge. arXiv. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://arxiv.org/abs/1803.05457\u003c/span\u003e\u003cspan address=\"https://arxiv.org/abs/1803.05457\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZellers R, Holtzman A, Bisk Y et al. HellaSwag: Can a machine really finish your sentence? arXiv. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://arxiv.org/abs/1905.07830\u003c/span\u003e\u003cspan address=\"https://arxiv.org/abs/1905.07830\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHendrycks D, Burns C, Basart S et al. Measuring massive multitask language understanding. arXiv. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://arxiv.org/abs/2009.03300\u003c/span\u003e\u003cspan address=\"https://arxiv.org/abs/2009.03300\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLin S, Hilton J, Evans O, TruthfulQA. Measuring how models mimic human falsehoods. arXiv. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://arxiv.org/abs/2109.07958\u003c/span\u003e\u003cspan address=\"https://arxiv.org/abs/2109.07958\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSakaguchi K, Le Bras R, Bhagavatula C, Choi Y. WINOGRANDE: An adversarial Winograd schema challenge at scale. arXiv. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://arxiv.org/abs/1907.10641\u003c/span\u003e\u003cspan address=\"https://arxiv.org/abs/1907.10641\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCobbe K, Kosaraju V, Bavarian M et al. Training verifiers to solve math word problems. arXiv. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://arxiv.org/abs/2110.14168\u003c/span\u003e\u003cspan address=\"https://arxiv.org/abs/2110.14168\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eXu C, Sun Q, Zheng K et al. WizardLM: Empowering large language models to follow complex instructions. arXiv:2304.12244v2.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eVicuna. An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality. LMSYS Org. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://lmsys.org/blog/2023-03-30-vicuna/\u003c/span\u003e\u003cspan address=\"https://lmsys.org/blog/2023-03-30-vicuna/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. Accessed April 17, 2024.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLin J, AWQ. Activation-aware weight quantization for LLM compression and acceleration. Retrieved from \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://arxiv.org/abs/2306.00978\u003c/span\u003e\u003cspan address=\"https://arxiv.org/abs/2306.00978\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAWQ. Activation-aware weight quantization for LLM compression and acceleration - GitHub. GitHub. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://github.com/mit-han-lab/llm-awq\u003c/span\u003e\u003cspan address=\"https://github.com/mit-han-lab/llm-awq\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. Accessed April 11, 2024.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMcIntosh A. The jackknife estimation method. arXiv: Methodology.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHuang L, Yu W, Feng Z, Feng X. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. Retrieved from \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://arxiv.org/abs/2311.05232\u003c/span\u003e\u003cspan address=\"https://arxiv.org/abs/2311.05232\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJi Z, Yu T, Xu Y, et al. Towards mitigating hallucination in large language models via self-reflection. ArXiv. 2023. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.48550/arXiv.2310.06271\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.2310.06271\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. abs/2310.06271.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMcKenna N, Li T, Cheng L, Hosseini MJ, Johnson M, Steedman M. Sources of hallucination by large language models on inference tasks. ArXiv. 2023;2758\u0026ndash;74. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.48550/arXiv.2305.14552\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.2305.14552\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003eabs/2305.14552\u003c/span\u003e\u003cspan address=\"http://abs/2305.14552\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSeals SM, Shalin VL. Evaluating the deductive competence of large language models. Retrieved from \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://arxiv.org/abs/2309.05452\u003c/span\u003e\u003cspan address=\"https://arxiv.org/abs/2309.05452\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen K, Wang C, Yang K, et al. Gaining wisdom from setbacks: Aligning large language models via mistake analysis. ArXiv. 2023. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.48550/arXiv.2310.10477\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.2310.10477\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. abs/2310.10477.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLiu R, Shah NB. ReviewerGPT? An exploratory study on using large language models for paper reviewing. ArXiv. 2023. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.48550/arXiv.2306.00622\u003c/span\u003e\u003cspan address=\"10.48550/arXiv.2306.00622\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003eabs/2306.00622\u003c/span\u003e\u003cspan address=\"http://abs/2306.00622\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"bmc-medical-research-methodology","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"bmrm","sideBox":"Learn more about [BMC Medical Research Methodology](http://bmcmedresmethodol.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/bmrm/default.aspx","title":"BMC Medical Research Methodology","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Social Determinants of Health (SDoH), Electronic Health Records (EHR), Natural Language Processing (NLP), Large Language Models (LLMs), Clinical Information Extraction","lastPublishedDoi":"10.21203/rs.3.rs-4714451/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4714451/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eBackground\u003c/h2\u003e \u003cp\u003eA vast amount of potentially useful information such as description of patient symptoms, family, and social history is recorded as free-text notes in electronic health records (EHRs) but is difficult to reliably extract at scale, limiting their utility in research. This study aims to assess whether an \u0026ldquo;out of the box\u0026rdquo; implementation of open-source large language models (LLMs) without any fine-tuning can accurately extract social determinants of health (SDoH) data from free-text clinical notes.\u003c/p\u003e\u003ch2\u003eMethods\u003c/h2\u003e \u003cp\u003eWe conducted a cross-sectional study using EHR data from the Mass General Brigham (MGB) system, analyzing free-text notes for SDoH information. We selected a random sample of 200 patients and manually labeled nine SDoH aspects. Eight advanced open-source LLMs were evaluated against a baseline pattern-matching model. Two human reviewers provided the manual labels, achieving 93% inter-annotator agreement. LLM performance was assessed using accuracy metrics for overall, mentioned, and non-mentioned SDoH, and macro F1 scores.\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e \u003cp\u003eLLMs outperformed the baseline pattern-matching approach, particularly for explicitly mentioned SDoH, achieving up to 40% higher Accuracy\u003csub\u003ementioned\u003c/sub\u003e. openchat_3.5 was the best-performing model, surpassing the baseline in overall accuracy across all nine SDoH aspects. The refined pipeline with prompt engineering reduced hallucinations and improved accuracy.\u003c/p\u003e\u003ch2\u003eConclusions\u003c/h2\u003e \u003cp\u003eOpen-source LLMs are effective and scalable tools for extracting SDoH from unstructured EHRs, surpassing traditional pattern-matching methods. Further refinement and domain-specific training could enhance their utility in clinical research and predictive analytics, improving healthcare outcomes and addressing health disparities.\u003c/p\u003e","manuscriptTitle":"Scalable information extraction from free text electronic health records using large language models","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-08-10 12:18:41","doi":"10.21203/rs.3.rs-4714451/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2024-10-21T18:33:40+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-10-20T19:00:32+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-09-22T19:09:02+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"235256637873724856093448249011477561385","date":"2024-09-22T14:15:53+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"212947774680669947472498681455453624984","date":"2024-09-20T13:38:44+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"224811584244754023298283626115122752232","date":"2024-08-24T05:46:32+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"43179720924568610505334522146820030039","date":"2024-08-20T21:50:30+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-07-19T19:01:45+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2024-07-10T17:27:44+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-07-10T17:25:06+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-07-10T17:24:39+00:00","index":"","fulltext":""},{"type":"submitted","content":"BMC Medical Research Methodology","date":"2024-07-09T21:55:00+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"bmc-medical-research-methodology","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"bmrm","sideBox":"Learn more about [BMC Medical Research Methodology](http://bmcmedresmethodol.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/bmrm/default.aspx","title":"BMC Medical Research Methodology","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"47ee39f6-54a8-4d1a-8633-f9861b16e873","owner":[],"postedDate":"August 10th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2025-02-03T16:09:59+00:00","versionOfRecord":{"articleIdentity":"rs-4714451","link":"https://doi.org/10.1186/s12874-025-02470-z","journal":{"identity":"bmc-medical-research-methodology","isVorOnly":false,"title":"BMC Medical Research Methodology"},"publishedOn":"2025-01-28 15:58:04","publishedOnDateReadable":"January 28th, 2025"},"versionCreatedAt":"2024-08-10 12:18:41","video":"","vorDoi":"10.1186/s12874-025-02470-z","vorDoiUrl":"https://doi.org/10.1186/s12874-025-02470-z","workflowStages":[]},"version":"v1","identity":"rs-4714451","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4714451","identity":"rs-4714451","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}