Towards automated phenotype definition extraction using large language models

Towards automated phenotype definition extraction using large language models | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Short Report Towards automated phenotype definition extraction using large language models Ramya Tekumalla, Juan M. Banda This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4798033/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 9 You are reading this latest preprint version Abstract Electronic phenotyping involves a detailed analysis of both structured and unstructured data, employing rule-based methods, machine learning, natural language processing, and hybrid approaches. Currently, the development of accurate phenotype definitions demands extensive literature reviews and clinical experts, rendering the process time-consuming and inherently unscalable. Large Language Models offer a promising avenue for automating phenotype definition extraction but come with significant drawbacks, including reliability issues, the tendency to generate non-factual data ('hallucinations'), misleading results, and potential harm. To address these challenges, our study embarked on two key objectives: (1) defining a standard evaluation set to ensure Large Language Models outputs are both useful and reliable, and (2) evaluating various prompting approaches to extract phenotype definitions from Large Language Models, assessing them with our established evaluation task. Our findings reveal promising results that still require human evaluation and validation for this task. However, enhanced phenotype extraction is possible, reducing the amount of time spent in literature review and evaluation. ChatGPT Electronic Phenotyping Large Language Models (LLMs) Evaluation Introduction In the era of digital healthcare, the advent of electronic health records (EHRs) and the proliferation of digital health data are catalyzing a paradigm shift in medical research and patient care. At the heart of this transformation is electronic phenotyping, a process that utilizes these vast datasets to identify and classify patient phenotypes. Phenotyping, in the context of biomedical research, refers to the process of extracting relevant health characteristics from patient data that can be correlated with specific health outcomes, diseases, or conditions. This process is critical for advancing personalized medicine, streamlining patient care, and driving forward biomedical discoveries. Electronic phenotyping harnesses both structured and unstructured data, integrating rule-based systems, machine learning techniques, natural language processing (NLP), and hybrid methodologies to analyze and categorize patient information [ 1 ] . The significance of electronic phenotyping cannot be overstated; it forms the cornerstone of efforts to tailor healthcare to individual patient needs [ 2 ], enhance the understanding of disease mechanisms[ 3 ], and facilitate the development of novel therapeutic interventions. However, the scalability of electronic phenotyping poses a formidable challenge. Currently, defining a phenotype requires exhaustive literature reviews and intensive collaboration among clinicians, domain experts, and researchers to achieve consensus on precise phenotype definitions[ 4 ]. This iterative and collaborative process is time-consuming and resource-intensive, making the current approach to phenotyping less scalable and adaptable to the fast-paced advancements in medical research and emerging health crises. As the volume of digital health data explodes and the complexity of diseases becomes more apparent, the ability to quickly and accurately define, refine, and utilize phenotypes is paramount [ 5 ]. Scalable and portable phenotyping processes can accelerate the pace of research, enable the rapid identification of patient cohorts for clinical trials, and improve the detection and treatment of diseases at an individual level [ 6 ]. Leveraging machine learning for electronic phenotyping introduces a scalable approach to processing and interpreting healthcare data, fundamentally shifting the paradigm from manual, labor-intensive methods to automated, data-driven insights[ 7 ] [ 8 ] [ 9 ]. In the beginning, both rule based and machine learning models were utilized to identify phenotypes[ 10 ]. However, the expansion of large language models (LLMs) to include hundreds of billions of parameters has introduced new abilities like few-shot learning[ 11 ]. This development enables LLMs to achieve good performance on tasks with minimal training, using only a small number of examples[ 12 ]. Several large language models like PhenoBCBERT and PhenoGPT are accurately able to infer essential phenotypic information from the given context[ 13 ]. This rapid increase in experimentation with LLMs has created pathways for researchers to utilize LLMs for electronic phenotyping. In this work, we propose an innovative approach to address the scalability challenge in electronic phenotyping. Our work is anchored in two main objectives: first, to define a standard evaluation task/set specifically tailored for this domain, and second, to evaluate various prompting approaches for extracting phenotype definitions from LLMs. The establishment of a standard evaluation task is crucial as it serves as a benchmark to ensure that the outputs produced by LLMs are not only useful but reliable. Following this, we explore and assess different prompting strategies to effectively extract phenotype definitions from LLMs, utilizing the evaluation task we have created. Additionally we focus on the behavior of LLMs. This dual approach represents a significant step forward in automating the phenotype definition process, leveraging the advanced capabilities of LLMs to interpret and generate natural language. By doing so, we aim to significantly reduce the time and effort currently required to define phenotypes, thereby enhancing the scalability and efficiency of electronic phenotyping. Our exploration into the use of LLMs for phenotype definition extraction is intended to pave the way for more scalable and adaptable phenotyping processes, ultimately accelerating innovation and improving outcomes in healthcare and biomedical research. Data Preparation The primary objective of this work is to create an evaluation set. We identified 10 professionally created phenotypes, five from PheKB [ 14 ] and five from the OHDSI phenotype library [ 15 ]. The extraction of the phenotypes from sources like OHDSI phenotype library and HDRUK phenotype library [ 16 ] are relatively easier as the phenotypes are in a structured format. OHDSI uses the OMOP (Observational Medical Outcomes Partnership) Common Data Model (CDM), which standardizes healthcare data into a consistent format, facilitating efficient and scalable analysis across different databases. This standardization reduces the complexity and effort required to extract and analyze phenotypes, as researchers can apply the same query across multiple datasets without needing to adjust for disparate data structures However, PheKB provides a platform for developing, validating, and sharing phenotype algorithms without mandating a specific data model. This approach offers flexibility and can accommodate a wide variety of data structures but may require more effort to adapt and apply algorithms across different EHR systems and databases. Hence, manual curation was required to format the PheKB phenotype definitions. We developed an automated computer code to automatically extract and format the elements from the phenotypes to facilitate automatic evaluation. Table 1 presents the extracted elements of the 10 professionally created phenotypes. Table 1 Extracted elements from the phenotypes Logic Vocabulary Concept Code Concept Name Code Count Inclusion SNOMED 194823009 Acute coronary insufficiency 1+ Inclusion SNOMED 791000119109 Angina associated with type 2 diabetes mellitus 1 Inclusion SNOMED 61490001 Angina, class I 1+ Inclusion SNOMED 41334000 Angina, class II 1 Inclusion SNOMED 85284003 Angina, class III 1+ Inclusion SNOMED 89323001 Angina, class IV 1+ Evaluation Setup One of our objectives in this work is to evaluate prompting approaches to extract phenotype definitions from LLMs and assess them using the evaluation set created in the Data Preparation section. We experimented with several prompts to create a prompt which can be utilized for extracting all elements of a phenotype and finally used a prompt which brought in relatively consistent results from the LLMs. We experimented with several prompting methods like Zero shot, One-shot, Iterative prompting, Seeding and finally developed a prompt. The following is our final prompt used for evaluation - “Provide a computational phenotype for with codes needed and their name, and logical conditions as well as how many codes are needed. In the following tabular format: Logic (inclusion or exclusion), code vocabulary, code identifier, code name, and code count.” To evaluate the efficiency of LLMs we considered 2 different scenarios. In the first scenario, we compared the definitions extracted by GPT 3.5 and GPT 4. In the second scenario, we compared the definitions extracted by GPT4 and manually curated definitions (by humans). In both the scenarios, we present the metrics on overlap of codes, overlap of strings and logical matching. Additionally, we measured the inconsistencies and incorrect definitions and presented them in our discussion. Results We present the results for scenario 1 and 2 in Tables 2 and 3 . We calculated the average, minimum and maximum percentage of each of the metrics (eg: codes overlap). Table 2 Comparison between GPT 3.5 vs GPT 4 Metric Average % Minimum % Maximum % Codes overlap 41.26 0.00 75.00 Logic overlap 80.00 50.00 100.00 Strings overlap 28.52 0.00 50.00 The key findings of this scenario indicate that GPT models are better in generating precise codes over textual strings. There is a considerable variability in the models’ outputs indicating a challenge in achieving consistent results across different iterations. An interesting result here is that LLMs demonstrate solid competency while extracting the logical conditions of inclusion/exclusion of codes in phenotype definitions. These insights show that one potential reason for the low overlap in codes and strings within definitions is the great variability of code systems used in phenotype definitions found in literature, and on the definitions themselves [ 17 ]. We theorize that papers and abstracts are part of the GPT model training sets and this is reflected in the inconsistent LLM output. Table 3 Comparison between human definition vs GPT models Model Metric Average % Minimum % Maximum % Codes overlap 50.94 20.00 88.89 GPT 4 Logic overlap 90.00 50.00 100.00 Strings overlap 48.59 0.00 100.00 Codes overlap 27.51 10.00 85.20 GPT 3.5 Logic overlap 70.20 0.00 90.00 Strings overlap 41.28 0.00 75.12 A noteworthy observation is that the codes generated by GPT-4 exhibit a marginally higher reliability compared to the textual strings or concept names. The codes denominator for the codes overlap metric is the number of codes from GPT4. Furthermore, despite the overall fewer codes generated by GPT-4, a closer examination suggests these codes may possess a higher positive predictive value (PPV) for accurately identifying the intended phenotypes. This finding suggests that while the volume of generated codes is limited, their specificity and relevance to the phenotypes are notably high, indicating that the model might be averaging out from source and could be surfacing the most popular ones. Table 4 , presents the GPT hallucinations with codes. In this work we compared the performance of GPT-3.5 and GPT-4 models in generating phenotype codes, using Biomedical Content Explorer[ 18 ] linked with PubDictionaries, ICD10 and ICD 10 CM dictionaries, with this comparison we show the biggest weakness of these LLM model as it is highly inaccurate and full of hallucinations. We discovered that hallucinations were notably present in both models, with GPT-3.5 showing a higher tendency towards these inaccuracies compared to GPT-4. These observations emphasize the imperative for cautious use and meticulous verification of data produced by LLMs, especially for phenotypes less documented in scientific literature. The pattern observed suggests a direct relationship between the scarcity of literature on specific phenotypes and the models' propensity to generate non-existent codes, pointing to a crucial area for enhancement in the training methodologies of these models. Table 4 Comparisons of GPT hallucinations when producing codes Model Average % Minimum % Maximum % GPT 3.5 38 0 83 GPT 4 32 0 69 Additionally, we performed a detailed evaluation of the capabilities of GPT-3.5 and GPT-4 models in accurately extracting phenotype definitions, a crucial step toward their integration into medical informatics. This entails an extensive series of evaluations comparing these Large Language Models (LLMs) against human-generated definitions to assess various aspects: the accuracy of code mapping, the consistency of code names, the logical structuring of definitions, and the degree of overlap in the codes identified. In our experiment, we compared human-generated definitions of phenotypes against those produced by GPT-4, focusing on the process of code mapping to align disparate coding systems into a unified framework. Table 5 presents the results of our evaluations Table 5 Code mapping evaluation of GPT models, comparing codes as extracted vs mapped by the LLM Model Metric Average % Minimum % Maximum % GPT 4 Extracted codes overlap 50.94 20.00 89.00 Mapped codes overlap 72.89 28.98 97.00 GPT 3.5 Extracted codes overlap 27.51 10.00 85.20 Mapped codes overlap 58.15 19.87 62.20 Discussion Our exploration into the utilization of GPT models for medical coding reveals several noteworthy challenges. This underscores the importance of carefully crafted prompts to ensure reliable and consistent results. The second challenge is the non-deterministic nature of generative LLMs. Identical prompts applied to different diseases generate stylistically different outputs, and depending on the prompting strategy, these lead to completely hallucinated responses. We included the screenshots of the GPT inaccuracies and hallucinations in the Supplementary Material section. Our findings show some promising results for certain phenotypes but not all. One of the bigger dangers here is the generation of hallucinations when asking for specific coding systems to standardize the definitions. While this could be easily overcome by using knowledge graphs [ 19 ], some codes are completely fabricated and will never map to anything, as recently shown by Soroush et al. [ 20 ]. Future Work The next phase of this work will involve actually using the extracted phenotype definitions by the GPT models and comparing the patient cohorts they select. We will use tools available within the OHDSI community, such as Cohort Diagnostics [ 21 ] and PheValuator [ 22 ] to observe if these definitions are even close to human generated definitions. Our hypothesis here is that since the LLM identifies the most commonly used codes, and logic, for these definitions, it should do a decent job in identifying the core of the phenotype cohort, without including additional edge cases, thus not leaving too many patients out. Success will be measured by the similarity between these cohorts if they closely align, it signifies a significant achievement. Conversely, substantial differences would indicate a need for further refinement of the models, highlighting the iterative nature of improving LLM applications in healthcare. This rigorous evaluation process is essential for advancing our understanding and application of AI in enhancing medical research and patient care. Conclusions Our exploration into utilizing LLMs for automating phenotype definition extraction presents a promising avenue for enhancing the scalability and efficiency of phenotyping in digital healthcare. While our results underscore the potential of LLMs, particularly GPT-3.5 and GPT-4, in generating medically relevant codes, they also highlight the challenges of consistency in textual output and the occurrence of inaccuracies. The critical insight from our study is the importance of developing robust evaluation and validation frameworks to ensure the reliability of LLM outputs. The findings indicate that despite the hallucinations and inconsistencies, GPT models hold potential value as an initial step or augmentation tool in the phenotyping process which could significantly streamline and enhance electronic phenotyping methodologies. Abbreviations Electronic Health Records (EHRs) Natural Language Processing (NLP) Large Language Models (LLMs) Phenotype KnowledgeBase (PheKB) Observational Health Data Sciences and Informatics (OHDSI) Health Data Research UK (HDRUK) Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM) Declarations Availability of data and materials Data and code is provided at the following repository: https://github.com/jmbanda/blah8 Competing interests No competing or financial interests Funding Travel support for BLAH8 was provided by the National Bioscience Database Center (NBDC) of Japan Science and Technology Agency (JST) and Research Organization of Information and Systems (ROIS). Authors' contributions RT and JMB wrote the main manuscript text, prepared figures and tables, designed and executed the experiments. All authors reviewed the manuscript. Acknowledgements This work has been completed at BLAH 8 References Banda JM, Seneviratne M, Hernandez-Boussard T, Shah NH (2018) Advances in Electronic Phenotyping: From Rule-Based Definitions to Machine Learning Models. Annu Rev Biomed Data Sci 1:53–68 Smoller JW (2018) The use of electronic health records for psychiatric phenotyping and genomics. Am J Med Genet B Neuropsychiatr Genet 177:601–612 Nadkarni GN, Gottesman O, Linneman JG, et al (2014) Development and validation of an electronic phenotyping algorithm for chronic kidney disease. AMIA Annu Symp Proc 2014:907–916 Weng C, Shah NH, Hripcsak G (2020) Deep phenotyping: Embracing complexity and temporality-Towards scalability, portability, and interoperability. J Biomed Inform 105:103433 Huckvale K, Venkatesh S, Christensen H (2019) Toward clinical digital phenotyping: a timely opportunity to consider purpose, quality, and safety. NPJ Digit Med 2:88 Rasmussen LV, Brandt PS, Jiang G, et al (2019) Considerations for Improving the Portability of Electronic Health Record-Based Phenotype Algorithms. AMIA Annu Symp Proc 2019:755–764 Agarwal V, Podchiyska T, Banda JM, Goel V, Leung TI, Minty EP, Sweeney TE, Gyang E, Shah NH (2016) Learning statistical models of phenotypes using noisy labeled training data. J Am Med Inform Assoc 23:1166–1173 Yang Z, Dehmer M, Yli-Harja O, Emmert-Streib F (2020) Combining deep learning with token selection for patient phenotyping from electronic health records. Sci Rep 10:1432 Beaulieu-Jones BK, Greene CS, Pooled Resource Open-Access ALS Clinical Trials Consortium (2016) Semi-supervised learning of the electronic health record for phenotype stratification. J Biomed Inform 64:168–178 Luo L, Yan S, Lai P-T, Veltri D, Oler A, Xirasagar S, Ghosh R, Similuk M, Robinson PN, Lu Z (2021) PhenoTagger: a hybrid method for phenotype concept recognition using human phenotype ontology. Bioinformatics 37:1884–1890 Brown T, Mann B, Ryder N, et al (2020) Language models are few-shot learners. Adv Neural Inf Process Syst 33:1877–1901 Tekumalla R, Banda JM (2023) Leveraging Large Language Models and Weak Supervision for Social Media Data Annotation: An Evaluation Using COVID-19 Self-reported Vaccination Tweets. In: HCI International 2023 – Late Breaking Papers. Springer Nature Switzerland, pp 356–366 Yang J, Liu C, Deng W, Wu D, Weng C, Zhou Y, Wang K (2024) Enhancing phenotype recognition in clinical notes using large language models: PhenoBCBERT and PhenoGPT. Patterns (N Y) 5:100887 Kirby JC, Speltz P, Rasmussen LV, et al (2016) PheKB: a catalog and workflow for creating electronic phenotype algorithms for transportability. J Am Med Inform Assoc 23:1046–1052 Banda JM, Halpern Y, Sontag D, Shah NH (2017) Electronic phenotyping with APHRODITE and the Observational Health Sciences and Informatics (OHDSI) data network. AMIA Jt Summits Transl Sci Proc 2017:48–57 HRDUK Phenotype Library. https://phenotypes.healthdatagateway.org/. Accessed 22 Mar 2024 Brandt PS, Kho A, Luo Y, et al (2023) Characterizing variability of electronic health record-driven phenotype definitions. J Am Med Inform Assoc 30:427–437 Kim J (2023) Biomedical Content Explorer. https://chat.openai.com/g/g-wdWOSr2gs-biomedical-content-explorer. Callahan TJ, Stefanski AL, Wyrwa JM, et al (2023) Ontologizing health systems data at scale: making translational discovery a reality. NPJ Digit Med 6:89 Soroush Ali, Glicksberg Benjamin S., Zimlichman Eyal, Barash Yiftach, Freeman Robert, Charney Alexander W., Nadkarni Girish N, Klang Eyal (2024) Large Language Models Are Poor Medical Coders — Benchmarking of Medical Code Querying. NEJM AI 1:AIdbp2300040 Gilbert J, Rao G, Schuemie M, Ryan P, Weaver J (2023) CohortDiagnostics: Diagnostics for OHDSI Cohorts. Swerdel JN, Hripcsak G, Ryan PB (2019) PheValuator: Development and evaluation of a phenotype algorithm evaluator. J Biomed Inform 97:103258 Additional Declarations No competing interests reported. Supplementary Files BLAH8supplemental.docx Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 26 Jul, 2024 Reviews received at journal 26 Jul, 2024 Reviewers agreed at journal 16 Jul, 2024 Reviews received at journal 28 Jun, 2024 Reviewers agreed at journal 16 Jun, 2024 Reviewers invited by journal 15 Jun, 2024 Editor assigned by journal 03 Jun, 2024 Submission checks completed at journal 02 Jun, 2024 First submitted to journal 31 May, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4798033","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Short Report","associatedPublications":[],"authors":[{"id":332248895,"identity":"67465d8c-7d74-4706-8adc-e17d7c45ef9f","order_by":0,"name":"Ramya Tekumalla","email":"","orcid":"","institution":"Mercer University","correspondingAuthor":false,"prefix":"","firstName":"Ramya","middleName":"","lastName":"Tekumalla","suffix":""},{"id":332248896,"identity":"bf772272-8bc0-4f0b-a937-e7715af16113","order_by":1,"name":"Juan M. Banda","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA5ElEQVRIiWNgGAWjYBACxgYeIMnGIAfmVYAEGBgMiNJiDMQMDGeI0cLAANGS2EC0FuZpZw9+LiizS58/v/nZgwMV92Qb2Ju3SeB12Oy8ZOkZ55JzNxxjMzc4cKbYuIHnWBkBLTkG0rxtzLkb2BjMpD+2JSQ2SOSYEdJi/Ju3rT5dvo39m8RBkBb5NwS1mAFtOZzAcIzHDKJFgoeQlrw0a55zxw03HMspkzhwJsG4jSet2AKfFsPZuYdv85RVy8s3H98mcaAiQbaf/fDGG3i1NKCLsOFTDgLyhBSMglEwCkbBKGAAALWoSSSDTD1HAAAAAElFTkSuQmCC","orcid":"","institution":"Stanford Health Care","correspondingAuthor":true,"prefix":"","firstName":"Juan","middleName":"M.","lastName":"Banda","suffix":""}],"badges":[],"createdAt":"2024-07-24 23:53:18","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4798033/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4798033/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":62924022,"identity":"7a4f7a66-f3b7-491f-adbb-2232d75f978a","added_by":"auto","created_at":"2024-08-21 06:18:44","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":356705,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4798033/v1/3258b809-c782-45c7-a722-7af25de005f3.pdf"},{"id":62924021,"identity":"5259928b-a1f3-49a4-9d9d-d9f57af634d8","added_by":"auto","created_at":"2024-08-21 06:18:39","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":608076,"visible":true,"origin":"","legend":"","description":"","filename":"BLAH8supplemental.docx","url":"https://assets-eu.researchsquare.com/files/rs-4798033/v1/b0bb948cb2d033fc74651ed1.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Towards automated phenotype definition extraction using large language models","fulltext":[{"header":"Introduction","content":"\u003cp\u003eIn the era of digital healthcare, the advent of electronic health records (EHRs) and the proliferation of digital health data are catalyzing a paradigm shift in medical research and patient care. At the heart of this transformation is electronic phenotyping, a process that utilizes these vast datasets to identify and classify patient phenotypes. Phenotyping, in the context of biomedical research, refers to the process of extracting relevant health characteristics from patient data that can be correlated with specific health outcomes, diseases, or conditions. This process is critical for advancing personalized medicine, streamlining patient care, and driving forward biomedical discoveries. Electronic phenotyping harnesses both structured and unstructured data, integrating rule-based systems, machine learning techniques, natural language processing (NLP), and hybrid methodologies to analyze and categorize patient information [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e] .\u003c/p\u003e \u003cp\u003eThe significance of electronic phenotyping cannot be overstated; it forms the cornerstone of efforts to tailor healthcare to individual patient needs [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e], enhance the understanding of disease mechanisms[\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e], and facilitate the development of novel therapeutic interventions. However, the scalability of electronic phenotyping poses a formidable challenge. Currently, defining a phenotype requires exhaustive literature reviews and intensive collaboration among clinicians, domain experts, and researchers to achieve consensus on precise phenotype definitions[\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. This iterative and collaborative process is time-consuming and resource-intensive, making the current approach to phenotyping less scalable and adaptable to the fast-paced advancements in medical research and emerging health crises. As the volume of digital health data explodes and the complexity of diseases becomes more apparent, the ability to quickly and accurately define, refine, and utilize phenotypes is paramount [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. Scalable and portable phenotyping processes can accelerate the pace of research, enable the rapid identification of patient cohorts for clinical trials, and improve the detection and treatment of diseases at an individual level [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eLeveraging machine learning for electronic phenotyping introduces a scalable approach to processing and interpreting healthcare data, fundamentally shifting the paradigm from manual, labor-intensive methods to automated, data-driven insights[\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e] [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e] [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. In the beginning, both rule based and machine learning models were utilized to identify phenotypes[\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. However, the expansion of large language models (LLMs) to include hundreds of billions of parameters has introduced new abilities like few-shot learning[\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]. This development enables LLMs to achieve good performance on tasks with minimal training, using only a small number of examples[\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. Several large language models like PhenoBCBERT and PhenoGPT are accurately able to infer essential phenotypic information from the given context[\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e]. This rapid increase in experimentation with LLMs has created pathways for researchers to utilize LLMs for electronic phenotyping.\u003c/p\u003e \u003cp\u003eIn this work, we propose an innovative approach to address the scalability challenge in electronic phenotyping. Our work is anchored in two main objectives: first, to define a standard evaluation task/set specifically tailored for this domain, and second, to evaluate various prompting approaches for extracting phenotype definitions from LLMs. The establishment of a standard evaluation task is crucial as it serves as a benchmark to ensure that the outputs produced by LLMs are not only useful but reliable. Following this, we explore and assess different prompting strategies to effectively extract phenotype definitions from LLMs, utilizing the evaluation task we have created. Additionally we focus on the behavior of LLMs. This dual approach represents a significant step forward in automating the phenotype definition process, leveraging the advanced capabilities of LLMs to interpret and generate natural language. By doing so, we aim to significantly reduce the time and effort currently required to define phenotypes, thereby enhancing the scalability and efficiency of electronic phenotyping. Our exploration into the use of LLMs for phenotype definition extraction is intended to pave the way for more scalable and adaptable phenotyping processes, ultimately accelerating innovation and improving outcomes in healthcare and biomedical research.\u003c/p\u003e\n\u003ch3\u003eData Preparation\u003c/h3\u003e\n\u003cp\u003eThe primary objective of this work is to create an evaluation set. We identified 10 professionally created phenotypes, five from PheKB [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e] and five from the OHDSI phenotype library [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]. The extraction of the phenotypes from sources like OHDSI phenotype library and HDRUK phenotype library [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e] are relatively easier as the phenotypes are in a structured format. OHDSI uses the OMOP (Observational Medical Outcomes Partnership) Common Data Model (CDM), which standardizes healthcare data into a consistent format, facilitating efficient and scalable analysis across different databases. This standardization reduces the complexity and effort required to extract and analyze phenotypes, as researchers can apply the same query across multiple datasets without needing to adjust for disparate data structures However, PheKB provides a platform for developing, validating, and sharing phenotype algorithms without mandating a specific data model. This approach offers flexibility and can accommodate a wide variety of data structures but may require more effort to adapt and apply algorithms across different EHR systems and databases. Hence, manual curation was required to format the PheKB phenotype definitions. We developed an automated computer code to automatically extract and format the elements from the phenotypes to facilitate automatic evaluation. Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e presents the extracted elements of the 10 professionally created phenotypes.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eExtracted elements from the phenotypes\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLogic\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eVocabulary\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eConcept Code\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eConcept Name\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eCode Count\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eInclusion\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSNOMED\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e194823009\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eAcute coronary insufficiency\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e1+\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eInclusion\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSNOMED\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e791000119109\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eAngina associated with type 2 diabetes mellitus\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eInclusion\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSNOMED\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e61490001\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eAngina, class I\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e1+\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eInclusion\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSNOMED\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e41334000\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eAngina, class II\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eInclusion\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSNOMED\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e85284003\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eAngina, class III\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e1+\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eInclusion\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSNOMED\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e89323001\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eAngina, class IV\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e1+\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e"},{"header":"Evaluation Setup","content":"\u003cp\u003eOne of our objectives in this work is to evaluate prompting approaches to extract phenotype definitions from LLMs and assess them using the evaluation set created in the \u003cspan refid=\"Sec2\" class=\"InternalRef\"\u003eData Preparation\u003c/span\u003e section. We experimented with several prompts to create a prompt which can be utilized for extracting all elements of a phenotype and finally used a prompt which brought in relatively consistent results from the LLMs. We experimented with several prompting methods like Zero shot, One-shot, Iterative prompting, Seeding and finally developed a prompt. The following is our final prompt used for evaluation - \u0026ldquo;Provide a computational phenotype for \u0026lt;\u0026thinsp;INSERT_PHENOTYPE\u0026thinsp;\u0026gt;\u0026thinsp;with codes needed and their name, and logical conditions as well as how many codes are needed. In the following tabular format: Logic (inclusion or exclusion), code vocabulary, code identifier, code name, and code count.\u0026rdquo;\u003c/p\u003e \u003cp\u003eTo evaluate the efficiency of LLMs we considered 2 different scenarios. In the first scenario, we compared the definitions extracted by GPT 3.5 and GPT 4. In the second scenario, we compared the definitions extracted by GPT4 and manually curated definitions (by humans). In both the scenarios, we present the metrics on overlap of codes, overlap of strings and logical matching. Additionally, we measured the inconsistencies and incorrect definitions and presented them in our discussion.\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003eWe present the results for scenario 1 and 2 in Tables\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e and \u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e. We calculated the average, minimum and maximum percentage of each of the metrics (eg: codes overlap).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eComparison between GPT 3.5 vs GPT 4\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMetric\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAverage %\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eMinimum %\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eMaximum %\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCodes overlap\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e41.26\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e75.00\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLogic overlap\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e80.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e50.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e100.00\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eStrings overlap\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e28.52\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e50.00\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eThe key findings of this scenario indicate that GPT models are better in generating precise codes over textual strings. There is a considerable variability in the models\u0026rsquo; outputs indicating a challenge in achieving consistent results across different iterations. An interesting result here is that LLMs demonstrate solid competency while extracting the logical conditions of inclusion/exclusion of codes in phenotype definitions. These insights show that one potential reason for the low overlap in codes and strings within definitions is the great variability of code systems used in phenotype definitions found in literature, and on the definitions themselves [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. We theorize that papers and abstracts are part of the GPT model training sets and this is reflected in the inconsistent LLM output.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eComparison between human definition vs GPT models\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModel\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMetric\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAverage %\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eMinimum %\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eMaximum %\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eCodes overlap\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e50.94\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e20.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e88.89\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGPT 4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eLogic overlap\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e90.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e50.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e100.00\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eStrings overlap\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e48.59\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e100.00\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eCodes overlap\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e27.51\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e10.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e85.20\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGPT 3.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eLogic overlap\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e70.20\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e90.00\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eStrings overlap\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e41.28\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e75.12\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eA noteworthy observation is that the codes generated by GPT-4 exhibit a marginally higher reliability compared to the textual strings or concept names. The codes denominator for the codes overlap metric is the number of codes from GPT4. Furthermore, despite the overall fewer codes generated by GPT-4, a closer examination suggests these codes may possess a higher positive predictive value (PPV) for accurately identifying the intended phenotypes. This finding suggests that while the volume of generated codes is limited, their specificity and relevance to the phenotypes are notably high, indicating that the model might be averaging out from source and could be surfacing the most popular ones. Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e, presents the GPT hallucinations with codes. In this work we compared the performance of GPT-3.5 and GPT-4 models in generating phenotype codes, using Biomedical Content Explorer[\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e] linked with PubDictionaries, ICD10 and ICD 10 CM dictionaries, with this comparison we show the biggest weakness of these LLM model as it is highly inaccurate and full of hallucinations. We discovered that hallucinations were notably present in both models, with GPT-3.5 showing a higher tendency towards these inaccuracies compared to GPT-4. These observations emphasize the imperative for cautious use and meticulous verification of data produced by LLMs, especially for phenotypes less documented in scientific literature. The pattern observed suggests a direct relationship between the scarcity of literature on specific phenotypes and the models' propensity to generate non-existent codes, pointing to a crucial area for enhancement in the training methodologies of these models.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab4\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 4\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eComparisons of GPT hallucinations when producing codes\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModel\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAverage %\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eMinimum %\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eMaximum %\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGPT 3.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e38\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e83\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGPT 4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e32\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e69\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eAdditionally, we performed a detailed evaluation of the capabilities of GPT-3.5 and GPT-4 models in accurately extracting phenotype definitions, a crucial step toward their integration into medical informatics. This entails an extensive series of evaluations comparing these Large Language Models (LLMs) against human-generated definitions to assess various aspects: the accuracy of code mapping, the consistency of code names, the logical structuring of definitions, and the degree of overlap in the codes identified. In our experiment, we compared human-generated definitions of phenotypes against those produced by GPT-4, focusing on the process of code mapping to align disparate coding systems into a unified framework. Table\u0026nbsp;\u003cspan refid=\"Tab5\" class=\"InternalRef\"\u003e5\u003c/span\u003e presents the results of our evaluations\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab5\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 5\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eCode mapping evaluation of GPT models, comparing codes as extracted vs mapped by the LLM\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModel\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMetric\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAverage %\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eMinimum %\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eMaximum %\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGPT 4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eExtracted codes overlap\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e50.94\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e20.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e89.00\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMapped codes overlap\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e72.89\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e28.98\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e97.00\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGPT 3.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eExtracted codes overlap\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e27.51\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e10.00\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e85.20\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMapped codes overlap\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e58.15\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e19.87\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e62.20\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eOur exploration into the utilization of GPT models for medical coding reveals several noteworthy challenges. This underscores the importance of carefully crafted prompts to ensure reliable and consistent results. The second challenge is the non-deterministic nature of generative LLMs. Identical prompts applied to different diseases generate stylistically different outputs, and depending on the prompting strategy, these lead to completely hallucinated responses. We included the screenshots of the GPT inaccuracies and hallucinations in the Supplementary Material section. Our findings show some promising results for certain phenotypes but not all. One of the bigger dangers here is the generation of hallucinations when asking for specific coding systems to standardize the definitions. While this could be easily overcome by using knowledge graphs [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e], some codes are completely fabricated and will never map to anything, as recently shown by Soroush et al. [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e].\u003c/p\u003e\n\u003ch3\u003eFuture Work\u003c/h3\u003e\n\u003cp\u003eThe next phase of this work will involve actually using the extracted phenotype definitions by the GPT models and comparing the patient cohorts they select. We will use tools available within the OHDSI community, such as Cohort Diagnostics [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e] and PheValuator [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e] to observe if these definitions are even close to human generated definitions. Our hypothesis here is that since the LLM identifies the most commonly used codes, and logic, for these definitions, it should do a decent job in identifying the core of the phenotype cohort, without including additional edge cases, thus not leaving too many patients out. Success will be measured by the similarity between these cohorts if they closely align, it signifies a significant achievement. Conversely, substantial differences would indicate a need for further refinement of the models, highlighting the iterative nature of improving LLM applications in healthcare. This rigorous evaluation process is essential for advancing our understanding and application of AI in enhancing medical research and patient care.\u003c/p\u003e"},{"header":"Conclusions","content":"\u003cp\u003eOur exploration into utilizing LLMs for automating phenotype definition extraction presents a promising avenue for enhancing the scalability and efficiency of phenotyping in digital healthcare. While our results underscore the potential of LLMs, particularly GPT-3.5 and GPT-4, in generating medically relevant codes, they also highlight the challenges of consistency in textual output and the occurrence of inaccuracies. The critical insight from our study is the importance of developing robust evaluation and validation frameworks to ensure the reliability of LLM outputs. The findings indicate that despite the hallucinations and inconsistencies, GPT models hold potential value as an initial step or augmentation tool in the phenotyping process which could significantly streamline and enhance electronic phenotyping methodologies.\u003c/p\u003e"},{"header":"Abbreviations","content":"\u003cp\u003eElectronic Health Records (EHRs)\u003c/p\u003e\n\u003cp\u003eNatural Language Processing (NLP)\u003c/p\u003e\n\u003cp\u003eLarge Language Models (LLMs)\u003c/p\u003e\n\u003cp\u003ePhenotype KnowledgeBase (PheKB)\u003c/p\u003e\n\u003cp\u003eObservational Health Data Sciences and Informatics (OHDSI)\u003c/p\u003e\n\u003cp\u003eHealth Data Research UK (HDRUK)\u003c/p\u003e\n\u003cp\u003eObservational Medical Outcomes Partnership (OMOP)\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eCommon Data Model (CDM)\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eAvailability of data and materials\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eData and code is provided at the following repository: https://github.com/jmbanda/blah8\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting interests\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNo competing or financial interests\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eTravel support for BLAH8 was provided by the National Bioscience Database Center (NBDC) of Japan Science and Technology Agency (JST) and Research Organization of Information and Systems (ROIS).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthors\u0026apos; contributions\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eRT and JMB wrote the main manuscript text, prepared figures and tables, designed and executed the experiments. All authors reviewed the manuscript.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAcknowledgements\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis work has been completed at BLAH 8\u0026nbsp;\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eBanda JM, Seneviratne M, Hernandez-Boussard T, Shah NH (2018) Advances in Electronic Phenotyping: From Rule-Based Definitions to Machine Learning Models. Annu Rev Biomed Data Sci 1:53\u0026ndash;68\u003c/li\u003e\n\u003cli\u003eSmoller JW (2018) The use of electronic health records for psychiatric phenotyping and genomics. Am J Med Genet B Neuropsychiatr Genet 177:601\u0026ndash;612\u003c/li\u003e\n\u003cli\u003eNadkarni GN, Gottesman O, Linneman JG, et al (2014) Development and validation of an electronic phenotyping algorithm for chronic kidney disease. AMIA Annu Symp Proc 2014:907\u0026ndash;916\u003c/li\u003e\n\u003cli\u003eWeng C, Shah NH, Hripcsak G (2020) Deep phenotyping: Embracing complexity and temporality-Towards scalability, portability, and interoperability. J Biomed Inform 105:103433\u003c/li\u003e\n\u003cli\u003eHuckvale K, Venkatesh S, Christensen H (2019) Toward clinical digital phenotyping: a timely opportunity to consider purpose, quality, and safety. NPJ Digit Med 2:88\u003c/li\u003e\n\u003cli\u003eRasmussen LV, Brandt PS, Jiang G, et al (2019) Considerations for Improving the Portability of Electronic Health Record-Based Phenotype Algorithms. AMIA Annu Symp Proc 2019:755\u0026ndash;764\u003c/li\u003e\n\u003cli\u003eAgarwal V, Podchiyska T, Banda JM, Goel V, Leung TI, Minty EP, Sweeney TE, Gyang E, Shah NH (2016) Learning statistical models of phenotypes using noisy labeled training data. J Am Med Inform Assoc 23:1166\u0026ndash;1173\u003c/li\u003e\n\u003cli\u003eYang Z, Dehmer M, Yli-Harja O, Emmert-Streib F (2020) Combining deep learning with token selection for patient phenotyping from electronic health records. Sci Rep 10:1432\u003c/li\u003e\n\u003cli\u003eBeaulieu-Jones BK, Greene CS, Pooled Resource Open-Access ALS Clinical Trials Consortium (2016) Semi-supervised learning of the electronic health record for phenotype stratification. J Biomed Inform 64:168\u0026ndash;178\u003c/li\u003e\n\u003cli\u003eLuo L, Yan S, Lai P-T, Veltri D, Oler A, Xirasagar S, Ghosh R, Similuk M, Robinson PN, Lu Z (2021) PhenoTagger: a hybrid method for phenotype concept recognition using human phenotype ontology. Bioinformatics 37:1884\u0026ndash;1890\u003c/li\u003e\n\u003cli\u003eBrown T, Mann B, Ryder N, et al (2020) Language models are few-shot learners. Adv Neural Inf Process Syst 33:1877\u0026ndash;1901\u003c/li\u003e\n\u003cli\u003eTekumalla R, Banda JM (2023) Leveraging Large Language Models and Weak Supervision for Social Media Data Annotation: An Evaluation Using COVID-19 Self-reported Vaccination Tweets. In: HCI International 2023 \u0026ndash; Late Breaking Papers. Springer Nature Switzerland, pp 356\u0026ndash;366\u003c/li\u003e\n\u003cli\u003eYang J, Liu C, Deng W, Wu D, Weng C, Zhou Y, Wang K (2024) Enhancing phenotype recognition in clinical notes using large language models: PhenoBCBERT and PhenoGPT. Patterns (N Y) 5:100887\u003c/li\u003e\n\u003cli\u003eKirby JC, Speltz P, Rasmussen LV, et al (2016) PheKB: a catalog and workflow for creating electronic phenotype algorithms for transportability. J Am Med Inform Assoc 23:1046\u0026ndash;1052\u003c/li\u003e\n\u003cli\u003eBanda JM, Halpern Y, Sontag D, Shah NH (2017) Electronic phenotyping with APHRODITE and the Observational Health Sciences and Informatics (OHDSI) data network. AMIA Jt Summits Transl Sci Proc 2017:48\u0026ndash;57\u003c/li\u003e\n\u003cli\u003eHRDUK Phenotype Library. https://phenotypes.healthdatagateway.org/. Accessed 22 Mar 2024\u003c/li\u003e\n\u003cli\u003eBrandt PS, Kho A, Luo Y, et al (2023) Characterizing variability of electronic health record-driven phenotype definitions. J Am Med Inform Assoc 30:427\u0026ndash;437\u003c/li\u003e\n\u003cli\u003eKim J (2023) Biomedical Content Explorer. https://chat.openai.com/g/g-wdWOSr2gs-biomedical-content-explorer.\u003c/li\u003e\n\u003cli\u003eCallahan TJ, Stefanski AL, Wyrwa JM, et al (2023) Ontologizing health systems data at scale: making translational discovery a reality. NPJ Digit Med 6:89\u003c/li\u003e\n\u003cli\u003eSoroush Ali, Glicksberg Benjamin S., Zimlichman Eyal, Barash Yiftach, Freeman Robert, Charney Alexander W., Nadkarni Girish N, Klang Eyal (2024) Large Language Models Are Poor Medical Coders \u0026mdash; Benchmarking of Medical Code Querying. NEJM AI 1:AIdbp2300040\u003c/li\u003e\n\u003cli\u003eGilbert J, Rao G, Schuemie M, Ryan P, Weaver J (2023) CohortDiagnostics: Diagnostics for OHDSI Cohorts.\u003c/li\u003e\n\u003cli\u003eSwerdel JN, Hripcsak G, Ryan PB (2019) PheValuator: Development and evaluation of a phenotype algorithm evaluator. J Biomed Inform 97:103258\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"genomics-and-informatics","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"Learn more about [Genomics \u0026 Informatics](https://genomicsinform.biomedcentral.com/)","snPcode":"44342","submissionUrl":"https://submission.springernature.com/new-submission/44342/3","title":"Genomics \u0026 Informatics","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"BMC/SO AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"ChatGPT, Electronic Phenotyping, Large Language Models (LLMs), Evaluation","lastPublishedDoi":"10.21203/rs.3.rs-4798033/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4798033/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eElectronic phenotyping involves a detailed analysis of both structured and unstructured data, employing rule-based methods, machine learning, natural language processing, and hybrid approaches. Currently, the development of accurate phenotype definitions demands extensive literature reviews and clinical experts, rendering the process time-consuming and inherently unscalable. Large Language Models offer a promising avenue for automating phenotype definition extraction but come with significant drawbacks, including reliability issues, the tendency to generate non-factual data ('hallucinations'), misleading results, and potential harm. To address these challenges, our study embarked on two key objectives: (1) defining a standard evaluation set to ensure Large Language Models outputs are both useful and reliable, and (2) evaluating various prompting approaches to extract phenotype definitions from Large Language Models, assessing them with our established evaluation task. Our findings reveal promising results that still require human evaluation and validation for this task. However, enhanced phenotype extraction is possible, reducing the amount of time spent in literature review and evaluation.\u003c/p\u003e","manuscriptTitle":"Towards automated phenotype definition extraction using large language models","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-08-21 06:18:35","doi":"10.21203/rs.3.rs-4798033/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2024-07-26T14:44:12+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-07-26T13:13:30+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"61008265401100471491264741981145706806","date":"2024-07-16T23:43:42+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-06-29T03:45:03+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"321986167981245904396368077040203839739","date":"2024-06-16T15:01:15+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-06-15T13:14:58+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-06-03T06:43:07+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-06-03T00:38:53+00:00","index":"","fulltext":""},{"type":"submitted","content":"Genomics \u0026 Informatics","date":"2024-05-31T19:33:37+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"genomics-and-informatics","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"Learn more about [Genomics \u0026 Informatics](https://genomicsinform.biomedcentral.com/)","snPcode":"44342","submissionUrl":"https://submission.springernature.com/new-submission/44342/3","title":"Genomics \u0026 Informatics","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"BMC/SO AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"83a4776e-263a-42e7-beae-05f53e91e11b","owner":[],"postedDate":"August 21st, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2024-09-29T23:08:13+00:00","versionOfRecord":[],"versionCreatedAt":"2024-08-21 06:18:35","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4798033","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4798033","identity":"rs-4798033","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}