mCodeGPT: Bridging the Gap between Unstructured Medical Text and Structured Cancer Ontologies

mCodeGPT: Bridging the Gap between Unstructured Medical Text and Structured Cancer Ontologies | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article mCodeGPT: Bridging the Gap between Unstructured Medical Text and Structured Cancer Ontologies Kai Zhang, Tongtong Huang, Bradley A Malin, Travis Osterman, Qi Long, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-3940535/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract The rapidly growing collection of clinical free text data about cancer provides an unprecedented opportunity to harness rich, real-world insights for advancing cancer research and treatment. At the same time, there is a formidable challenge in efficiently extracting, standardizing, and structuring this vast and unstructured information to derive meaningful conclusions. In this paper, we address the problem of information extraction to standardize and structure such free text using large language models (LLMs) following the minimal Common Oncology Data Elements (mCODE™) structure. To this end, we implement mCodeGPT, an approach that ingests the cancer ontology knowledgebase and hierarchically generates prompts to extract relevant information from clinical notes. The generated output is subsequently organized and displayed in a tabular and structured format for improved analysis and interpretation. Our approach is annotation-free and eliminates the need for model training. mCodeGPT achieved over 95% accuracy for named entity recognition on mCODE concepts, Biochemical Research Methods Figures Figure 1 Figure 2 Figure 3 1 Introduction Cancer research has evolved dramatically along with the emergence of modern technologies [ 1 ]. The process of diagnosing, formulating treatment plans, conducting medical assessments, and creating genetic profiles generates a vast and diverse volume of data. The minimal Common Data Elements (short as mCODE™) initiative represents a concerted effort to collate a fundamental set of structured data elements that will serve as a foundational dataset for oncology research and treatment [ 2 ]. The mCODE data standard has the potential to significantly augment the volume of high-quality, shareable data for various cancer types, thereby potentially improving the quality of patient care [ 3 ]. However, in EHRs, unstructured data accounts for nearly 80% of the entire data [ 4 ]. This data creates an opportunity to extract deep insights from real-world clinical encounters that could significantly advance the field. However, there are significant challenges in the standardization and structuring of this voluminous and disorganized data to enable the derivation of meaningful and actionable conclusions. These challenges are further exacerbated by a lack of interoperability [ 5 ]. The core of the complexity in standardizing unstructured data is attributed to the large heterogeneity in the information, much of which is in an unstructured data format [ 6 ]. Converting unstructured data into a structured format is a nontrivial challenge when dealing with specialized domains like cancer, where the medical terminology and oncology concepts are highly domain-specific [ 7 ]. Nevertheless, there is a growing recognition that addressing the intricacies of data standardization in specialized domains, such as cancer, necessitates a closer examination of information extraction methods. Data standardization methods for clinical notes largely focus on the concept of information extraction and the natural language processing (NLP) techniques that support it [ 8 ]. Traditional NLP methodologies typically depend on manually curated rules and human-annotated datasets for their training procedures. This results in named entity recognition (NER) or information extraction systems being less generalizable, limiting their portability [ 9 ]. Specifically, these systems frequently demonstrate limited adaptability, as evidenced by the prevalent challenge of out-of-distribution prediction, where performance significantly deteriorates when models encounter data that differ from the training set. This lack of generalizability primarily stems from an over-reliance on the specific characteristics of the training data and labels. Recent studies in information extraction from unstructured clinical notes have ventured away from manual rule creation and human-annotated datasets to leverage the capabilities of large language models (LLMs). For example, OntoGPT uses LLMs to query and extract ontology information from free text [ 10 ]. In doing so, OntoGPT designs prompts for each named entity in the ontology and queries LLM to extract the information in a semi-structured format. OntoGPT was developed to support general ontology query-answer tasks and was not designed to handle large-scale ontologies such as mCODE. Therefore, the prompt generation method in the OntoGPT pipeline easily exceeds the token size limitation of LLMs, which is typically four thousand for modern LLMs. Moreover, OntoGPT requires representing an ontology in YAML language and requires a large amount of human labor to represent an ontology to YAML template. To address these challenges, we have advanced the OntoGPT framework by developing an extended pipeline, mCodeGPT. We leverage the fact that an ontology typically embeds an intricate hierarchical structure with various relations (such as a parent-child relation) between the named entities inside the ontology. This type of structure is naturally captured in a graph structure. The manual preparation of the YAML ontology template file can be replaced by an automated graph traversal algorithm, which generates a prompt by traversing the graphical model. 2 Methods We introduce a novel approach to NER with respect to ontologies. This approach performs NER in a zero-shot manner, which is distinct from prior approaches that require extensive training [ 11 , 12 ]. 2.1 Graphical Model An ontology formalizes a shared understanding of concepts, or named entities, and their relations - typically in a hierarchical structure from general to specific. To represent named entities and their relations within the ontology efficiently, we leverage a graph structure, not only due to its efficiency for data representation but also its potential to automate the prompts generation by leveraging graph traversal algorithms. We represent the ontology using a Directed Acyclic Graph (DAG) \(G(V,E)\) , where \(V\) represents the set of named entities and \(E\) represents the set of edges between the named entities. We introduce five attributes for each named entity \({v}_{i}(\in V)\) : Attribute( \({v}_{i}\) ) ={Label( \({v}_{i}\) ), Description( \({v}_{i}\) ), IsLeaf( \({v}_{i}\) ), Level( \({v}_{i}\) ), Present \(({v}_{i}\) )} (1) where Label( \({v}_{i}\) ) is the name of the entity, Description( \({v}_{i}\) ) is a string that represents the prompt message to be used for extracting the named entity \({v}_{i}\) from the input text, and IsLeaf( \({v}_{i}\) ) is a Boolean value that represents if the node is a leaf node (i.e., a node devoid of siblings in the graph). Level \(\left({v}_{i}\right)\) is an integer that represents the level of the graph model (tree) the node \({v}_{i}\) is from and Present \(\left({v}_{i}\right)\) is a Boolean value that represents if the information associated with \({v}_{i}\) is present in the input text. Each directed edge \({e}_{ij}\in E\) from the node \({v}_{i}\) to the node \({v}_{j}\) represents the relation between the two nodes. In the mCODE context, all relations are essentially “has” (or parent-child) relations where a named entity \({v}_{i}\) has a named entity \({v}_{j}\) as an attribute. 2.2 Algorithms We introduce three distinct algorithms that demonstrate varied strategies for capitalizing on LLMs in conjunction with ontology. Our comparative analysis seeks to highlight their strengths and limitations, thereby providing insight into the optimal conditions for enhancing LLM performance with ontology integration. An illustrative example for each of the three algorithms is provided in Appendix A. Root-to-Leaf Streamliner (RLS) Although the leaf node information is what is needed by LLM, the named entities on the path provide background information for the leaf node. The RLS strategically harmonizes the information from the root node to each leaf node. This results in a compact, yet informative, prompt that encapsulates the entire lineage of the ontology tree. For each leaf node \({v}_{i}\) where IsLeaf \(\left({v}_{i}\right)\) is true, RLS finds the path from the root node to \({v}_{i}\) , $$root\to {v}_{a}\to {v}_{b}\to \dots \to {v}_{i}$$ 2 and generates a prompt for \({v}_{i}\) . Given that \({v}_{i}\) alone might contain limited information, we incorporate comprehensive background information by including the descriptions of all its parent nodes. For example, consider the following path: $$cancer patient \left(root\right)\to tumor \left({v}_{a}\right)\to tumor size \left({v}_{b}\right)\to$$ $$measurement method \left({v}_{i}\right)$$ 3 Instead of Description \(\left({v}_{i}\right)\) being “What is the tumor measurement method?” , RLS will create Description \(\left({v}_{i}\right)\) as “ If the patient has a tumor, what is the measurement method to measure the tumor size? ”. The detailed prompt format can be found in Appendix C. Breadth-First Ontology Pruner (BFOP) In clinical data extraction, it is crucial to recognize that an individual clinical note may not encapsulate all entities present in a comprehensive ontology. Utilizing an extensive prompt for information retrieval can be counterproductive, as it may induce “hallucination”, a scenario where models erroneously produce or identify non-existent entities within the data. These hallucinations often result from models being overly influenced by a detailed ontology, leading to false identifications that could undermine the accuracy of clinical decision-making. Consequently, careful pruning of ontological entities is imperative to ensure precision in the extraction process. BFOP is engineered to pinpoint a relevant subgraph tailored to the designated target text. This approach is inspired by the Chain-of-Thought prompting methodology [ 13 ], which utilizes a parsing strategy akin to the breadth-first search algorithm, employing a hierarchical level-wise analysis. Essentially, BFOP is a layer-wise prompt generation method. It refines the next-level named entity prompts based on whether the information of the named entities from the current level is present. Initially, entities on the first level (the root entity) of the ontology are parsed and transformed into prompts suitable for querying ( Appendix D: Algorithm BFOP lines 1–7). Subsequently, these prompts are fed into the LLM to extract pertinent information from patient notes ( Appendix D: Algorithm BFOP line 8). A salient feature of our approach is its dynamic pruning mechanism. If a particular named entity is absent in the patient's record, then it is natural to infer that its children will also be devoid of relevant data, which eliminates the need to parse them ( Appendix D: Algorithm BFOP lines 9–12). Therefore, in the next level, \(L+1\) , based on the previous level ( \(L\) ) result, some with Present \(({v}_{i}\) ) = False can be skipped. Two-Phase Ontology Parser (2POP) The main idea of the 2POP methodology is that queries about the mere existence of a named entity are simpler to resolve than those seeking details about the entity, particularly in the context of intricate ontologies such as the mCODE. 2POP aims to isolate a pertinent subgraph from the comprehensive ontology that correlates with the target text. Unlike the incremental, hierarchical extraction process characteristic of BFOP, 2POP adopts a comprehensive approach, extracting the entire pertinent structure in one step. The 2POP algorithm employs a two-pronged strategy: First, it queries all leaf nodes, classifying the outcomes based on binary responses. Then, the second phase utilizes only the entities confirmed by a “Yes” response to create a prompt. This prompt is then utilized to prompt an LLM to extract information from unstructured text, thereby enabling the retrieval of relevant data ( Appendix D: Algorithm 2POP). The procedure for NER using the mCodeGPT pipeline is shown in Fig. 1 . 3 Experiments and Results 3.1 Experiment setting We evaluate the efficacy of the proposed LLM-based named entity extraction and the effectiveness of each prompt-generating method. The experiments were conducted using high-quality synthetic oncology notes, produced by the advanced generative model GPT3.5-turbo using OpenAI API (Chat Completions API). The model was configured with a temperature setting of 0.8 and a maximum token limit of 4,096 to create a dataset of simulated clinical notes. These notes encompassed a diverse range of cancer types, with the subject of each note being randomly selected from a predetermined list of cancer categories. Table 1 specifies the Cancer types in this experiment. A total of 400 cancer reports (clinical notes) were generated, each focusing on the characteristics and details specific to the cancer type it represents. For each note, a random cancer type was chosen from Table 1 to generate biologically meaningful descriptions. Table 1 Types of Cancers Category Types of Cancer Skin Cancers Basal Cell Carcinoma, Squamous Cell Carcinoma, Melanoma Kidney Cancers Renal Cell Carcinoma Breast Cancers Ductal Carcinoma In Situ, Invasive Ductal Carcinoma, Adenocarcinoma Soft Tissue Sarcomas Osteosarcoma, Liposarcoma, Rhabdomyosarcoma Blood Cancers Leukemia, Acute Lymphoblastic Leukemia, Acute Myeloid Leukemia, Chronic Lymphocytic Leukemia, Chronic Myeloid Leukemia Lymphomas Non-Hodgkin Lymphoma, Hodgkin Lymphoma Brain and Central Nervous System Glioblastoma, Astrocytoma, Meningioma, Medulloblastoma Gynecologic Cancers Cervical Cancer, Ovarian Cancer, Uterine Cancer, Vaginal Cancer, Vulvar Cancer Respiratory Cancers Lung Cancer, Small Cell Lung Cancer, Non-Small Cell Lung Cancer Genitourinary Cancers Prostate Cancer, Bladder Cancer Gastrointestinal Cancers Colon and Rectal Cancer, Pancreatic Cancer, Stomach (Gastric) Cancer, Esophageal Cancer Endocrine Cancers Thyroid Cancer Other Organ-Specific Cancers Liver Cancer, Testicular Cancer, Nasopharyngeal Cancer, Laryngeal and Hypopharyngeal Cancer, Penile Cancer, Oral and Oropharyngeal Cancer Pediatric Cancers Neuroblastoma, Wilms Tumor, Retinoblastoma We examined the effectiveness of various models, including open-source models and OpenAI models, along with the proposed methods, RLS, BFOP, and 2POP. The open-source models include four from GPT4ALL [ 14 ], namely Orca-mini-13b, Nous-hermes-13b, Llama-2-7b-chat, WizardLM-13b-uncensored, and one from FastChat [ 15 ], the Fastchat-vicuna-13b-1.5. The experiments were performed on a local server with an AMD EPYC 7742 64-core Processor. Optionally, GPU resources can be used if available, which can reduce the running time for queries. 3.2 Results 3.2.1 Performance This section evaluates the performance of mCodeGPT on named entity extraction tasks from the mCODE ontology using different LLM as backbones combined with different prompt generation methods. We apply the following rules when counting correct, error, and hallucinated named entities. Correct : The information extraction result (IE) is the same as the ground truth (GT), or semantically equivalent; for example, GT is “Female” but IE is “female”; GT is “Normal” but IE is “within the Norm range”; GT is “15-Jan-75” but IE is “Jan 15, 1975” Error : The information extraction result is inconsistent with ground truth. This includes two cases: Misidentification : The named entity has information in the notes (GT) but the information extraction result is none (“N/A”, “non”, “Not mentioned”, “N/A (not mentioned in the text)”, etc. Misplacement : The named entity has information in the notes, and the information extraction result is not correct, but from the notes itself (not hallucinated). Hallucination : Information extraction results in hallucinated information (not from the original notes), i.e., information that is fabricated or generated and is not present in the original text. Table 2 represents the accuracy of each model on the 400 clinical notes. With respect to the incorrect results, we also identified the incorrect incidence rate and hallucination rate. The table shows our pipeline generally performed better with the 2POP method, consistently having the highest accuracy. This aligns with the initial design intent of the method, to perform a first-round screening (ask presence) before prompting more difficult questions (ask details). Among the open-source models, Orca-mini-13b achieved the highest accuracy with 2POP at 0.937, closely followed by Llama-2-7b-chat at 0.925. Among all the models, the OpenAI model GPT3.5-turbo-16k outperformed its counterparts, achieving an accuracy of 0.956 using 2POP. By contrast, RLS tended to achieve the lowest accuracy across most models. This finding aligns well with the nature of the RLS method, which issues queries for all named entities straightforwardly, without leveraging hierarchical structuring or an easy-to-difficult design approach. Hallucinations happened quite rarely (< 2.0%) for both open-source models and Open-AI API-based models. In general, BFOP and 2POP slightly reduced hallucination compared with RLS, which can be explained by the designed chain-of-thought queries. Overall, the OpenAI models hallucinated less than the open-source models. Table 2 Evaluation of Information Extraction Efficacy. This table presents the assessment of information extraction performance using different model-method pairings, quantified through accuracy, error rate, and hallucination rate, which collectively sum to unity. The analysis was performed on a corpus of 400 notes. The bold fonts represent the best performance entry in each column. Accuracy Error Rate Hallucination Rate RLS BFOP 2POP RLS BFOP 2POP RLS BFOP 2POP Open-Source Models Orca-mini-13b 0.792 0.887 0.937 0.194 0.100 0.049 0.014 0.013 0.014 Nous-hermes-13b 0.843 0.836 0.920 0.142 0.154 0.067 0.015 0.010 0.013 Llama-2-7b-chat 0.840 0.900 0.925 0.146 0.087 0.063 0.014 0.013 0.012 Fastchat-vicuna-13b-1.5 0.787 0.867 0.902 0.194 0.116 0.088 0.019 0.017 0.010 WizardLM-13b-uncensored 0.831 0.865 0.940 0.153 0.120 0.048 0.016 0.015 0.012 OpenAI Models GPT3.5-turbo-16k 0.865 0.952 0.956 0.120 0.038 0.034 0.015 0.010 0.010 GPT3.5-turbo 0.868 0.926 0.934 0.118 0.065 0.057 0.014 0.009 0.009 GPT4 0.879 0.932 0.946 0.108 0.056 0.045 0.013 0.012 0.009 GPT4-32k 0.870 0.930 0.950 0.118 0.059 0.042 0.012 0.011 0.008 3.2.2 Running Time The running times varied across models depending on the method employed, which provides insights into the efficiency and directly impacts the practical utility and feasibility of each model-method combination. Figure 2 shows the running time for each model concerning each prompting method. In general, open-source models took longer to process depending on the local computing resources. WizardLM-13b-uncensored, exhibited considerably longer running times. This model required 80 hours with the RLS method to run on 400 notes. By contrast, the OpenAI models demonstrated considerably faster running times. The GPT3.5-turbo model emerged as one of the most efficient, with running times under 1.5 hours across all methods. It is worth noting that the maximum token per minute (TPM) of Azure OpenAI models differs by what pricing tier is being chosen, which can affect the frequency of calling OpenAI APIs and, therefore affect the OpenAI models' performance using our pipeline. For example, under the pricing tier s0, our Azure OpenAI limits up to 240 TPM (GPT3.5-turbo, GPT3.5-turbo-16k), 20 TPM (GPT4), and 60 TPM (GPT4-32k), respectively. It should be noted that the extended processing time does not necessarily imply ineffectiveness. All models exhibited only around a 5% accuracy drop compared to OpenAI's advanced models (GPT4, GPT4-32k) – and, more importantly, they do not require any overhead for token consumption. 3.2.3 Performance on Medical Abbreviations In practice, physicians often use abbreviations in their notes. To assess our model’s performance in more realistic situations, we modified the clinical notes generation process and let the clinical notes embed as many abbreviations, drawn from various medical terminologies, as possible. The abbreviations are provided in Appendix B. Table 3 Comparative Performance Analysis of Notes with High Utilization of Medical Terminology Abbreviations Versus Notes Without Abbreviations. Accuracy RLS BFOP 2POP Open-Source Models Orca-mini-13b 0.6589 ↓0.1334 0.6807 ↓0.2064 0.7051 ↓0.2318 Nous-hermes-13b 0.7254 ↓0.1178 0.7262 ↓0.1102 0.7601 ↓0.1602 Llama-2-7b-chat 0.7191 ↓0.1208 0.7441 ↓0.1562 0.7600 ↓0.1651 Fastchat-vicuna-13b-1.5 0.6508 ↓0.1366 0.6984 ↓0.1726 0.7290 ↓0.1731 WizardLM-13b-uncensored 0.6934 ↓0.1378 0.7312 ↓0.1337 0.7333 ↓0.2070 OpenAI Models GPT3.5-turbo-16k 0.7314 ↓0.1331 0.7451 ↓0.2070 0.7501 ↓0.2061 GPT3.5-turbo 0.7266 ↓0.1411 0.7199 ↓0.2065 0.7334 ↓0.2010 GPT4 0.7452 ↓0.1340 0.7564 ↓0.1757 0.7795 ↓0.1660 GPT4-32k 0.7460 ↓0.1243 0.7614 ↓0.1686 0.7571 ↓0.1923 Table 3 summarizes the performance of the different models and prompting methods in terms of accuracy. GPT-4 demonstrated superior proficiency in interpreting medical terminologies and abbreviations. For all models and methods, the performance dropped by 11–20% when compared to results based on notes without abbreviations. The decline in performance does not diminish the utility of the proposed tool, since medical abbreviations are usually limited and can easily be summarized. It is also worth noting that shows the worst-case performance because all medical terms were replaced by their abbreviations in the 400 notes, which is unlikely to happen in reality. A full list of common abbreviations of medical concepts can be found in Appendix B. 3.2.4 Heterogenous Difficulty of Named Entities The performance varied across various named entities within specific information extraction tasks. Figure 3 illustrates the evaluation of named entity recognition performance within the dataset comprising 400 documents. Each bar in the graph represents the accuracy with which the system identifies each named entity, with a filled bar denoting flawless recognition across the entire corpus. The data indicates a perfect extraction for 65% of the named entities, while the accuracy for the remaining 35% varied between 25% and 99%. The named entities that achieved perfect performance primarily consisted of terms related to medical terms, genomic variants, and cancer-related information. These terms are technical and likely contain domain-specific abbreviations. By contrast, the imperfect (< 100% accurate on all 400 notes) named entities comprise terms pertinent to patient details, such as demographics, procedures, and medication administration. These terms are more focused on patient-related data and administrative dimensions. 4 Discussion Our findings offer valuable insight into the possibilities and power of LLMs, especially when used with clinical ontologies. Beyond its direct application to cancer ontologies, our methodologies can be extended and applied to various other medical specialties and healthcare domains. As with any study, it is essential to consider the context and limitations of our findings. Despite achieving high accuracy, it is essential to have human oversight to ensure the fidelity of the results. There were instances where not every named entity was accurately identified. Thus, a review of the outcomes is necessary to prevent any oversight or misidentification. To promote in-context learning and increase mCodeGPT’s performance when handling abbreviations, a list of abbreviations can be injected into the prompt as background knowledge -- used as a one-shot exemplar. Earlier studies have shown that one-shot in-context learning [ 16 ], by presenting in the prompt an example of the task it is asked to do, can significantly boost the performance of LLMs. There are several potential areas for future research for further refinement and adaptation of mCodeGPT. First, multimodal integration – with the evolving capability of OpenAI GPT models to incorporate multimodality, mCodeGPT can be enhanced to leverage both textual and image-based information for NER tasks, especially in the fields of radiology and pathology. Second, the automated generation of ontologies becomes crucial for expanding into domains where ontological frameworks are not well-established. Parsing clinical notes and autonomously creating ontologies for NER is of paramount importance in such scenarios. 5 Conclusions Our mCodeGPT architecture represents a significant advancement in cancer research and clinical informatics, introducing innovative methods for zero-shot extraction of structured data from complex clinical narratives. This transformation of unstructured information into actionable insights has the potential to fundamentally alter our comprehension and approach to cancer treatment and research. By streamlining the process of NER, mCodeGPT can substantially reduce the time and labor traditionally required for data extraction and standardization. This increased efficiency allows healthcare professionals to allocate more time to direct patient care and clinical assessments. Consequently, the reduced administrative burden may accelerate diagnostic procedures and the initiation of treatment, which could lead to enhanced patient outcomes. References Ya RA, Depinho M, Ernst K. Cancer Research: Past, Present and Future. Nature Reviews Cancer 2011; 11 :749–54. Osterman TJ, Terry M, Miller RS. Improving cancer data interoperability: The promise of the Minimal Common Oncology Data Elements (mCODE) initiative. JCO Clin Cancer Inform 2020; 4 :993–1001. Osterman TJ, Yao JC, Krzyzanowska MK. Implementing innovation: Informatics-based technologies to improve care delivery and clinical research. Am Soc Clin Oncol Educ Book 2023; 43 :e389880. Li I, Pan J, Goldwasser J, et al. Neural Natural Language Processing for unstructured data in electronic health records: A review. Computer Science Review 2022; 46 :100511. Chin L, Andersen JN, Futreal PA. Cancer genomics: from discovery science to personalized medicine. Nat Med 2011; 17 :297–303. Kowalczyk S, Shankar K. Data sharing in the sciences. Annual review of information science and technology 2011; 45 :247–94. Warner JL, Maddux SE, Hughes KS, et al. Development, implementation, and initial evaluation of a foundational open interoperability standard for oncology treatment planning and summarization. J Am Med Inform Assoc 2015; 22 :577–86. Parsing C. Speech and language processing. Power Point Slides Published Online First: 2009.https://people.cs.pitt.edu/~litman/courses/cs2731_f19/lec/slp12_f19.pdf Sarawagi S. Information Extraction. Foundations and Trends® in Databases 2008; 1 :261–377. Caufield JH, Hegde H, Emonet V, et al. Structured prompt interrogation and recursive extraction of semantics (SPIRES): A method for populating knowledge bases using zero-shot learning. arXiv [cs.AI]. 2023.http://arxiv.org/abs/2304.02711 Devlin J, Chang M-W, Lee K, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv [cs.CL]. 2018.http://arxiv.org/abs/1810.04805 Liu Y, Ott M, Goyal N, et al. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv [cs.CL]. 2019.http://arxiv.org/abs/1907.11692 Wei J, Wang X, Schuurmans D, et al. Chain-of-thought prompting elicits reasoning in large language models. Adv Neural Inf Process Syst 2022; 35 :24824–37. Anand Y, Nussbaum Z, Duderstadt B, et al. GPT4All: Training an assistant-style chatbot with large scale data distillation from GPT-3.5-turbo. 2023.http://static.nomic.ai.s3.amazonaws.com/gpt4all/2023_GPT4All_Technical_Report.pdf (accessed 2 Jan 2024). Zheng L, Chiang W-L, Sheng Y, et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv [cs.CL]. 2023.http://arxiv.org/abs/2306.05685 Brown T, Mann B, Ryder N, et al. Language models are few-shot learners. Adv Neural Inf Process Syst 2020; 33 :1877–901. Additional Declarations The authors declare no competing interests. Supplementary Files Appendix.docx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-3940535","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":271817961,"identity":"b9963212-5aea-4026-923d-8c6905834b81","order_by":0,"name":"Kai Zhang","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA/ElEQVRIiWNgGAWjYBACCQbGBhACAsbGBwwMB6DibERpYW42YEg4wMBDWAvIfLAW9jYJorRIth9uYPy5wy5P3oGxrZr3xx0Ge7HDDxg+lB3GqUWaJ7GBQfJMcrHhAca22zwJzxh4pNMMGGecw61FjgGoxbCNOXFjA1jLYaCWHAZm3jY8WvgfNjAkttWDtRTDtfzFo0VaAmjLwbbDifMZGNuY4VoY8WiRnPGw4WBj2/HEDcyMzZJz0g7z8NxOMzjYcy4dpxaJ8+kPH/5sq06c397+8MMbm8Ny7LOTHz74UWaNUwsIHAARBlCX8MBFCAL5BqKUjYJRMApGwUgEAB44VVOnzamTAAAAAElFTkSuQmCC","orcid":"https://orcid.org/0000-0002-4519-609X","institution":"Department of Data Science and Artificial Intelligence, McWilliams School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, TX 77030","correspondingAuthor":true,"prefix":"","firstName":"Kai","middleName":"","lastName":"Zhang","suffix":""},{"id":271817962,"identity":"d83b1c01-8c70-4de8-afcd-0a45966ff161","order_by":1,"name":"Tongtong Huang","email":"","orcid":"","institution":"Department of Data Science and Artificial Intelligence, McWilliams School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, TX 77030","correspondingAuthor":false,"prefix":"","firstName":"Tongtong","middleName":"","lastName":"Huang","suffix":""},{"id":271817963,"identity":"525c6edc-1c92-4423-9f03-21647c4029ce","order_by":2,"name":"Bradley A Malin","email":"","orcid":"","institution":"Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37203","correspondingAuthor":false,"prefix":"","firstName":"Bradley","middleName":"A","lastName":"Malin","suffix":""},{"id":271817964,"identity":"c0534815-e2c6-465a-bd84-55a902906dcc","order_by":3,"name":"Travis Osterman","email":"","orcid":"","institution":"Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37203","correspondingAuthor":false,"prefix":"","firstName":"Travis","middleName":"","lastName":"Osterman","suffix":""},{"id":271817965,"identity":"22e0910f-7420-4df7-a175-3a753be1287a","order_by":4,"name":"Qi Long","email":"","orcid":"","institution":"Department of Biostatistics, Epidemiology and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA 19104","correspondingAuthor":false,"prefix":"","firstName":"Qi","middleName":"","lastName":"Long","suffix":""},{"id":271817966,"identity":"4f0e466b-a75c-499a-bfa5-eb102f410bd7","order_by":5,"name":"Xiaoqian Jiang","email":"","orcid":"","institution":"Department of Data Science and Artificial Intelligence, McWilliams School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, TX 77030","correspondingAuthor":false,"prefix":"","firstName":"Xiaoqian","middleName":"","lastName":"Jiang","suffix":""}],"badges":[],"createdAt":"2024-02-08 16:20:17","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-3940535/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-3940535/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":50911299,"identity":"d078eee0-c884-4e45-8678-5ea36dc87f14","added_by":"auto","created_at":"2024-02-09 12:27:17","extension":"jpeg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":244201,"visible":true,"origin":"","legend":"\u003cp\u003eThe pipeline of mCodeGPT and explanation of steps.\u003c/p\u003e","description":"","filename":"floatimage1.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-3940535/v1/53fe532da8683a717b7ddaec.jpeg"},{"id":50911017,"identity":"0191a58b-a29f-4bc0-bece-1e6638f5085a","added_by":"auto","created_at":"2024-02-09 12:19:17","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":51579,"visible":true,"origin":"","legend":"\u003cp\u003eThe running time for each model when using different algorithms.\u003c/p\u003e","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-3940535/v1/9cbf70b18d76bb4f55ddf470.png"},{"id":50911300,"identity":"4a4e96df-0366-44cf-aa32-b2ef4da80a12","added_by":"auto","created_at":"2024-02-09 12:27:17","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":477110,"visible":true,"origin":"","legend":"\u003cp\u003eAccuracy for each named entity on the 400 notes using model GPT3.5.\u003c/p\u003e","description":"","filename":"floatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-3940535/v1/c9a352c727f2daa405083865.png"},{"id":50911638,"identity":"2b0cdf94-3e5b-489e-84ce-f165a9b52803","added_by":"auto","created_at":"2024-02-09 12:35:18","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":962822,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3940535/v1/b28cb09b-1194-4baa-afe8-5b5f52860618.pdf"},{"id":50911015,"identity":"f133d601-69a1-4c20-ac0d-f6b0ba8b1427","added_by":"auto","created_at":"2024-02-09 12:19:17","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":1212357,"visible":true,"origin":"","legend":"","description":"","filename":"Appendix.docx","url":"https://assets-eu.researchsquare.com/files/rs-3940535/v1/bd0059435dd4f898f99ff927.docx"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003cp\u003emCodeGPT: Bridging the Gap between Unstructured Medical Text and Structured Cancer Ontologies\u003c/p\u003e","fulltext":[{"header":"1 Introduction","content":"\u003cp\u003eCancer research has evolved dramatically along with the emergence of modern technologies [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]. The process of diagnosing, formulating treatment plans, conducting medical assessments, and creating genetic profiles generates a vast and diverse volume of data. The minimal Common Data Elements (short as mCODE\u0026trade;) initiative represents a concerted effort to collate a fundamental set of structured data elements that will serve as a foundational dataset for oncology research and treatment [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]. The mCODE data standard has the potential to significantly augment the volume of high-quality, shareable data for various cancer types, thereby potentially improving the quality of patient care [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. However, in EHRs, unstructured data accounts for nearly 80% of the entire data [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. This data creates an opportunity to extract deep insights from real-world clinical encounters that could significantly advance the field. However, there are significant challenges in the standardization and structuring of this voluminous and disorganized data to enable the derivation of meaningful and actionable conclusions. These challenges are further exacerbated by a lack of interoperability [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThe core of the complexity in standardizing unstructured data is attributed to the large heterogeneity in the information, much of which is in an unstructured data format [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e]. Converting unstructured data into a structured format is a nontrivial challenge when dealing with specialized domains like cancer, where the medical terminology and oncology concepts are highly domain-specific [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]. Nevertheless, there is a growing recognition that addressing the intricacies of data standardization in specialized domains, such as cancer, necessitates a closer examination of information extraction methods. Data standardization methods for clinical notes largely focus on the concept of information extraction and the natural language processing (NLP) techniques that support it [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]. Traditional NLP methodologies typically depend on manually curated rules and human-annotated datasets for their training procedures. This results in named entity recognition (NER) or information extraction systems being less generalizable, limiting their portability [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. Specifically, these systems frequently demonstrate limited adaptability, as evidenced by the prevalent challenge of out-of-distribution prediction, where performance significantly deteriorates when models encounter data that differ from the training set. This lack of generalizability primarily stems from an over-reliance on the specific characteristics of the training data and labels.\u003c/p\u003e \u003cp\u003eRecent studies in information extraction from unstructured clinical notes have ventured away from manual rule creation and human-annotated datasets to leverage the capabilities of large language models (LLMs). For example, OntoGPT uses LLMs to query and extract ontology information from free text [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. In doing so, OntoGPT designs prompts for each named entity in the ontology and queries LLM to extract the information in a semi-structured format. OntoGPT was developed to support general ontology query-answer tasks and was not designed to handle large-scale ontologies such as mCODE. Therefore, the prompt generation method in the OntoGPT pipeline easily exceeds the token size limitation of LLMs, which is typically four thousand for modern LLMs. Moreover, OntoGPT requires representing an ontology in YAML language and requires a large amount of human labor to represent an ontology to YAML template.\u003c/p\u003e \u003cp\u003eTo address these challenges, we have advanced the OntoGPT framework by developing an extended pipeline, mCodeGPT. We leverage the fact that an ontology typically embeds an intricate hierarchical structure with various relations (such as a parent-child relation) between the named entities inside the ontology. This type of structure is naturally captured in a graph structure. The manual preparation of the YAML ontology template file can be replaced by an automated graph traversal algorithm, which generates a prompt by traversing the graphical model.\u003c/p\u003e"},{"header":"2 Methods","content":"\u003cp\u003eWe introduce a novel approach to NER with respect to ontologies. This approach performs NER in a zero-shot manner, which is distinct from prior approaches that require extensive training [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e, \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e].\u003c/p\u003e \u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003e2.1 Graphical Model\u003c/h2\u003e \u003cp\u003eAn ontology formalizes a shared understanding of concepts, or named entities, and their relations - typically in a hierarchical structure from general to specific. To represent named entities and their relations within the ontology efficiently, we leverage a graph structure, not only due to its efficiency for data representation but also its potential to automate the prompts generation by leveraging graph traversal algorithms. We represent the ontology using a Directed Acyclic Graph (DAG) \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(G(V,E)\\)\u003c/span\u003e\u003c/span\u003e, where \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(V\\)\u003c/span\u003e\u003c/span\u003e represents the set of named entities and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(E\\)\u003c/span\u003e\u003c/span\u003e represents the set of edges between the named entities.\u003c/p\u003e \u003cp\u003eWe introduce five attributes for each named entity \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({v}_{i}(\\in V)\\)\u003c/span\u003e\u003c/span\u003e:\u003c/p\u003e \u003cp\u003eAttribute(\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({v}_{i}\\)\u003c/span\u003e\u003c/span\u003e) ={Label(\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({v}_{i}\\)\u003c/span\u003e\u003c/span\u003e), Description(\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({v}_{i}\\)\u003c/span\u003e\u003c/span\u003e), IsLeaf(\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({v}_{i}\\)\u003c/span\u003e\u003c/span\u003e), Level(\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({v}_{i}\\)\u003c/span\u003e\u003c/span\u003e), Present\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(({v}_{i}\\)\u003c/span\u003e\u003c/span\u003e)}\u003c/p\u003e \u003cp\u003e(1)\u003c/p\u003e \u003cp\u003ewhere\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003eLabel(\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({v}_{i}\\)\u003c/span\u003e\u003c/span\u003e) is the name of the entity,\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eDescription(\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({v}_{i}\\)\u003c/span\u003e\u003c/span\u003e) is a string that represents the prompt message to be used for extracting the named entity \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({v}_{i}\\)\u003c/span\u003e\u003c/span\u003e from the input text, and\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eIsLeaf(\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({v}_{i}\\)\u003c/span\u003e\u003c/span\u003e) is a Boolean value that represents if the node is a leaf node (i.e., a node devoid of siblings in the graph).\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eLevel\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\left({v}_{i}\\right)\\)\u003c/span\u003e\u003c/span\u003e is an integer that represents the level of the graph model (tree) the node \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({v}_{i}\\)\u003c/span\u003e\u003c/span\u003e is from and\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003ePresent\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\left({v}_{i}\\right)\\)\u003c/span\u003e\u003c/span\u003e is a Boolean value that represents if the information associated with \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({v}_{i}\\)\u003c/span\u003e\u003c/span\u003e is present in the input text.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003cp\u003eEach directed edge \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({e}_{ij}\\in E\\)\u003c/span\u003e\u003c/span\u003e from the node \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({v}_{i}\\)\u003c/span\u003e\u003c/span\u003e to the node \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({v}_{j}\\)\u003c/span\u003e\u003c/span\u003e represents the relation between the two nodes. In the mCODE context, all relations are essentially \u0026ldquo;has\u0026rdquo; (or parent-child) relations where a named entity \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({v}_{i}\\)\u003c/span\u003e\u003c/span\u003e has a named entity \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({v}_{j}\\)\u003c/span\u003e\u003c/span\u003e as an attribute.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e2.2 Algorithms\u003c/h2\u003e \u003cp\u003eWe introduce three distinct algorithms that demonstrate varied strategies for capitalizing on LLMs in conjunction with ontology. Our comparative analysis seeks to highlight their strengths and limitations, thereby providing insight into the optimal conditions for enhancing LLM performance with ontology integration. An illustrative example for each of the three algorithms is provided in \u003cspan refid=\"Sec14\" class=\"InternalRef\"\u003eAppendix\u003c/span\u003e A.\u003c/p\u003e \u003cp\u003e \u003cb\u003eRoot-to-Leaf Streamliner (RLS)\u003c/b\u003e \u003c/p\u003e \u003cp\u003eAlthough the leaf node information is what is needed by LLM, the named entities on the path provide background information for the leaf node. The RLS strategically harmonizes the information from the root node to each leaf node. This results in a compact, yet informative, prompt that encapsulates the entire lineage of the ontology tree. For each leaf node \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({v}_{i}\\)\u003c/span\u003e\u003c/span\u003e where IsLeaf\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\left({v}_{i}\\right)\\)\u003c/span\u003e\u003c/span\u003e is true, RLS finds the path from the root node to \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({v}_{i}\\)\u003c/span\u003e\u003c/span\u003e,\u003cdiv id=\"Equ1\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ1\" name=\"EquationSource\"\u003e\n$$root\\to {v}_{a}\\to {v}_{b}\\to \\dots \\to {v}_{i}$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e2\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eand generates a prompt for \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({v}_{i}\\)\u003c/span\u003e\u003c/span\u003e. Given that \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({v}_{i}\\)\u003c/span\u003e\u003c/span\u003e alone might contain limited information, we incorporate comprehensive background information by including the descriptions of all its parent nodes. For example, consider the following path:\u003cdiv id=\"Equa\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equa\" name=\"EquationSource\"\u003e\n$$cancer patient \\left(root\\right)\\to tumor \\left({v}_{a}\\right)\\to tumor size \\left({v}_{b}\\right)\\to$$\u003c/div\u003e\u003c/div\u003e\u003cdiv id=\"Equ2\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ2\" name=\"EquationSource\"\u003e\n$$measurement method \\left({v}_{i}\\right)$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e3\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eInstead of Description\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\left({v}_{i}\\right)\\)\u003c/span\u003e\u003c/span\u003e being \u003cem\u003e\u0026ldquo;What is the tumor measurement method?\u0026rdquo;\u003c/em\u003e, RLS will create Description\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\left({v}_{i}\\right)\\)\u003c/span\u003e\u003c/span\u003e as \u0026ldquo;\u003cem\u003eIf the patient has a tumor, what is the measurement method to measure the tumor size?\u003c/em\u003e\u0026rdquo;. The detailed prompt format can be found in \u003cspan refid=\"Sec14\" class=\"InternalRef\"\u003eAppendix\u003c/span\u003e C.\u003c/p\u003e \u003cp\u003e \u003cb\u003eBreadth-First Ontology Pruner (BFOP)\u003c/b\u003e \u003c/p\u003e \u003cp\u003eIn clinical data extraction, it is crucial to recognize that an individual clinical note may not encapsulate all entities present in a comprehensive ontology. Utilizing an extensive prompt for information retrieval can be counterproductive, as it may induce \u0026ldquo;hallucination\u0026rdquo;, a scenario where models erroneously produce or identify non-existent entities within the data. These hallucinations often result from models being overly influenced by a detailed ontology, leading to false identifications that could undermine the accuracy of clinical decision-making. Consequently, careful pruning of ontological entities is imperative to ensure precision in the extraction process. BFOP is engineered to pinpoint a relevant subgraph tailored to the designated target text. This approach is inspired by the Chain-of-Thought prompting methodology [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e], which utilizes a parsing strategy akin to the breadth-first search algorithm, employing a hierarchical level-wise analysis.\u003c/p\u003e \u003cp\u003eEssentially, BFOP is a layer-wise prompt generation method. It refines the next-level named entity prompts based on whether the information of the named entities from the current level is present. Initially, entities on the first level (the root entity) of the ontology are parsed and transformed into prompts suitable for querying (\u003cspan refid=\"Sec14\" class=\"InternalRef\"\u003eAppendix\u003c/span\u003e D: Algorithm BFOP lines 1\u0026ndash;7). Subsequently, these prompts are fed into the LLM to extract pertinent information from patient notes (\u003cspan refid=\"Sec14\" class=\"InternalRef\"\u003eAppendix\u003c/span\u003e D: Algorithm BFOP line 8).\u003c/p\u003e \u003cp\u003eA salient feature of our approach is its dynamic pruning mechanism. If a particular named entity is absent in the patient's record, then it is natural to infer that its children will also be devoid of relevant data, which eliminates the need to parse them (\u003cspan refid=\"Sec14\" class=\"InternalRef\"\u003eAppendix\u003c/span\u003e D: Algorithm BFOP lines 9\u0026ndash;12). Therefore, in the next level, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(L+1\\)\u003c/span\u003e\u003c/span\u003e, based on the previous level (\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(L\\)\u003c/span\u003e\u003c/span\u003e) result, some with Present\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(({v}_{i}\\)\u003c/span\u003e\u003c/span\u003e) = False can be skipped.\u003c/p\u003e \u003cp\u003e \u003cb\u003eTwo-Phase Ontology Parser (2POP)\u003c/b\u003e \u003c/p\u003e \u003cp\u003eThe main idea of the 2POP methodology is that queries about the mere existence of a named entity are simpler to resolve than those seeking details about the entity, particularly in the context of intricate ontologies such as the mCODE. 2POP aims to isolate a pertinent subgraph from the comprehensive ontology that correlates with the target text. Unlike the incremental, hierarchical extraction process characteristic of BFOP, 2POP adopts a comprehensive approach, extracting the entire pertinent structure in one step. The 2POP algorithm employs a two-pronged strategy: First, it queries all leaf nodes, classifying the outcomes based on binary responses. Then, the second phase utilizes only the entities confirmed by a \u0026ldquo;Yes\u0026rdquo; response to create a prompt. This prompt is then utilized to prompt an LLM to extract information from unstructured text, thereby enabling the retrieval of relevant data (\u003cspan refid=\"Sec14\" class=\"InternalRef\"\u003eAppendix\u003c/span\u003e D: Algorithm 2POP).\u003c/p\u003e \u003cp\u003eThe procedure for NER using the mCodeGPT pipeline is shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"3 Experiments and Results","content":"\u003cdiv id=\"Sec6\" class=\"Section2\"\u003e\n\u003ch2\u003e3.1 Experiment setting\u003c/h2\u003e\n\u003cp\u003eWe evaluate the efficacy of the proposed LLM-based named entity extraction and the effectiveness of each prompt-generating method. The experiments were conducted using high-quality synthetic oncology notes, produced by the advanced generative model GPT3.5-turbo using OpenAI API (Chat Completions API). The model was configured with a temperature setting of 0.8 and a maximum token limit of 4,096 to create a dataset of simulated clinical notes. These notes encompassed a diverse range of cancer types, with the subject of each note being randomly selected from a predetermined list of cancer categories. Table\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e specifies the Cancer types in this experiment. A total of 400 cancer reports (clinical notes) were generated, each focusing on the characteristics and details specific to the cancer type it represents. For each note, a random cancer type was chosen from Table\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e to generate biologically meaningful descriptions.\u003c/p\u003e\n\u003cdiv class=\"gridtable\"\u003e\n\u003ctable id=\"Tab1\" border=\"1\"\u003e\u003ccaption\u003e\n\u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e\n\u003cdiv class=\"CaptionContent\"\u003e\n\u003cp\u003eTypes of Cancers\u003c/p\u003e\n\u003c/div\u003e\n\u003c/caption\u003e\n\u003cthead\u003e\n\u003ctr\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eCategory\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eTypes of Cancer\u003c/p\u003e\n\u003c/th\u003e\n\u003c/tr\u003e\n\u003c/thead\u003e\n\u003ctbody\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eSkin Cancers\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eBasal Cell Carcinoma, Squamous Cell Carcinoma, Melanoma\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eKidney Cancers\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eRenal Cell Carcinoma\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eBreast Cancers\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eDuctal Carcinoma In Situ, Invasive Ductal Carcinoma, Adenocarcinoma\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eSoft Tissue Sarcomas\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eOsteosarcoma, Liposarcoma, Rhabdomyosarcoma\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eBlood Cancers\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eLeukemia, Acute Lymphoblastic Leukemia, Acute Myeloid Leukemia, Chronic Lymphocytic Leukemia, Chronic Myeloid Leukemia\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eLymphomas\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eNon-Hodgkin Lymphoma, Hodgkin Lymphoma\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eBrain and Central Nervous System\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eGlioblastoma, Astrocytoma, Meningioma, Medulloblastoma\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eGynecologic Cancers\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eCervical Cancer, Ovarian Cancer, Uterine Cancer, Vaginal Cancer, Vulvar Cancer\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eRespiratory Cancers\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eLung Cancer, Small Cell Lung Cancer, Non-Small Cell Lung Cancer\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eGenitourinary Cancers\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eProstate Cancer, Bladder Cancer\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eGastrointestinal Cancers\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eColon and Rectal Cancer, Pancreatic Cancer, Stomach (Gastric) Cancer, Esophageal Cancer\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eEndocrine Cancers\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eThyroid Cancer\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eOther Organ-Specific Cancers\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eLiver Cancer, Testicular Cancer, Nasopharyngeal Cancer, Laryngeal and Hypopharyngeal Cancer, Penile Cancer, Oral and Oropharyngeal Cancer\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003ePediatric Cancers\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eNeuroblastoma, Wilms Tumor, Retinoblastoma\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\u003cp\u003eWe examined the effectiveness of various models, including open-source models and OpenAI models, along with the proposed methods, RLS, BFOP, and 2POP. The open-source models include four from GPT4ALL [\u003cspan class=\"CitationRef\"\u003e14\u003c/span\u003e], namely Orca-mini-13b, Nous-hermes-13b, Llama-2-7b-chat, WizardLM-13b-uncensored, and one from FastChat [\u003cspan class=\"CitationRef\"\u003e15\u003c/span\u003e], the Fastchat-vicuna-13b-1.5. The experiments were performed on a local server with an AMD EPYC 7742 64-core Processor. Optionally, GPU resources can be used if available, which can reduce the running time for queries.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec7\" class=\"Section2\"\u003e\n\u003ch2\u003e3.2 Results\u003c/h2\u003e\n\u003cdiv id=\"Sec8\" class=\"Section3\"\u003e\n\u003ch2\u003e3.2.1 Performance\u003c/h2\u003e\n\u003cp\u003eThis section evaluates the performance of mCodeGPT on named entity extraction tasks from the mCODE ontology using different LLM as backbones combined with different prompt generation methods. We apply the following rules when counting correct, error, and hallucinated named entities.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003e\u003cstrong\u003eCorrect\u003c/strong\u003e: The information extraction result (IE) is the same as the ground truth (GT), or semantically equivalent; for example, GT is \u0026ldquo;Female\u0026rdquo; but IE is \u0026ldquo;female\u0026rdquo;; GT is \u0026ldquo;Normal\u0026rdquo; but IE is \u0026ldquo;within the Norm range\u0026rdquo;; GT is \u0026ldquo;15-Jan-75\u0026rdquo; but IE is \u0026ldquo;Jan 15, 1975\u0026rdquo;\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003e\u003cstrong\u003eError\u003c/strong\u003e: The information extraction result is inconsistent with ground truth. This includes two cases:\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003e\u003cstrong\u003eMisidentification\u003c/strong\u003e: The named entity has information in the notes (GT) but the information extraction result is none (\u0026ldquo;N/A\u0026rdquo;, \u0026ldquo;non\u0026rdquo;, \u0026ldquo;Not mentioned\u0026rdquo;, \u0026ldquo;N/A (not mentioned in the text)\u0026rdquo;, etc.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003e\u003cstrong\u003eMisplacement\u003c/strong\u003e: The named entity has information in the notes, and the information extraction result is not correct, but from the notes itself (not hallucinated).\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003e\u003cstrong\u003eHallucination\u003c/strong\u003e: Information extraction results in hallucinated information (not from the original notes), i.e., information that is fabricated or generated and is not present in the original text.\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eTable\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e represents the accuracy of each model on the 400 clinical notes. With respect to the incorrect results, we also identified the incorrect incidence rate and hallucination rate. The table shows our pipeline generally performed better with the 2POP method, consistently having the highest accuracy. This aligns with the initial design intent of the method, to perform a first-round screening (ask presence) before prompting more difficult questions (ask details).\u003c/p\u003e\n\u003cp\u003eAmong the open-source models, Orca-mini-13b achieved the highest accuracy with 2POP at 0.937, closely followed by Llama-2-7b-chat at 0.925. Among all the models, the OpenAI model GPT3.5-turbo-16k outperformed its counterparts, achieving an accuracy of 0.956 using 2POP. By contrast, RLS tended to achieve the lowest accuracy across most models. This finding aligns well with the nature of the RLS method, which issues queries for all named entities straightforwardly, without leveraging hierarchical structuring or an easy-to-difficult design approach.\u003c/p\u003e\n\u003cp\u003eHallucinations happened quite rarely (\u0026lt;\u0026thinsp;2.0%) for both open-source models and Open-AI API-based models. In general, BFOP and 2POP slightly reduced hallucination compared with RLS, which can be explained by the designed chain-of-thought queries. Overall, the OpenAI models hallucinated less than the open-source models.\u003c/p\u003e\n\u003cdiv class=\"gridtable\"\u003e\n\u003ctable id=\"Tab2\" border=\"1\"\u003e\u003ccaption\u003e\n\u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e\n\u003cdiv class=\"CaptionContent\"\u003e\n\u003cp\u003eEvaluation of Information Extraction Efficacy. This table presents the assessment of information extraction performance using different model-method pairings, quantified through accuracy, error rate, and hallucination rate, which collectively sum to unity. The analysis was performed on a corpus of 400 notes. The bold fonts represent the best performance entry in each column.\u003c/p\u003e\n\u003c/div\u003e\n\u003c/caption\u003e\n\u003cthead\u003e\n\u003ctr\u003e\n\u003cth rowspan=\"2\" align=\"left\"\u003e\u0026nbsp;\u003c/th\u003e\n\u003cth colspan=\"3\" align=\"left\"\u003e\n\u003cp\u003eAccuracy\u003c/p\u003e\n\u003c/th\u003e\n\u003cth colspan=\"3\" align=\"left\"\u003e\n\u003cp\u003eError Rate\u003c/p\u003e\n\u003c/th\u003e\n\u003cth colspan=\"3\" align=\"left\"\u003e\n\u003cp\u003eHallucination Rate\u003c/p\u003e\n\u003c/th\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eRLS\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eBFOP\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003e2POP\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eRLS\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eBFOP\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003e2POP\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eRLS\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003eBFOP\u003c/p\u003e\n\u003c/th\u003e\n\u003cth align=\"left\"\u003e\n\u003cp\u003e2POP\u003c/p\u003e\n\u003c/th\u003e\n\u003c/tr\u003e\n\u003c/thead\u003e\n\u003ctbody\u003e\n\u003ctr\u003e\n\u003ctd colspan=\"10\" align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003eOpen-Source Models\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eOrca-mini-13b\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.792\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.887\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.937\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.194\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.100\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.049\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003e0.014\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.013\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.014\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eNous-hermes-13b\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003e0.843\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.836\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.920\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003e0.142\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.154\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.067\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.015\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003e0.010\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.013\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eLlama-2-7b-chat\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.840\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003e0.900\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.925\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.146\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003e0.087\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.063\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003e0.014\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.013\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.012\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eFastchat-vicuna-13b-1.5\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.787\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.867\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.902\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.194\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.116\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.088\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.019\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.017\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003e0.010\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eWizardLM-13b-uncensored\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.831\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.865\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003e0.940\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.153\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.120\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003e0.048\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.016\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.015\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.012\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd colspan=\"10\" align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003eOpenAI Models\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eGPT3.5-turbo-16k\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.865\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.952\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.956\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.120\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.038\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.034\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.015\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.010\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.010\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eGPT3.5-turbo\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.868\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.926\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.934\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.118\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.065\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.057\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.014\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003e0.009\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.009\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eGPT4\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003e0.879\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003e0.932\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.946\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003e0.108\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003e0.056\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.045\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.013\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.012\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.009\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eGPT4-32k\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.870\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.930\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003e0.950\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.118\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.059\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003e0.042\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.012\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.011\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003e0.008\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec9\" class=\"Section3\"\u003e\n\u003ch2\u003e3.2.2 Running Time\u003c/h2\u003e\n\u003cp\u003eThe running times varied across models depending on the method employed, which provides insights into the efficiency and directly impacts the practical utility and feasibility of each model-method combination.\u003c/p\u003e\n\u003cp\u003eFigure \u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e shows the running time for each model concerning each prompting method. In general, open-source models took longer to process depending on the local computing resources. WizardLM-13b-uncensored, exhibited considerably longer running times. This model required 80 hours with the RLS method to run on 400 notes. By contrast, the OpenAI models demonstrated considerably faster running times. The GPT3.5-turbo model emerged as one of the most efficient, with running times under 1.5 hours across all methods. It is worth noting that the maximum token per minute (TPM) of Azure OpenAI models differs by what pricing tier is being chosen, which can affect the frequency of calling OpenAI APIs and, therefore affect the OpenAI models' performance using our pipeline. For example, under the pricing tier s0, our Azure OpenAI limits up to 240 TPM (GPT3.5-turbo, GPT3.5-turbo-16k), 20 TPM (GPT4), and 60 TPM (GPT4-32k), respectively.\u003c/p\u003e\n\u003cp\u003eIt should be noted that the extended processing time does not necessarily imply ineffectiveness. All models exhibited only around a 5% accuracy drop compared to OpenAI's advanced models (GPT4, GPT4-32k) \u0026ndash; and, more importantly, they do not require any overhead for token consumption.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec10\" class=\"Section3\"\u003e\n\u003ch2\u003e3.2.3 Performance on Medical Abbreviations\u003c/h2\u003e\n\u003cp\u003eIn practice, physicians often use abbreviations in their notes. To assess our model\u0026rsquo;s performance in more realistic situations, we modified the clinical notes generation process and let the clinical notes embed as many abbreviations, drawn from various medical terminologies, as possible. The abbreviations are provided in \u003cspan class=\"InternalRef\"\u003eAppendix\u003c/span\u003e B.\u003c/p\u003e\n\u003cdiv class=\"gridtable\"\u003e\n\u003ctable id=\"Tab3\" border=\"1\"\u003e\u003ccaption\u003e\n\u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e\n\u003cdiv class=\"CaptionContent\"\u003e\n\u003cp\u003eComparative Performance Analysis of Notes with High Utilization of Medical Terminology Abbreviations Versus Notes Without Abbreviations.\u003c/p\u003e\n\u003c/div\u003e\n\u003c/caption\u003e\n\u003cthead\u003e\n\u003ctr\u003e\n\u003cth align=\"left\"\u003e\u0026nbsp;\u003c/th\u003e\n\u003cth colspan=\"3\" align=\"left\"\u003e\n\u003cp\u003eAccuracy\u003c/p\u003e\n\u003c/th\u003e\n\u003c/tr\u003e\n\u003c/thead\u003e\n\u003ctbody\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eRLS\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eBFOP\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e2POP\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003eOpen-Source Models\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eOrca-mini-13b\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.6589\u003csub\u003e\u0026darr;0.1334\u003c/sub\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.6807\u003csub\u003e\u0026darr;0.2064\u003c/sub\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.7051\u003csub\u003e\u0026darr;0.2318\u003c/sub\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eNous-hermes-13b\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.7254\u003csub\u003e\u0026darr;0.1178\u003c/sub\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.7262\u003csub\u003e\u0026darr;0.1102\u003c/sub\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.7601\u003csub\u003e\u0026darr;0.1602\u003c/sub\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eLlama-2-7b-chat\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.7191\u003csub\u003e\u0026darr;0.1208\u003c/sub\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.7441\u003csub\u003e\u0026darr;0.1562\u003c/sub\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.7600\u003csub\u003e\u0026darr;0.1651\u003c/sub\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eFastchat-vicuna-13b-1.5\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.6508\u003csub\u003e\u0026darr;0.1366\u003c/sub\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.6984\u003csub\u003e\u0026darr;0.1726\u003c/sub\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.7290\u003csub\u003e\u0026darr;0.1731\u003c/sub\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eWizardLM-13b-uncensored\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.6934\u003csub\u003e\u0026darr;0.1378\u003c/sub\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.7312\u003csub\u003e\u0026darr;0.1337\u003c/sub\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.7333\u003csub\u003e\u0026darr;0.2070\u003c/sub\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e\u003cstrong\u003eOpenAI Models\u003c/strong\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eGPT3.5-turbo-16k\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.7314\u003csub\u003e\u0026darr;0.1331\u003c/sub\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.7451\u003csub\u003e\u0026darr;0.2070\u003c/sub\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.7501\u003csub\u003e\u0026darr;0.2061\u003c/sub\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eGPT3.5-turbo\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.7266\u003csub\u003e\u0026darr;0.1411\u003c/sub\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.7199\u003csub\u003e\u0026darr;0.2065\u003c/sub\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.7334\u003csub\u003e\u0026darr;0.2010\u003c/sub\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eGPT4\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.7452\u003csub\u003e\u0026darr;0.1340\u003c/sub\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.7564\u003csub\u003e\u0026darr;0.1757\u003c/sub\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.7795\u003csub\u003e\u0026darr;0.1660\u003c/sub\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003ctr\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003eGPT4-32k\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.7460\u003csub\u003e\u0026darr;0.1243\u003c/sub\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.7614\u003csub\u003e\u0026darr;0.1686\u003c/sub\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003ctd align=\"left\"\u003e\n\u003cp\u003e0.7571\u003csub\u003e\u0026darr;0.1923\u003c/sub\u003e\u003c/p\u003e\n\u003c/td\u003e\n\u003c/tr\u003e\n\u003c/tbody\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\u003cp\u003eTable\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e summarizes the performance of the different models and prompting methods in terms of accuracy. GPT-4 demonstrated superior proficiency in interpreting medical terminologies and abbreviations. For all models and methods, the performance dropped by 11\u0026ndash;20% when compared to results based on notes without abbreviations. The decline in performance does not diminish the utility of the proposed tool, since medical abbreviations are usually limited and can easily be summarized. It is also worth noting that shows the worst-case performance because all medical terms were replaced by their abbreviations in the 400 notes, which is unlikely to happen in reality. A full list of common abbreviations of medical concepts can be found in \u003cspan class=\"InternalRef\"\u003eAppendix\u003c/span\u003e B.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec11\" class=\"Section3\"\u003e\n\u003ch2\u003e3.2.4 Heterogenous Difficulty of Named Entities\u003c/h2\u003e\n\u003cp\u003eThe performance varied across various named entities within specific information extraction tasks. Figure\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e illustrates the evaluation of named entity recognition performance within the dataset comprising 400 documents. Each bar in the graph represents the accuracy with which the system identifies each named entity, with a filled bar denoting flawless recognition across the entire corpus. The data indicates a perfect extraction for 65% of the named entities, while the accuracy for the remaining 35% varied between 25% and 99%.\u003c/p\u003e\n\u003cp\u003eThe named entities that achieved perfect performance primarily consisted of terms related to medical terms, genomic variants, and cancer-related information. These terms are technical and likely contain domain-specific abbreviations. By contrast, the imperfect (\u0026lt;\u0026thinsp;100% accurate on all 400 notes) named entities comprise terms pertinent to patient details, such as demographics, procedures, and medication administration. These terms are more focused on patient-related data and administrative dimensions.\u003c/p\u003e\n\u003c/div\u003e\n\u003c/div\u003e"},{"header":"4 Discussion","content":"\u003cp\u003eOur findings offer valuable insight into the possibilities and power of LLMs, especially when used with clinical ontologies. Beyond its direct application to cancer ontologies, our methodologies can be extended and applied to various other medical specialties and healthcare domains.\u003c/p\u003e \u003cp\u003eAs with any study, it is essential to consider the context and limitations of our findings. Despite achieving high accuracy, it is essential to have human oversight to ensure the fidelity of the results. There were instances where not every named entity was accurately identified. Thus, a review of the outcomes is necessary to prevent any oversight or misidentification.\u003c/p\u003e \u003cp\u003eTo promote in-context learning and increase mCodeGPT\u0026rsquo;s performance when handling abbreviations, a list of abbreviations can be injected into the prompt as background knowledge -- used as a one-shot exemplar. Earlier studies have shown that one-shot in-context learning [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e], by presenting in the prompt an example of the task it is asked to do, can significantly boost the performance of LLMs.\u003c/p\u003e \u003cp\u003eThere are several potential areas for future research for further refinement and adaptation of mCodeGPT. First, multimodal integration \u0026ndash; with the evolving capability of OpenAI GPT models to incorporate multimodality, mCodeGPT can be enhanced to leverage both textual and image-based information for NER tasks, especially in the fields of radiology and pathology. Second, the automated generation of ontologies becomes crucial for expanding into domains where ontological frameworks are not well-established. Parsing clinical notes and autonomously creating ontologies for NER is of paramount importance in such scenarios.\u003c/p\u003e"},{"header":"5 Conclusions","content":"\u003cp\u003eOur mCodeGPT architecture represents a significant advancement in cancer research and clinical informatics, introducing innovative methods for zero-shot extraction of structured data from complex clinical narratives. This transformation of unstructured information into actionable insights has the potential to fundamentally alter our comprehension and approach to cancer treatment and research. By streamlining the process of NER, mCodeGPT can substantially reduce the time and labor traditionally required for data extraction and standardization. This increased efficiency allows healthcare professionals to allocate more time to direct patient care and clinical assessments. Consequently, the reduced administrative burden may accelerate diagnostic procedures and the initiation of treatment, which could lead to enhanced patient outcomes.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eYa RA, Depinho M, Ernst K. Cancer Research: Past, Present and Future. \u003cem\u003eNature Reviews Cancer\u003c/em\u003e 2011;\u003cstrong\u003e11\u003c/strong\u003e:749\u0026ndash;54.\u003c/li\u003e\n\u003cli\u003eOsterman TJ, Terry M, Miller RS. Improving cancer data interoperability: The promise of the Minimal Common Oncology Data Elements (mCODE) initiative. \u003cem\u003eJCO Clin Cancer Inform\u003c/em\u003e 2020;\u003cstrong\u003e4\u003c/strong\u003e:993\u0026ndash;1001.\u003c/li\u003e\n\u003cli\u003eOsterman TJ, Yao JC, Krzyzanowska MK. Implementing innovation: Informatics-based technologies to improve care delivery and clinical research. \u003cem\u003eAm Soc Clin Oncol Educ Book\u003c/em\u003e 2023;\u003cstrong\u003e43\u003c/strong\u003e:e389880.\u003c/li\u003e\n\u003cli\u003eLi I, Pan J, Goldwasser J, \u003cem\u003eet al.\u003c/em\u003e Neural Natural Language Processing for unstructured data in electronic health records: A review. \u003cem\u003eComputer Science Review\u003c/em\u003e 2022;\u003cstrong\u003e46\u003c/strong\u003e:100511.\u003c/li\u003e\n\u003cli\u003eChin L, Andersen JN, Futreal PA. Cancer genomics: from discovery science to personalized medicine. \u003cem\u003eNat Med\u003c/em\u003e 2011;\u003cstrong\u003e17\u003c/strong\u003e:297\u0026ndash;303.\u003c/li\u003e\n\u003cli\u003eKowalczyk S, Shankar K. Data sharing in the sciences. \u003cem\u003eAnnual review of information science and technology\u003c/em\u003e 2011;\u003cstrong\u003e45\u003c/strong\u003e:247\u0026ndash;94.\u003c/li\u003e\n\u003cli\u003eWarner JL, Maddux SE, Hughes KS, \u003cem\u003eet al.\u003c/em\u003e Development, implementation, and initial evaluation of a foundational open interoperability standard for oncology treatment planning and summarization. \u003cem\u003eJ Am Med Inform Assoc\u003c/em\u003e 2015;\u003cstrong\u003e22\u003c/strong\u003e:577\u0026ndash;86.\u003c/li\u003e\n\u003cli\u003eParsing C. Speech and language processing. \u003cem\u003ePower Point Slides\u003c/em\u003e Published Online First: 2009.https://people.cs.pitt.edu/~litman/courses/cs2731_f19/lec/slp12_f19.pdf\u003c/li\u003e\n\u003cli\u003eSarawagi S. Information Extraction. \u003cem\u003eFoundations and Trends\u0026reg; in Databases\u003c/em\u003e 2008;\u003cstrong\u003e1\u003c/strong\u003e:261\u0026ndash;377.\u003c/li\u003e\n\u003cli\u003eCaufield JH, Hegde H, Emonet V, \u003cem\u003eet al.\u003c/em\u003e Structured prompt interrogation and recursive extraction of semantics (SPIRES): A method for populating knowledge bases using zero-shot learning. arXiv [cs.AI]. 2023.http://arxiv.org/abs/2304.02711\u003c/li\u003e\n\u003cli\u003eDevlin J, Chang M-W, Lee K, \u003cem\u003eet al.\u003c/em\u003e BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv [cs.CL]. 2018.http://arxiv.org/abs/1810.04805\u003c/li\u003e\n\u003cli\u003eLiu Y, Ott M, Goyal N, \u003cem\u003eet al.\u003c/em\u003e RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv [cs.CL]. 2019.http://arxiv.org/abs/1907.11692\u003c/li\u003e\n\u003cli\u003eWei J, Wang X, Schuurmans D, \u003cem\u003eet al.\u003c/em\u003e Chain-of-thought prompting elicits reasoning in large language models. \u003cem\u003eAdv Neural Inf Process Syst\u003c/em\u003e 2022;\u003cstrong\u003e35\u003c/strong\u003e:24824\u0026ndash;37.\u003c/li\u003e\n\u003cli\u003eAnand Y, Nussbaum Z, Duderstadt B, \u003cem\u003eet al.\u003c/em\u003e GPT4All: Training an assistant-style chatbot with large scale data distillation from GPT-3.5-turbo. 2023.http://static.nomic.ai.s3.amazonaws.com/gpt4all/2023_GPT4All_Technical_Report.pdf (accessed 2 Jan 2024).\u003c/li\u003e\n\u003cli\u003eZheng L, Chiang W-L, Sheng Y, \u003cem\u003eet al.\u003c/em\u003e Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv [cs.CL]. 2023.http://arxiv.org/abs/2306.05685\u003c/li\u003e\n\u003cli\u003eBrown T, Mann B, Ryder N, \u003cem\u003eet al.\u003c/em\u003e Language models are few-shot learners. \u003cem\u003eAdv Neural Inf Process Syst\u003c/em\u003e 2020;\u003cstrong\u003e33\u003c/strong\u003e:1877\u0026ndash;901.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"McWilliams School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, TX 77030","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-3940535/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-3940535/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThe rapidly growing collection of clinical free text data about cancer provides an unprecedented opportunity to harness rich, real-world insights for advancing cancer research and treatment. At the same time, there is a formidable challenge in efficiently extracting, standardizing, and structuring this vast and unstructured information to derive meaningful conclusions. In this paper, we address the problem of information extraction to standardize and structure such free text using large language models (LLMs) following the minimal Common Oncology Data Elements (mCODE\u0026trade;) structure. To this end, we implement mCodeGPT, an approach that ingests the cancer ontology knowledgebase and hierarchically generates prompts to extract relevant information from clinical notes. The generated output is subsequently organized and displayed in a tabular and structured format for improved analysis and interpretation. Our approach is annotation-free and eliminates the need for model training. mCodeGPT achieved over 95% accuracy for named entity recognition on mCODE concepts,\u003c/p\u003e","manuscriptTitle":"mCodeGPT: Bridging the Gap between Unstructured Medical Text and Structured Cancer Ontologies","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-02-09 12:19:12","doi":"10.21203/rs.3.rs-3940535/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"42d1318c-2299-4cb2-92a5-23a10a691b47","owner":[],"postedDate":"February 9th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":28663092,"name":"Biochemical Research Methods"}],"tags":[],"updatedAt":"2024-02-09T12:19:12+00:00","versionOfRecord":[],"versionCreatedAt":"2024-02-09 12:19:12","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-3940535","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-3940535","identity":"rs-3940535","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}