BioKGrapher: Initial evaluation of automated knowledge graph construction from biomedical literature.

This work utilizes two types of data sources: PubMed, for automatic KG construction, and clinical practice guideline that are used for reference alignment and evaluation purposes. The descriptive statistics of the corpora used are summarized in Table 1 . The PubMed annual baseline dataset, comprising over 37 million citations, serves to build global concept frequency weights. These weights are then applied to concepts identified in six publication sets related to specific conditions, including Breast Cancer, Cervical Cancer, Colorectal Cancer, Actinic Keratosis and Cutaneous Squamous Cell Carcinoma of the Skin (AK and SCC of the Skin), Malignant Melanoma, and Adult Soft Tissue Sarcomas. These publication sets were collected through MeSH descriptors. Table 1 Descriptive statistics of the corpora used for constructing and evaluating knowledge graphs. The PubMed annual baseline dataset serves to build global concept frequency weights. These weights are then applied to concepts found in six publication sets related to the conditions: Breast Cancer, Colorectal Cancer, Actinic Keratosis and Cutaneous Squamous Cell Carcinoma of the Skin, Malignant Melanoma and Adult Soft Tissue Sarcomas, collected through MeSH descriptors. Clinical practice guidelines of the same conditions serve as a baseline for the evaluation of the identified prevalent concepts. All six guidelines meet the highest possible level of the AWMF (S3). † denotes that the document was translated with the fairseq Transformer ported model wmt19-de-en, ⁎ denotes statistics are based on citations with abstract. Table 1 Data Set Descriptor Citations/Documents Citation has Abstract Token Token per Document (std) ⁎ Concepts [distinct] Concepts per Document (std) ⁎ Distinct Concepts per Document (std) ⁎ MEDLINE PubMed Baseline 24n 37,522,738 26,171,810 6,399,477,076 239.432 (97.874) 1,557,162,536 [150,628] 58.085 (24.948) 36.782 (13.393)  Breast Cancer  MeSH:D001943 344,172 273,194 72,423,131 261.885 (94.614) 18,284,652 [33,283] 65.959 (24.047) 40.379 (12.217)  Colorectal Cancer  MeSH:D015179 117,332 102,972 28,623,408 276.152 (89.047) 6,979,430 [21,791] 67.231 (22.183) 41.063 (11.290)  Endometrial Cancer  MeSH:D016889 25,326 22,420 6,153,435 272.816 (95.686) 1,523,742 [12,669] 67.471 (23.564) 41.122 (12.032)  AK and SCC of the Skin  MeSH:D055623;D002294 146,815 115,902 29,109,540 247.698 (95.575) 7,308,888 [29,284] 62.032 (23.392) 39.338 (12.549)  Malignant Melanoma  MeSH:D008545 104,531 78,341 19,080,198 239.594 (96.327) 4,949,795 [23,888] 61.949 (24.397) 39.106 (12.958)  Adult Soft Tissue Sarcomas  MeSH:D012509 35,681 20,980 5,205,761 239.567 (105.747) 1,366,514 [16,719] 62.618 (26.704) 40.753 (14.398) AWMF: 032/045OL v4.4 [58] Breast Cancer - - 120,617 - 24,566 [2,302] - - AWMF: 021/007OL v2.1 [59] Colorectal Cancer - - 84,182 - 16,343 [1,726] - - AWMF: 032/034OL v2.0 [60] Endometrial Cancer - - 135,257 - 20,925 [1,897] - - AWMF: 032/022OL v2.0 [61] AK and SCC of the Skin - - 211,265 - 39,566 [2,164] - - AWMF: 032/024OL v3.3 [62] Malignant Melanoma† - - 69,847 - 14,045 [1,858] - - AWMF: 032/044OL v1.1 [63] Adult Soft Tissue Sarcomas - - 65,849 - 13,663 [1,828] - - Descriptive statistics of the corpora used for constructing and evaluating knowledge graphs. The PubMed annual baseline dataset serves to build global concept frequency weights. These weights are then applied to concepts found in six publication sets related to the conditions: Breast Cancer, Colorectal Cancer, Actinic Keratosis and Cutaneous Squamous Cell Carcinoma of the Skin, Malignant Melanoma and Adult Soft Tissue Sarcomas, collected through MeSH descriptors. Clinical practice guidelines of the same conditions serve as a baseline for the evaluation of the identified prevalent concepts. All six guidelines meet the highest possible level of the AWMF (S3). † denotes that the document was translated with the fairseq Transformer ported model wmt19-de-en, ⁎ denotes statistics are based on citations with abstract. For the biomedical literature foundation of this work, the annual NLM PubMed baseline version of December 14, 2023, is used. Daily update files were included to incorporate citations up to May 2024 (Files 24n0001-24n1424). This corresponds to 37 , 522 , 738 citations, of which 26 , 171 , 810 citations contain an abstract. Descriptive statistics for the collected PubMed data, including extracted and linked concepts, are presented in Table 1 . The PubMed abstracts feature approximately 6.4 billion tokens, with an average of 239 tokens per abstract. It identifies over 1.5 billion concepts, 150 , 628 of which are distinct. Each condition-specific publication set varies in size, with Breast Cancer having the largest number of documents ( 344 , 172 ) and Adult Soft Tissue Sarcomas the fewest ( 35 , 681 ). The token count per abstract and the average number of concepts per abstract are consistent across conditions, with minor variations. The top semantic types across the entire PubMed can be found in Appendix A.3 . Evidence-based clinical practice guidelines from the German Guideline Program in Oncology (GGPO), 4 are used to align and evaluate the automatically constructed concept sets. GGPO is jointly funded by the Association of the Scientific Medical Societies in Germany (AWMF), the German Cancer Society and the German Cancer Aid. They provide clinical practice guidelines in machine-readable JSON formats, which are also used in the German Guideline Program in Oncology NLP Corpus (GGPONC) project [64] , [65] . From 30 German oncology guidelines, there are six English versions available in machine-readable format. Table 1 shows the available publication sets of conditions along with descriptive statistics on the extracted concepts. The guidelines, all meeting the highest quality level of AWMF (S3), provide a benchmark for evaluating the identified prevalent concepts. These guidelines range from around 65,849 to 211,265 tokens and encompass a substantial number of distinct concepts, averaging around 2,000 distinct concepts per guideline, that describe each condition. The guideline for Malignant Melanoma was translated using the fairseq Transformers port of wmt19de_en [66] , a model that has previously demonstrated reasonable performance in translating biomedical and clinical data between English and German [67] , [68] . The evaluation using these guidelines ensures that the identified concepts are based on the best available research evidence, forming a valid representation of each condition.

Henning Schäfer: Writing – original draft, Visualization, Software, Methodology, Investigation, Formal analysis, Conceptualization. Ahmad Idrissi-Yaghir: Writing – review & editing, Visualization, Software. Kamyar Arzideh: Writing – review & editing, Visualization, Software. Hendrik Damm: Writing – review & editing, Data curation. Tabea M.G. Pakull: Writing – review & editing, Data curation. Cynthia S. Schmidt: Writing – review & editing, Validation. Mikel Bahn: Writing – review & editing, Validation. Georg Lodde: Writing – review & editing, Software, Investigation. Elisabeth Livingstone: Writing – review & editing, Validation, Investigation. Dirk Schadendorf: Supervision, Resources. Felix Nensa: Supervision, Resources. Peter A. Horn: Writing – review & editing, Supervision, Conceptualization. Christoph M. Friedrich: Writing – review & editing, Supervision, Conceptualization.

The methods are organized into components that collectively describe how user-provided PubMed IDs (PMIDs) are transformed into structured knowledge representations. The “BioKGrapher Pipeline” shows an overview of the workflow, detailing how the tool processes biomedical literature to create KGs that can be explored or exported for further analysis. Next, “Named Entity Recognition and Entity Linking” explains how the application uses NLP to identify and normalize biomedical concepts within the provided document set. Following this, “Prevalent Concept Weights” describes the statistical methods used to weight the identified concepts based on their relevance to the specific document set, ensuring that the important concepts are highlighted. “Frequency Re-ranking” further refines these weights by incorporating local frequency data, addressing the challenge of balancing global and local relevance of concepts. “Hierarchical Structuring and Semantic Triples” covers the process of building the KGs hierarchical structure and defining the relationships between concepts, which are represented as semantic triples. Finally, “Guideline Alignment for Concept Evaluation” outlines the methods used to evaluate the relevance of the generated concepts by comparing them to those found in clinical practice guidelines. The proposed pipeline as shown in Fig. 1 for processing uploaded PMIDs, begins with the use of pre-cached concept annotations that are normalized and linked to the UMLS for each citation. Different mentions that refer to the same concept are consistently identified. After normalization, these concepts undergo a weighting process to determine their importance within the specific context of the document set, for example 104k Melanoma PMIDs. This is achieved by calculating the prevalence of each concept within the provided document set relative to its frequency across the entire PubMed database. The weighting is done using Kullback-Leibler divergence (KLD) [69] , a statistical measure that quantifies how much one probability distribution diverges from a second, expected probability distribution. In this case, KLD highlights terms that are disproportionately prevalent in the specific document set compared to the general PubMed database, thereby identifying concepts that are particularly relevant to the provided PMID set. To address the challenge of filtering out relevant concepts that are also common across other conditions due to their high global frequency, the pipeline includes a re-ranking process that incorporates local frequency weighting. This step adjusts the importance of concepts by considering their frequency within the local document set, ensuring that globally common terms, which may still be highly relevant in the specific context, are not down-weighted or excluded. Fig. 1 Steps in the BioKGrapher application pipeline for constructing KGs from PubMed citations. The process begins with NER+NEL, followed by concept weighting using Kullback-Leibler divergence. It continues with re-ranking through local frequency weighting, hierarchical structuring, terminology mapping, and the incorporation of relationships. The final step involves visualization and export options of the constructed KG. In the sunburst chart, different colors represent top-level categories within the hierarchy. Fig. 1 Steps in the BioKGrapher application pipeline for constructing KGs from PubMed citations. The process begins with NER+NEL, followed by concept weighting using Kullback-Leibler divergence. It continues with re-ranking through local frequency weighting, hierarchical structuring, terminology mapping, and the incorporation of relationships. The final step involves visualization and export options of the constructed KG. In the sunburst chart, different colors represent top-level categories within the hierarchy. Once the top concepts are identified and ranked, the BioKGrapher application proceeds to map these concepts to a standardized biomedical terminology, such as SNOMED CT or NCIt. This mapping is done using a path to root approach, which traces each concept to its hierarchical ancestors within the selected terminology. This approach constructs a hierarchical structure and forms a view on the KG, where each node represents a concept and its place within the broader condition specific biomedical ontology. To further enrich the KG, BioKGrapher incorporates semantic relationships between concepts as defined in the chosen set of concepts. These relationships are represented as edges connecting the nodes in the graph in the form of semantic triples. By default, BioKGrapher restricts triples to concepts within the chosen set, ensuring that relationships remain consistent with the established ontology. However, this approach also limits the introduction of new concepts into the graph when building the relations. The final Knowledge Graph can be explored using a variety of interactive visualization tools, such as treemaps 5 or sunburst charts 6 (as shown in Fig. 1 , Fig. 4 ), which provide a hierarchical view of the concepts related to a condition, with branches colored to represent different top-level categories within the hierarchy. The software provides sorted tables for each semantic type, allowing users to intuitively navigate the hierarchical relationships and the underlying data. For those who require further analysis or integration with other systems, the KG can also be exported in the Web Ontology Language (OWL) [70] format. MedCAT [45] is used for NER+NEL to identify and normalize biomedical concepts to the UMLS. The process is built upon two core components: the Vocabulary (VCB) and the Concept Database (CDB). The VCB is a comprehensive list of potential words that can appear in the documents, derived from Wikipedia and UMLS terms, and is used for spell-checking. The CDB is a structured repository of biomedical concepts, where each concept is represented by a UMLS Concept Unique Identifier (CUI) and can be referenced by multiple synonymous terms. NER begins with text preprocessing, by using a lightweight spell checker that relies on word frequency and edit distance to correct errors, ensuring that variations in terminology or misspellings still allow accurate concept identification. Spell-checking is performed against the VCB, but corrections are applied only if the corrected word matches a concept in the CDB. This process preserves critical abbreviations. In the preprocessing step, MedCAT uses SciSpaCy for tokenization and lemmatization. The detection of entity candidates is achieved through a dictionary-based approach along with a moving expanding window technique. Starting with a single word, the algorithm expands the window until a match with a concept in the CDB is found, which allows for identification of biomedical concepts, even when they span multiple words or appear in varied order. Concept disambiguation is tackled by using a self-supervised learning approach on the Medical Information Mart for Intensive Care (MIMIC-III) [71] database. This involves generating dense vector representations, which capture the semantic meaning of the surrounding text of each concept. When an ambiguous concept is detected, its context embedding is compared to the CDB. If the similarity between these embeddings exceeds a defined threshold, the concept is annotated and linked to the correct CUI. To efficiently retrieve concepts for any PMID set, BioKGrapher uses an SQLite database. The data is stored in the form PMID1: [‘CUI1’, ‘CUI2’], with both an inverted index on CUIs and an index on PMIDs. The indexing allows for quickly aggregating counts of CUIs across the entire PubMed database and enables fast filtering by PMIDs for counting CUIs within subsets of PMIDs for subsequent weighting through KLD. The set of concepts is defined by any UMLS Concept C = { c 1 , c 2 , … , c n } that appears at least once in PubMed. Let F ( c i ) represent the frequency of concept c i in the PubMed dataset, excluding the documents from the subset and let f ( c i ) represent the frequency of the same concept in the subset. The total number of concept occurrences in PubMed is N = ∑ i = 1 n F ( c i ) , and in the subset is n = ∑ i = 1 n f ( c i ) . The probability of a concept c i in the PubMed dataset is P ( c i ) = F ( c i ) N , and in the subset is p ( c i ) = f ( c i ) n . The Kullback-Leibler Divergence for a concept c i is given by: (1) D K L ( c i ) = p ( c i ) ⋅ log ⁡ ( p ( c i ) P ( c i ) ) The overall divergence measure for all concepts between the subset and the entire dataset is: (2) D K L ( Total ) = ∑ i = 1 n p ( c i ) ⋅ log ⁡ ( p ( c i ) P ( c i ) ) A higher D K L ( c i ) indicates that the concept c i is more prevalent or specific to the subset compared to its general frequency in PubMed. Conversely, a lower D K L ( c i ) suggests that the concept c i is either equally prevalent or less specific to the subset. If a concept c i does not appear in the subset (i.e., f ( c i ) = 0 ), the KLD for that concept is not defined, and a low constant is applied. A justification for choosing KLD is provided in Appendix A.1 . To balance the identification of relevant concepts within a subset, a re-ranking process is introduced that integrates local frequency weighting with the initial KLD. This approach addresses the limitation of overlooking relevant but globally common concepts due to their high frequency in the broad PubMed dataset. The final ranking score, S ( c i ) , for each concept combines normalized KLD and local frequency (LF) scores, applying adjustable weights to balance their influence. The final score is defined as S ( c i ) = α × D K L ˜ ( c i ) + β × L F ˜ ( c i ) , where α = 0.5 and β = 0.5 are the default weights for the normalized KLD and LF scores, respectively. This weighting scheme is designed to account for the uniqueness of concepts within the subset while still considering globally common terms. KLD and LF scores are normalized to a 0 to 1 scale for the final score. The process calculates KLD and LF for all concepts, normalizes them, and computes the final scores using the set weight parameters, and then ranks the concepts, resulting in prevalent terms for that subset. A threshold t can be set to focus on concepts above a certain score, highlighting the most prevalent concepts that represent a condition. To generate the hierarchical structure of the KG, the MRHIER Computable Hierarchies (MRHIER.RRF) file from the UMLS version 2023AB is used. This file contains the hierarchical paths to the root for each atom, represented as a list of AUIs. Nodes for each concept are created along their path to the root, forming the hierarchical structure. The list of prevalent concepts, identified using KLD and re-ranking, guides this process. The resulting hierarchy is dependent on the selected target terminology in the BioKGrapher application, such as SNOMED CT, NCIt, or others. To establish relationships (triples) within the knowledge graph, the MRREL Related Concepts (MRREL.RRF) file from the same UMLS version is used. This file provides semantic relationships between concepts, represented as subject-predicate-object triples. These triples are applied to the prevalent concepts identified by KLD and re-ranking them to create connections between nodes in the knowledge graph. The specific relationships included also depend on the selected target terminology. The relevance of the re-ranked concepts is evaluated by comparing them to concepts extracted from corresponding clinical practice guidelines using the same NER+NEL pipeline. This alignment serves as a benchmark to assess how well the automatically constructed concept sets correspond to those manually curated in guidelines. Precision, Recall, and F 1 -Score metrics are calculated to determine the overlap between the top-ranked concepts from the publication sets and the guideline concepts. Precision is defined as the proportion of relevant concepts (those appearing in the guidelines) among the top k concepts from the publication sets. Recall is the proportion of guideline concepts that are also present in the top k publication concepts. The F 1 -Score provides a harmonic mean of precision and recall, offering a balanced measure. The top k concepts are retrieved based on the re-ranking algorithm. By adjusting k , the metrics show how alignment with guideline concepts changes.

This Section is organized as follows. Initially, a descriptive analysis of the distribution of semantic types is presented to show similarities and differences of concepts between literature sourced from PubMed and clinical practice guidelines. Following this, melanoma generated concepts are used to showcase the top k re-ranked concepts across three semantic types. Subsequently, the evaluation focuses on comparing the top k generated and ranked concepts for six medical conditions against those found in their respective clinical practice guidelines. Additionally, visual representations, including a sunburst chart, triples tables for target mappings such as SNOMED CT and NCIt, and a Fruchterman-Reingold [72] layout of relationships illustrated using the melanoma KG, are provided to demonstrate the outputs produced by the BioKGrapher application. The radar charts in Fig. 2 show the semantic type distribution overlap between all six guidelines and publication sets. This offers a visual representation of how various medical concepts are emphasized across both sources. The distributions of certain semantic types, such as Findings (T033) and Therapeutic or Preventive Procedures (T061) , show strong alignment between guidelines and publications, indicating consistency in the emphasis on these concepts across both sources. For example, findings in breast cancer have distributions of 0.080 in guidelines and 0.083 in publications, reflecting close alignment. Similarly, therapeutic procedures in colon cancer show distributions of 0.062 in guidelines and 0.048 in publications, again indicating a consistent focus on treatment modalities. Fig. 2 Comparison of semantic type distributions in clinical practice guidelines and PubMed publications based on UMLS semantic types. †Actinic keratosis and squamous cell carcinoma. Fig. 2 Comparison of semantic type distributions in clinical practice guidelines and PubMed publications based on UMLS semantic types. †Actinic keratosis and squamous cell carcinoma. However, there are notable differences in other categories. The distribution of Neoplastic Processes (T191) in melanoma is lower in guidelines, at 0.024, compared to publications, at 0.044. This suggests that publications may place a greater emphasis on detailed aspects of neoplastic processes. This trend is also observed in the distribution of Diseases or Syndromes (T047) , where publications show a much higher distribution for melanoma, at 0.085, compared to guidelines, at 0.036. This indicates a stronger focus on disease characterization in publications. The distribution for Pharmacologic Substances (T121) is generally higher in publications, such as in melanoma, where it is 0.062 in publications versus 0.031 in guidelines, reflecting the more exploratory nature of academic articles in discussing potential new treatments. Similarly, Organic Chemicals (T109) are more prominently represented in publications, as seen in breast cancer where the distribution is 0.075 in publications compared to 0.039 in guidelines. Concept re-ranking results for three semantic types in the context of melanoma are presented in Table 2 : Diagnostic Procedure , Amino Acid, Peptide, or Protein , and Disease or Syndrome . Rankings are shown as KLD values, frequency counts, and the re-ranking score used to compute the final score for concepts associated with the condition. Table 2 Top concept ranking before and after KLD re-ranking across different categories for melanoma. Table 2 (a) Diagnostic Procedure Concept KLD↓ Concept Freq.↓ Concept Re-rank↓ Sentinel Lymph Node Biopsy 0.003334 Magnetic Resonance Imaging 28,081 Sentinel Lymph Node Biopsy 0.708 Biopsy 0.002091 Echocardiography 10,321 Biopsy 0.477 Immunohistochemistry 0.001117 Chest CT 5,496 Immunohistochemistry 0.231 Lymphoscintigraphy 0.000884 Biopsy 4,902 Lymphoscintigraphy 0.180 Examination of skin 0.000530 Sentinel Lymph Node Biopsy 2,991 Examination of skin 0.108 Skin self-examination 0.000381 Biopsy of lymph node 2,375 Skin self-examination 0.078 Excision biopsy 0.000335 Radionuclide Imaging 2,233 Excision biopsy 0.069 Diagnostic Imaging 0.000224 Contrast used 2,088 Fine needle aspiration biopsy 0.039 Fine needle aspiration biopsy 0.000188 X-Ray Computed Tomography 1,896 Contrast used 0.038 PET 0.000180 MRI of abdomen 1,875 PET 0.037 Self-Examination 0.000137 Colonoscopy 1,851 Biopsy of lymph node 0.035 PET/CT scan 0.000133 Radiographic Examination 1,773 Self-Examination 0.030 Fluorescein Angiography 0.000123 CT of abdomen 1,350 Radionuclide Imaging 0.029 Biopsy of skin 0.000123 Ultrasonography 919 PET/CT scan 0.029 Incisional biopsy 0.000102 Palpation 738 Fluorescein Angiography 0.026 (b) Amino Acid, Peptide, or Protein Concept KLD↓ Concept Freq.↓ Concept Re-rank↓ Ipilimumab 0.00320 Nivolumab 48,767 Ipilimumab 0.759 Nivolumab 0.00187 Pembrolizumab 18,756 Nivolumab 0.709 Tumor Antigens 0.00137 Ipilimumab 9,553 Pembrolizumab 0.381 Interferons 0.00131 Levothyroxine 4,814 Interferons 0.311 Interleukin-2 0.00127 Interferons 4,013 Tumor Antigens 0.301 Monoclonal Antibodies 0.00126 Troponin 3,221 Interleukin-2 0.284 Pembrolizumab 0.00121 Denosumab 3,095 Monoclonal Antibodies 0.279 Antibodies 0.00104 Antibodies 2,481 Antibodies 0.243 HLA-A2 Antigen 0.00056 Transaminases 1,554 HLA-A2 Antigen 0.126 Growth Factor 0.00055 Atezolizumab 1,514 Growth Factor 0.121 Melphalan 0.00051 Lipase 1,511 Melphalan 0.112 Protein-Serine-Threonine Kinases 0.00043 Fibrin 1,053 Protein-Serine-Threonine Kinases 0.095 Proteins 0.00040 Interleukin-2 917 Proteins 0.090 Cytokine 0.00039 Pregabalin 899 Cytokine 0.085 T-Cell Receptor 0.00036 Insulin 859 T-Cell Receptor 0.080 (c) Disease or Syndrome Concept KLD↓ Concept Freq.↓ Concept Re-rank↓ Sentinel node (disorder) 0.00149 Inflammatory disorder 5,565 Lentigo 0.584 Lentigo 0.00144 Hypothyroidism 4,690 Sentinel node (disorder) 0.574 Skin lesion 0.00096 Communicable Diseases 4,231 Skin lesion 0.423 Pigmented skin lesion 0.00076 Pneumonitis 3,856 Pigmented skin lesion 0.295 Vitiligo 0.00041 Pericardial effusion 3,343 Vitiligo 0.191 Melanosis 0.00038 Uveoparotid Fever 3,269 Hyperkeratosis 0.155 Hyperkeratosis 0.00024 Lymphadenopathy 3,240 Melanosis 0.147 Hypopigmentation disorder 0.00021 Obesity 2,922 Lymphadenopathy 0.113 Recurrent disease 0.00018 Hyperthyroidism 2,889 Pneumonitis 0.112 Local disease 0.00018 Colitis 2,758 Colitis 0.107 Glaucoma 0.00017 Hepatitis 2,414 Uveoparotid Fever 0.097 Hypophysitis 0.00013 Lymphocytopenia 2,355 Hypophysitis 0.089 Dermatologic disorders 0.00013 Hyperkeratosis 2,260 Hypopigmentation disorder 0.087 Actinic porokeratosis 0.00010 Skin lesion 2,053 Lymphocytopenia 0.073 Secondary glaucoma 0.00010 Thyroiditis 1,631 Recurrent disease 0.072 Top concept ranking before and after KLD re-ranking across different categories for melanoma. Concept KLD↓ Concept Freq.↓ Concept Re-rank↓ Sentinel Lymph Node Biopsy 0.003334 Magnetic Resonance Imaging 28,081 Sentinel Lymph Node Biopsy 0.708 Biopsy 0.002091 Echocardiography 10,321 Biopsy 0.477 Immunohistochemistry 0.001117 Chest CT 5,496 Immunohistochemistry 0.231 Lymphoscintigraphy 0.000884 Biopsy 4,902 Lymphoscintigraphy 0.180 Examination of skin 0.000530 Sentinel Lymph Node Biopsy 2,991 Examination of skin 0.108 Skin self-examination 0.000381 Biopsy of lymph node 2,375 Skin self-examination 0.078 Excision biopsy 0.000335 Radionuclide Imaging 2,233 Excision biopsy 0.069 Diagnostic Imaging 0.000224 Contrast used 2,088 Fine needle aspiration biopsy 0.039 Fine needle aspiration biopsy 0.000188 X-Ray Computed Tomography 1,896 Contrast used 0.038 PET 0.000180 MRI of abdomen 1,875 PET 0.037 Self-Examination 0.000137 Colonoscopy 1,851 Biopsy of lymph node 0.035 PET/CT scan 0.000133 Radiographic Examination 1,773 Self-Examination 0.030 Fluorescein Angiography 0.000123 CT of abdomen 1,350 Radionuclide Imaging 0.029 Biopsy of skin 0.000123 Ultrasonography 919 PET/CT scan 0.029 Incisional biopsy 0.000102 Palpation 738 Fluorescein Angiography 0.026 Concept KLD↓ Concept Freq.↓ Concept Re-rank↓ Ipilimumab 0.00320 Nivolumab 48,767 Ipilimumab 0.759 Nivolumab 0.00187 Pembrolizumab 18,756 Nivolumab 0.709 Tumor Antigens 0.00137 Ipilimumab 9,553 Pembrolizumab 0.381 Interferons 0.00131 Levothyroxine 4,814 Interferons 0.311 Interleukin-2 0.00127 Interferons 4,013 Tumor Antigens 0.301 Monoclonal Antibodies 0.00126 Troponin 3,221 Interleukin-2 0.284 Pembrolizumab 0.00121 Denosumab 3,095 Monoclonal Antibodies 0.279 Antibodies 0.00104 Antibodies 2,481 Antibodies 0.243 HLA-A2 Antigen 0.00056 Transaminases 1,554 HLA-A2 Antigen 0.126 Growth Factor 0.00055 Atezolizumab 1,514 Growth Factor 0.121 Melphalan 0.00051 Lipase 1,511 Melphalan 0.112 Protein-Serine-Threonine Kinases 0.00043 Fibrin 1,053 Protein-Serine-Threonine Kinases 0.095 Proteins 0.00040 Interleukin-2 917 Proteins 0.090 Cytokine 0.00039 Pregabalin 899 Cytokine 0.085 T-Cell Receptor 0.00036 Insulin 859 T-Cell Receptor 0.080 Concept KLD↓ Concept Freq.↓ Concept Re-rank↓ Sentinel node (disorder) 0.00149 Inflammatory disorder 5,565 Lentigo 0.584 Lentigo 0.00144 Hypothyroidism 4,690 Sentinel node (disorder) 0.574 Skin lesion 0.00096 Communicable Diseases 4,231 Skin lesion 0.423 Pigmented skin lesion 0.00076 Pneumonitis 3,856 Pigmented skin lesion 0.295 Vitiligo 0.00041 Pericardial effusion 3,343 Vitiligo 0.191 Melanosis 0.00038 Uveoparotid Fever 3,269 Hyperkeratosis 0.155 Hyperkeratosis 0.00024 Lymphadenopathy 3,240 Melanosis 0.147 Hypopigmentation disorder 0.00021 Obesity 2,922 Lymphadenopathy 0.113 Recurrent disease 0.00018 Hyperthyroidism 2,889 Pneumonitis 0.112 Local disease 0.00018 Colitis 2,758 Colitis 0.107 Glaucoma 0.00017 Hepatitis 2,414 Uveoparotid Fever 0.097 Hypophysitis 0.00013 Lymphocytopenia 2,355 Hypophysitis 0.089 Dermatologic disorders 0.00013 Hyperkeratosis 2,260 Hypopigmentation disorder 0.087 Actinic porokeratosis 0.00010 Skin lesion 2,053 Lymphocytopenia 0.073 Secondary glaucoma 0.00010 Thyroiditis 1,631 Recurrent disease 0.072 In the Diagnostic Procedure category, the term Sentinel Lymph Node Biopsy has the highest KLD value of 0.0033, indicating a high specificity to melanoma research. Despite a moderate frequency of 2,991 occurrences, it ranks highest after re-ranking with a score of 0.708, underscoring its importance in melanoma diagnostics. Similarly, Biopsy exhibits a high KLD value of 0.0021 and a high re-ranking score of 0.477, emphasizing its importance in the melanoma context, while also having a high frequency of 4,902 occurrences. In contrast, terms like Magnetic Resonance Imaging and Echocardiography, which appear frequently in melanoma publications with 28,081 and 10,321 occurrences respectively, have low KLD values and do not make it into the top 16 KLD melanoma concepts. This suggests that these procedures are more commonly referenced across various medical fields and are not uniquely tied to melanoma, resulting in low re-ranking positions. Other notable diagnostic procedures with high KLD values, lower occurrences, and high final re-ranking positions include Immunohistochemistry, Lymphoscintigraphy, Examination of skin, and Excision biopsy, indicating their specific association with melanoma. In the Amino Acid, Peptide, or Protein category, the drugs Ipilimumab and Nivolumab have the highest KLD values of 0.0032 and 0.0019, while also having high local frequencies of 9,553 and 48,767 occurrences, respectively. These terms are therefore top-ranked after re-ranking with scores of 0.759 and 0.709, emphasizing their critical role in melanoma treatment. Other terms, such as Pembrolizumab, Interferons, and Tumor Antigens, also achieved high KLD values and high re-ranking positions, indicating their specific relevance to melanoma. Despite lower frequency counts of 18,756 for Pembrolizumab and 4,013 for Interferons these terms end up with high final re-ranking scores, 0.381 for Pembrolizumab and 0.311 for Interferons. Conversely, terms like Levothyroxine and Troponin, which appear to be frequent in general medical literature, show low KLD values. These terms are classified as less specific to melanoma, as indicated by their absence from the top re-ranked positions. In the Disease or Syndrome category, Sentinel node (disorder) shows a high KLD value of 0.0015, highlighting its specificity for this category. Although it has a moderate frequency of 5,565 occurrences, it is top-ranked after re-ranking with a score of 0.574. Lentigo similarly shows a high KLD score of 0.0014 and a high re-ranking score of 0.584 despite a lower frequency count. Other notable concepts with high KLD values and high re-ranking positions include Skin lesion and Pigmented skin lesion. In contrast, terms like the generic Inflammatory disorder and Hypothyroidism, which are frequently mentioned in general medical literature with 5,565 and 4,690 occurrences respectively, show lower KLD values. These terms appear to be frequent in other medical fields as well. Immunotherapy-induced undesirable inflammations such as Hypophysitis, Pneumonitis, and Colitis are brought into an important position by the final ranking of the Disease or Syndrome -related concepts. The ranking tables for the same semantic types for other conditions can be found in Appendix A.4 . Fig. 3 shows Precision, Recall, and F 1 -Score metrics for guideline concept overlap at different concept retrieval thresholds from publication sets across multiple conditions (a–f). Top k concepts are retrieved based on the re-ranking algorithm to balance term frequency and KLD. The results show how the metrics change at different concept retrieval amounts. The points where the Precision and Recall curves intersect represent the optimal threshold values for each condition based on the reference guideline alignment. The F 1 -Score provides a single score to evaluate the performance, though higher Recall might be preferred at the cost of Precision when evaluating KGs. It can be seen across all conditions that the concepts with the highest ranking often appear within the guidelines from the start. In the conditions Actinic keratosis and squamous cell carcinoma of the skin , Breast Cancer , and Soft Tissue Sarcoma , some of the first concepts show a constant alignment with the guideline, resulting in a short horizontal precision curve before this continuously decreases. The F 1 -Scores peak at around 0.6 when approximately 2,000 concepts are retrieved, suggesting that this threshold offers an optimal balance between Precision and Recall for aligning with reference guidelines. Fig. 3 Precision, recall, and F 1 -Score for different thresholds and across multiple conditions. The figure shows the performance of the re-ranking algorithm that balances local frequency (LF) and Kullback-Leibler divergence (KLD) to rank prevalent concepts. Parameter weights: α  = 0.5, β  = 0.5. †Actinic keratosis and squamous cell carcinoma of the skin. Fig. 3 Precision, recall, and F 1 -Score for different thresholds and across multiple conditions. The figure shows the performance of the re-ranking algorithm that balances local frequency (LF) and Kullback-Leibler divergence (KLD) to rank prevalent concepts. Parameter weights: α  = 0.5, β  = 0.5. †Actinic keratosis and squamous cell carcinoma of the skin. For all six conditions, the resulting triples for various specialized target terminologies and classification systems were analyzed, focusing on purpose and domain. The results of the different target terminologies are shown in Table 3 . The hierarchical order of the constructed KG at the example of melanoma can be seen in Fig. 4 , where Pharmacologic Substances are shown. An additional example of Colon Cancer Operative Surgical Procedures can be seen in Appendix A.2 . Table 3 Summary of constructed knowledge graph triples statistics for different vocabularies. † Orphan nodes are concepts that have no connections within the graph in addition to their hierarchical classification through isa relationships. Table 3 Cancer Type Concepts+PTR Mean Node Depth (std) Relationships Nodes with Relations Orphan Nodes† Systematized Nomenclature of Medicine and Clinical Terms (SNOMED CT) AK-SCC of the Skin 3,975 7.673 (3.311) 19,114 2,180 1,795 Breast Cancer 3,821 6.875 (2.556) 12,254 1,846 1,975 Colon Cancer 3,705 7.036 (2.914) 12,658 1,720 1,985 Endometrial Cancer 3,744 7.289 (3.355) 14,102 1,842 1,902 Malignant Melanoma 3,763 7.030 (2.910) 14,416 1,913 1,850 Soft Tissue Sarcoma 3,799 7.648 (3.343) 15,046 1,909 1,890 National Cancer Institute Thesaurus (NCIt) AK-SCC of the Skin 2,203 6.900 (2.121) 7,732 1,071 1,132 Breast Cancer 2,049 6.676 (2.032) 3,814 739 1,310 Colon Cancer 1,948 6.500 (1.990) 3,510 681 1,267 Endometrial Cancer 1,936 6.888 (2.188) 5,604 779 1,157 Malignant Melanoma 2,054 6.560 (1.948) 4,720 885 1,169 Soft Tissue Sarcoma 2,291 7.117 (2.216) 9,086 1,221 1,070 Logical Observation Identifier Names and Codes (LOINC) AK-SCC of the Skin 613 2.008 (3.393) 558 235 378 Breast Cancer 643 2.431 (3.657) 658 278 365 Colon Cancer 697 2.197 (3.497) 664 282 415 Endometrial Cancer 636 2.485 (3.671) 672 282 354 Malignant Melanoma 708 2.483 (3.584) 752 313 395 Soft Tissue Sarcoma 643 2.004 (3.347) 556 243 400 Foundational Model of Anatomy (FMA) AK-SCC of the Skin 535 11.0 (2.562) 310 117 418 Breast Cancer 252 10.766 (3.065) 34 19 233 Colon Cancer 287 11.117 (3.162) 196 51 236 Endometrial Cancer 325 10.614 (2.751) 180 66 259 Malignant Melanoma 445 10.466 (2.581) 130 71 374 Soft Tissue Sarcoma 518 10.812 (2.654) 280 98 420 Online Mendelian Inheritance in Man (OMIM) AK-SCC of the Skin 178 3.174 (1.180) 540 166 12 Breast Cancer 134 3.202 (1.180) 436 129 5 Colon Cancer 138 3.035 (1.363) 384 127 11 Endometrial Cancer 139 3.093 (1.212) 492 134 5 Melanoma 193 3.378 (0.996) 668 188 5 Soft Tissue Sarcoma 188 2.818 (1.340) 600 168 20 Drugs in RxNorm AK-SCC of the Skin 87 - 6 6 81 Breast Cancer 152 - 22 20 132 Colon Cancer 118 - 16 12 106 Endometrial Cancer 120 - 24 21 99 Malignant Melanoma 82 - 10 9 73 Soft Tissue Sarcoma 96 - 14 10 86 Fig. 4 Hierarchical structure automatically generated from melanoma publications (hierarchy classification by NCIt) with a re-ranking threshold of 2,000 concepts. The figure shows the Pharmacologic Substances subcategory visualized in an interactive sunburst chart, displaying a maximum of 5 depths at a time. Fig. 4 Summary of constructed knowledge graph triples statistics for different vocabularies. † Orphan nodes are concepts that have no connections within the graph in addition to their hierarchical classification through isa relationships. Hierarchical structure automatically generated from melanoma publications (hierarchy classification by NCIt) with a re-ranking threshold of 2,000 concepts. The figure shows the Pharmacologic Substances subcategory visualized in an interactive sunburst chart, displaying a maximum of 5 depths at a time. SNOMED CT, covering a comprehensive range of clinical terms, exhibited the highest density of concepts and relationships. Actinic keratosis and squamous cell carcinoma (AK-SCC) of the Skin demonstrated the largest dataset with 3,975 concepts and 19,114 relationships. However, Breast Cancer showed a high count of orphan nodes without any relationships, totaling 1,975. The NCIt, dedicated to cancer-related clinical and research terms, also showed substantial coverage but with fewer concepts than SNOMED CT. Soft Tissue Sarcoma had the highest number of concepts (2,291) and relationships (9,086) within this dataset. Breast Cancer had the most orphan nodes, with 1,310 instances, indicating potential gaps in connectivity. LOINC, focusing on laboratory and clinical observations, presented fewer concepts and relationships. Malignant Melanoma had the highest figures with 708 concepts and 752 relationships. Orphan nodes were most prevalent in Soft Tissue Sarcoma , totaling 400, reflecting potential isolation in specific test-related data. Foundational Model of Anatomy (FMA), which catalogs human anatomical structures, showed lower overall coverage and a higher number of orphan nodes. This indicates that while FMA provides detailed anatomical concepts, these concepts are less interconnected within the KG, resulting in a sparser network. Soft Tissue Sarcoma had the highest number of concepts (518) and relationships (280). Orphan nodes were notably high in Soft Tissue Sarcoma at 420. Online Mendelian Inheritance in Man (OMIM), cataloging genetic disorders, included fewer concepts but demonstrated high connectivity in diseases with genetic implications, like Melanoma , which had 193 concepts and 668 relationships. Orphan nodes were minimal in OMIM, indicating a highly interconnected network of genetic data. This high degree of connectivity reflects the close relationships among genetic concepts within this terminology. RxNorm, focusing on clinical drugs and having a flatter hierarchical structure compared to other terminologies, had very limited data with no depth values recorded, meaning that the concepts did not form hierarchical relationships within the KG. Breast Cancer featured the highest number of concepts (152) and relationships (22). Orphan nodes, indicating unlinked drugs, were predominant in Breast Cancer with 132 nodes. The Fruchterman-Reingold [72] layout was applied to visualize the relationships between biomedical concepts extracted from PubMed publications related to melanoma, as generated by BioKGrapher. The resulting KG includes a diverse set of concepts categorized under different semantic types, such as Diagnostic Procedures , Therapeutic Interventions , and Pharmacologic Substances . The relationships in Fig. 5 were mapped using SNOMED CT and reveal clusters of related concepts. Fig. 5 Fruchterman-Reingold layout of concepts for melanoma, with isa relationships omitted. The edges represent bidirectional relationships, i.e., all concepts that have a relation in SNOMED CT or NCIt within a re-ranking threshold of the top 2,000 concepts. Fig. 5 Fruchterman-Reingold layout of concepts for melanoma, with isa relationships omitted. The edges represent bidirectional relationships, i.e., all concepts that have a relation in SNOMED CT or NCIt within a re-ranking threshold of the top 2,000 concepts. The zoomed-in subgraphs highlight specific clusters focused on diagnostic methods like Shave biopsy and Light Microscopy, and key therapeutic strategies such as Immune checkpoint inhibitor therapy. The visualization allows exploration of concept connections and shows the complexity and depth of relationships within the melanoma KG. The chosen layout helps to represent these relationships as clusters to identify important concept groupings.

This section reviews the methodologies and advances in automatic KG construction, highlighting the role of NLP and ontology integration. Initial efforts in biomedical KG construction were largely manual [5] . These early systems were either labor-intensive or limited in scope, but they established foundational ontologies and controlled vocabularies that continue to be integral to modern approaches. Building on this, semi-automated methods with expert curation and clinician-in-the-loop approaches have been increasingly used in recent years. Most modern methods for automatically constructing knowledge graphs rely on NLP techniques [33] , [34] . When deriving KGs from natural language, NER is typically employed to identify entities, which are then classified (entity typing) and uniquely resolved (entity linking) [35] . After basic knowledge discovery, approaches vary, often using some form of relationship extraction [36] and methods to refine the resulting KG [37] . For this work, when selecting NER+NEL tools and methods, it is important that entities are linked to large concept databases such as UMLS to capture a wide semantic range of concepts related to various conditions across the large amounts of publication abstracts. Tools for NER+NEL, such as MetaMap [38] , SemEHR [39] , cTAKES [40] , CLAMP [41] , and ScispaCy [42] , have contributed to advancements in this area. MetaMap was an early tool developed to map biomedical text to the UMLS Metathesaurus, but it struggles with spelling variations and ambiguous concepts. SemEHR further advanced the approach by applying manual rules, although the rule-based approach can be time-consuming. cTAKES, CLAMP and ScispaCy offer open-source approaches, with CLAMP being developed with focus to narrative patient reports and ScispaCy providing modern supervised NER approaches but only limited linking capabilities. Additionally, BioPortal [43] , [44] offers extensive support for a wide range of ontologies, making it a versatile tool for use cases beyond UMLS. The Medical Concept Annotation Toolkit (MedCAT) [45] has emerged with a self-supervised approach to address the limitations of earlier tools, showing recent improved performance in extracting UMLS concepts from various datasets. A critical component of the NER+NEL tools is contextualization, which enhances the understanding and extraction of entities within text. Approaches like Word2Vec [46] , fastText [47] , GloVe [48] , and more recently, Bidirectional Encoder Representations from Transformers (BERT) [49] , [50] , provide contextual embeddings that capture semantic relationships between words in context, rather than treating them as isolated tokens. These embeddings can be used for architectures such as Bidirectional Long Short-Term Memory networks (Bi-LSTM) [51] , [52] and Transformers to improve the accuracy and robustness of NER models. While BERT-based models generally excel in NLP tasks, the authors of MedCAT [45] found that the Word2Vec embeddings outperformed Bio_ClinicalBERT [53] on average for the task of linking ambiguous mentions to a large database. This was attributed to the use of pre-trained static BERT embeddings that may not effectively capture context-specific similarities, especially with tokenization challenges in the medical domain, where terminology often includes complex and rare structures. Tokenization is the process of breaking down text into smaller units, such as words or sub-words, that can be processed by models like BERT. These advancements in NER+NEL tools and methods are also reflected in the construction of KGs within the biomedical domain. Rossanez et al. [54] developed a semi-automatic method using NLP and linking to biomedical ontologies for generating KGs from biomedical texts, particularly focusing on Alzheimer's disease research. It was evaluated against manual extractions by physicians. Their results demonstrated a high efficiency in extracting relevant triples and linking them to ontologies. Health Knowledge Graph Builder (HKGB) [55] is a platform for constructing disease-specific health knowledge graphs from various sources. This platform incorporates a clinician-in-the-loop methodology to leverage medical expertise during the graph construction process to improve the accuracy. The authors applied HKGB for cardiovascular diseases and knee osteoarthritis. Xu et al. [56] focused on enhancing knowledge discovery in the medical field by developing a PubMed Knowledge Graph. While it integrates bio-entities extracted from 29 million PubMed abstracts using BioBERT [26] , along with supplementary article metadata such as authors, funding sources, and affiliations, it takes a broader approach to biomedical knowledge integration. In contrast, this work focuses on constructing KGs specific to medical conditions for more targeted applications. Integrating existing medical ontologies into knowledge graphs enhances their applicability in specialized domains. For example, Maghawry et al. [57] developed a framework that integrates the Human Disease Ontology and Symptom Ontology into a knowledge graph. This framework allows for the extraction of specific subgraphs, such as those focused on organ cancer, which can be particularly useful for specialized healthcare applications.

This study introduced BioKGrapher, a tool for automatically constructing biomedical KGs from unstructured text sources, such as PubMed abstracts. The evaluation of BioKGrapher shows that it effectively identifies and ranks biomedical concepts, achieving a high degree of overlap with guideline-recommended terms. Across multiple conditions, the tool maintains F 1 -Scores around 0.6 when retrieving the top 2,000 concepts, balancing precision and recall. Semantic differences between publications and guidelines indicate that publications may focus on different aspects compared to guidelines. Overall, BioKGrapher proves to be an effective tool for generating and applying KGs in biomedical research, showing particular promise in enhancing information retrieval and improving downstream applications. The level of concept overlap observed in the guideline literature suggests that restricting the literature based on common characteristics found in guidelines is not needed. However, future work could explore using bibliometrics such as study phase, level of evidence, and publication recency to enhance the initial publication selection, potentially improving the alignment of biomedical concepts in the KG compared to the guidelines. BioKGrapher allows for the adjustment of metadata and bibliometrics when providing custom PMID lists. By using a list of only the most recent PMIDs on a given topic, one can construct a KG that captures recent trends in the field. Regularly updating guidelines with bibliometric criteria could enhance the precision of predicting which concepts should be included in future guideline revisions. Beyond traditional structure-based KG evaluations, task-based methods now enhance applications like information retrieval, LLM integration, and machine learning. Combining graph neural networks with RAG for tasks like question answering is a promising direction, and future work could improve relation extraction from texts to capture new insights.

The semantic type distributions between clinical practice guidelines and biomedical publications reveal both alignments and differences. For example, categories such as Findings and Therapeutic Procedures show a high degree of consistency, indicating similar priorities in both sources. However, differences are noticeable in areas such as Neoplastic Processes and Pharmacologic Substances , where publications tend to focus more on exploratory topics that may not yet be fully reflected in clinical practice guidelines. This divergence could highlight the different roles of research literature and clinical practice guidelines where the former may explore new frontiers, and the latter consolidates established knowledge. The translation from literature to clinical practice also reflects into the unrestricted selection of literature and therefore impacts the alignment evaluation. The re-ranking process effectively prioritizes concepts specific to certain conditions, such as melanoma and breast cancer. For instance, concepts like Sentinel Lymph Node Biopsy and Ipilimumab are ranked highly in their respective categories, reflecting their relevance in the diagnosis and treatment of melanoma. This indicates that the re-ranking mechanism, which balances local frequency and KLD, successfully identifies key concepts that are prevalent and frequent in the subset. However, common concepts with high global frequency, such as Magnetic Resonance Imaging, are down-weighted unless they have specific relevance to the condition, which underscores the importance of context-specific ranking parameters in KG construction. As the weighting of KLD and LF can be adjusted via the parameters, the final re-ranking of the BioKGrapher concepts can be fine-tuned to emphasize either specificity to the condition (as indicated by higher KLD values) or prevalence in the literature (as indicated by higher LF counts). For instance, in the Diagnostic Procedure category of melanoma, increasing the weight of KLD would push terms like Sentinel Lymph Node Biopsy and Biopsy even higher in the rankings, due to their strong association with melanoma. Conversely, prioritizing LF might elevate more commonly referenced procedures such as Magnetic Resonance Imaging and Echocardiography, which are frequently mentioned across various medical fields. The focus during re-ranking may vary depending on the intended application of the KG and the user. Therefore, the weighting is made adjustable in the software, allowing for modification from the default values of 0.5 for both frequency and KLD. The evaluation metrics demonstrate the effectiveness of the re-ranking algorithm in aligning automatically extracted concepts with those in clinical practice guidelines. The precision curves start high, indicating that the top-ranked concepts are highly relevant to the guidelines. However, precision generally decreases as more concepts are included, reflecting the broader scope of biomedical publications compared to the more focused nature of clinical practice guidelines. The F 1 -Score peaks suggest that a balance between precision and recall can be optimized at specific thresholds, providing practical guidance for KG construction with a focus on conditions. The analysis of constructed KGs across various vocabularies reveals differences in node depth, relationship density, and the presence of orphan nodes. These variations reflect the different focuses of each vocabulary, with SNOMED CT offering broader clinical coverage, while NCIt is more specialized in oncology. The presence of orphan nodes, particularly in more focused ontologies like RxNorm and FMA, suggests potential gaps in connectivity that could impact the usability of the graphs. The resulting KGs are versatile, enabling a range of potential use cases in intelligent medicine and healthcare. One possible key application is in clinical decision support systems, where KGs can assist healthcare providers in diagnosing conditions and suggesting evidence-based treatments that may align with the conceptual view on recent literature. Due to the nature of the proposed KLD re-ranking approach, the KGs not only represent strong associations but also strong disassociations, which could be employed for abnormality checks, flagging unusual or inconsistent data in patient records. For instance, if symptoms and treatments for a patient do not typically align, the KG could alert the clinician to this anomaly, prompting further investigation. The appropriate evaluation of ontologies and KGs across multiple conditions is difficult to implement. In this work, it was decided to leverage extracted concepts from consensus-based clinical practice guidelines as reference terminology. In addition to advantages such as a large data basis for a variety of conditions, this also has some disadvantages. For example, it does not achieve the evaluation quality that would be possible with a manual evaluation by domain experts and may lag behind the latest scientific discoveries due to the time required for validation and consensus-building. The initial extraction of concepts is based on the same extraction pipeline that produces references of silver-standard quality, which therefore may contain biases in entity extraction and linking that leads to degradation or improvement of the reported results. Until now, the relations have been obtained from existing terminologies and classification systems such as SNOMED CT or NCIt, which on the one hand guarantees correctness, but on the other hand means that no novel relations, for example from the latest publications, can be added to the KG. This principle also pertains to concepts. However, because BioKGrapher is open-source, users can enhance or replace UMLS MRREL relationships or concepts with NLP-mined relations or other data sources to capture novel concepts and emerging relations from the latest literature. It is also possible to recognize new concepts by separating the entity recognition process from the entity linking process during indexing.

Two downstream tasks are evaluated to further evaluate the effectiveness of the proposed BioKGrapher approach. The first task focuses on document multi-label classification using Adapter infusion, while the second task presents a small case study on drug repurposing. These evaluations are conducted in addition to the prior alignment analysis on clinical practice guidelines and descriptive statistical results. Recent approaches have focused on leveraging and embedding knowledge from KGs for downstream tasks. One of them is Mixture-of-Partitions (MoP) [29] , an infusion approach that can handle large KGs by partitioning it into smaller subgraphs and infusing their knowledge into BERT style models using Adapters [73] , [30] . Adapters are new initialized modules inserted between the layers of a pre-trained Transformer. For each sub-graph G k , the tail entity in each triple ( h , r , t ) ∈ G k is removed, and the remaining elements are transformed into a sequence of tokens: [CLS] h [SEP] r [SEP]. The sub-graph-specific Adapter is then trained to predict the tail entity using the representation of the [CLS] token. The parameters Φ G k are optimized by minimising the cross-entropy loss [74] . This allows information from the knowledge graph to be embedded in the models, which has led to improved performance on multiple biomedical downstream tasks (Natural Language Inference (NLI), Question Answering (QA) and Classification) when applying a large SNOMED CT-based knowledge graph [29] . For this evaluation, a large cancer KG is generated using the BioKGrapher application. All publications in which MeSH terms appear under the tree number C04, which represents neoplasms, are used to select cancer related publications. This resulted in 3,596,405 PMIDs as input for the application. The top 15,000 concepts, representing about 20% of all possible concepts in the publication set, were re-ranked and used to build a large cancer relation network using intra-concept triples from NCIt. This network, represented as triples, was then used for pre-training Adapter infusion on three models: BioBERT [26] , SciBERT [27] , and BiomedBERT (former PubMedBERT) [28] for comparison purpose. The graph was partitioned into five sub-graphs using METIS [75] and pre-training was performed for one epoch using the publicly available implementation from [29] . The resulting configuration used for the infusion is presented in Table 5 . In this setup, SFull and S20Rel, which were utilized in prior research, are compared to BioKGrapher Neoplasms, the cancer-specific KG designed to better align with the HoC data domain. The effectiveness of the infusion is evaluated using a downstream multi-label classification task on the Hallmarks of Cancer (HoC) corpus [76] , [77] , [78] . The corpus is derived from 1,852 PubMed abstracts. In this corpus, class labels are manually annotated by experts according to the HoC taxonomy [79] , [80] . While the taxonomy comprises 37 classes, organized hierarchically, only the ten top-level classes are used for consistency with previous research. Following the MoP experiments [29] , the publicly available train/dev/test splits provided by the Biomedical Language Understanding and Reasoning Benchmark (BLURB) [28] are used. Results are reported as the average micro F 1 -Score performance across five runs with standard deviation. The performance on the multi-label classification downstream task on the HoC corpus can be seen in Table 4 . The results show effectiveness of using the BioKGrapher Neoplasms KG for Adapter infusion. The MoP BioKGrapher Neoplasms variant increased the micro F 1 -Score across all three models SciBERT, BioBERT, and BiomedBERT compared to the previously reported results for MoP (SFull) and MoP (S20Rel). Notably, the highest score on HoC was observed with BiomedBERT, where the F 1 -Score increased by 0.85 percentage points over MoP (S20Rel). Despite the smaller size of the BioKGrapher Neoplasms KG (68,995 triples) relative to SFull and S20Rel, its tailored cancer-specific knowledge appears to enhance on the cancer specific HoC corpus, leading to improved downstream performance. Table 4 Micro F 1 -score HoC multi-label classification results grouped by model and knowledge graph adapter infusion variants averaged across 5 runs with standard deviation. † denotes results from MoP authors [29] . Table 4 Model F 1 -Score (%) SciBERT † 80.5 ±0.6   MoP (SFull) † 81.5 ±0.4   MoP (S20Rel) † 81.8 ±0.7   MoP (BioKGrapher Neoplasms) 82.7 ±0.4 

 BioBERT † 81.4 ±0.6   MoP (SFull) † 81.5 ±0.9   MoP (S20Rel) † 82.5 ±1.1   MoP (BioKGrapher Neoplasms) 82.9 ±0.6 

 BiomedBERT (PubMedBERT) † 82.3 ±0.5   MoP (SFull) † 82.7 ±0.6   MoP (S20Rel) † 83.3 ±0.3   MoP (BioKGrapher Neoplasms) 84.1 ±0.3 Table 5 Statistics of the knowledge graphs by MoP authors in comparison to BioKGrapher neoplasms knowledge graph. Table 5 Entities Relations Triples SFull 302,332 229 4,129,726 S20Rel 263,808 20 1,750,677 BioKGrapher Neoplasms 15,000 27 68,995 Micro F 1 -score HoC multi-label classification results grouped by model and knowledge graph adapter infusion variants averaged across 5 runs with standard deviation. † denotes results from MoP authors [29] . Statistics of the knowledge graphs by MoP authors in comparison to BioKGrapher neoplasms knowledge graph. Drug repurposing involves finding new therapeutic uses for existing drugs [81] . The semantic type Amino Acid, Peptide, or Protein (T116) as defined by the UMLS Semantic Network includes a wide range of biological molecules, such as amino acids, peptides, proteins, and protein-based therapeutics like monoclonal antibodies. Table 2 can give an impression of commonly used therapies associated with a disease that fall under this semantic type. Conversely, Neoplastic Processes associated with a drug can be determined on the basis of all publications in which the drug appears. In this small downstream case study, the aim is to retrieve already known indications using the BioKGrapher application on the example of Nivolumab (MeSH:D000077594) and Rituximab (MeSH:D000069283). This approach could be extended to identify novel associations for potential drug repurposing. By analyzing the ranked associations, researchers can explore ABC relations [82] and identify possible applications of existing drugs beyond their current indications. For Nivolumab and Rituximab, two large publication sets in which the drugs appear are collected and entered into the BioKGrapher application as a list of PMIDs. Subsequently, re-ranking is performed, and the results are sorted in descending order for the semantic type Neoplastic Process (T191) to find strong associated conditions. From the ranking results, the top neoplasms associated with each drug are extracted and manually tested against evidence in literature. The results of this small case study as shown in Table 6 provide clear evidence supporting the use of Nivolumab and Rituximab for various neoplastic conditions beyond their main indications. Table 6 Relevance scores for Nivolumab and Rituximab for top associated neoplasms (neoplastic process). Table 6 (a) Nivolumab (b) Rituximab Condition Re-rank↓ Evidence Melanoma 0.963 [83] Non-Small Cell Lung Carcinoma 0.711 [84] Metastatic Renal Cell Carcinoma 0.324 [85] Clear Cell Renal Cell Carcinoma 0.173 [86] Renal Cell Carcinoma 0.155 [87] Renal Carcinoma 0.153 [87] Malignant Neoplasm Of Stomach 0.104 [88] SCC Of The Head And Neck 0.104 [89] Secondary Neoplasm 0.100 - Metastatic Malignant Neoplasm To Brain 0.093 [90] Hodgkin Disease 0.078 [91] Cutaneous Melanoma 0.060 [83] , [92] Liver Carcinoma 0.060 [93] Secondary Malignant Neoplasm Of Liver 0.058 - Adenocarcinoma Of Lung (Disorder) 0.057 [94] Malignant Head And Neck Neoplasm 0.052 [89] Classical Hodgkin's Lymphoma 0.048 [91] Adenocarcinoma 0.046 [94] Secondary Malignant Neoplasm Of Lung 0.045 - Condition Re-rank↓ Evidence THRLBCL 1.000 [95] Lymphoma, Non-Hodgkin 0.691 [96] Lymphoma, Follicular 0.496 [97] Chronic Lymphocytic Leukemia 0.369 [98] B-Cell Lymphomas 0.362 [99] Mantle Cell Lymphoma 0.236 [100] Lymphoproliferative Disorders 0.225 [101] Marginal Zone B-Cell Lymphoma 0.127 [102] Waldenstrom Macroglobulinemia 0.102 [103] Residual Tumor 0.071 [104] , [105] Microglioma 0.062 [106] Burkitt-Like Lymphoma 0.049 [107] Hodgkin Disease 0.047 [108] Small Lymphocytic Lymphoma 0.046 [109] Hematologic Neoplasms 0.034 [110] Hairy Cell Leukemia 0.034 [111] Monoclonal Gammapathies 0.033 [112] Mediastinal large B-cell lymphoma 0.029 [113] Primary Cutaneous B-Cell Lymphoma 0.029 [114] Relevance scores for Nivolumab and Rituximab for top associated neoplasms (neoplastic process). Condition Re-rank↓ Evidence Melanoma 0.963 [83] Non-Small Cell Lung Carcinoma 0.711 [84] Metastatic Renal Cell Carcinoma 0.324 [85] Clear Cell Renal Cell Carcinoma 0.173 [86] Renal Cell Carcinoma 0.155 [87] Renal Carcinoma 0.153 [87] Malignant Neoplasm Of Stomach 0.104 [88] SCC Of The Head And Neck 0.104 [89] Secondary Neoplasm 0.100 - Metastatic Malignant Neoplasm To Brain 0.093 [90] Hodgkin Disease 0.078 [91] Cutaneous Melanoma 0.060 [83] , [92] Liver Carcinoma 0.060 [93] Secondary Malignant Neoplasm Of Liver 0.058 - Adenocarcinoma Of Lung (Disorder) 0.057 [94] Malignant Head And Neck Neoplasm 0.052 [89] Classical Hodgkin's Lymphoma 0.048 [91] Adenocarcinoma 0.046 [94] Secondary Malignant Neoplasm Of Lung 0.045 - Condition Re-rank↓ Evidence THRLBCL 1.000 [95] Lymphoma, Non-Hodgkin 0.691 [96] Lymphoma, Follicular 0.496 [97] Chronic Lymphocytic Leukemia 0.369 [98] B-Cell Lymphomas 0.362 [99] Mantle Cell Lymphoma 0.236 [100] Lymphoproliferative Disorders 0.225 [101] Marginal Zone B-Cell Lymphoma 0.127 [102] Waldenstrom Macroglobulinemia 0.102 [103] Residual Tumor 0.071 [104] , [105] Microglioma 0.062 [106] Burkitt-Like Lymphoma 0.049 [107] Hodgkin Disease 0.047 [108] Small Lymphocytic Lymphoma 0.046 [109] Hematologic Neoplasms 0.034 [110] Hairy Cell Leukemia 0.034 [111] Monoclonal Gammapathies 0.033 [112] Mediastinal large B-cell lymphoma 0.029 [113] Primary Cutaneous B-Cell Lymphoma 0.029 [114] For Nivolumab, the re-ranking results strongly associate it with melanoma with a re-ranking score of 0.963, which aligns with its known use as a treatment for advanced melanoma. Additionally, strong associations are found with non-small cell lung carcinoma with a re-ranking score of 0.711 and metastatic renal cell carcinoma with a re-ranking score of 0.324, which are also consistent with established clinical applications [84] , [85] . In the case of Rituximab, the top-ranked condition is THRLBCL ( T-cell/histiocyte-rich large B-cell lymphoma ) with a perfect re-ranking score of 1.000, indicating a strong association [95] . Other highly relevant neoplasms include non-Hodgkin lymphoma with a re-ranking score of 0.691 and follicular lymphoma with a re-ranking score of 0.496, which are well-documented indications for Rituximab [96] , [97] .

During the preparation of this work, the authors used DeepL Write and Grammarly in order to enhance language quality, grammar, and style. After using these tools/services, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication.

In the rapidly evolving field of healthcare and biomedical research, the exponential growth of data presents both opportunities and challenges. The advance of biomedical research methods, increase in clinical trials and the continuous generation of electronic health records have led to an exponential growth in biomedical information [1] . This flood of data contains a broad spectrum of knowledge, from clinical trial results and clinical observations to detailed patient records and results published in scientific journals. Among these resources, the PubMed database 2 stands out as a treasure of biomedical knowledge, providing insights into a variety of conditions, treatments, and outcomes. However, the volume and complexity of this information, much of which is only available in unstructured forms, often obscures valuable connections and insights, making it increasingly difficult for researchers and practitioners to navigate this vast and complicated landscape. One possible approach to deal with complexity is the representation of knowledge at the conceptual level through ontologies and knowledge graphs (KGs). At their core, knowledge graphs represent a collection of interlinked descriptions of entities, objects, events, or concepts allowing for a structured way to manage knowledge [2] , [3] , [4] . The manual construction of knowledge graphs is a labor-intensive and time-consuming process, often requiring extensive domain expertise and continuous updates to reflect new discoveries or changes in recent publications. Given the rapid pace of data generation, there is a critical need for more efficient methods of knowledge graph construction [5] , considering that the publication rate in the biomedical field is too high for curators to keep up with, making it difficult to find information using manual approaches. [6] . This necessity has led to the exploration and development of automatic knowledge graph construction methodologies [7] . Automatic and semi-automatic knowledge graph construction aims to minimize human intervention by leveraging algorithms and computational techniques to extract, organize, and link data automatically [8] , [9] , [5] . This approach involves the processes of data acquisition, entity recognition, relationship extraction, and integration into a graph structure [10] . Advances in machine learning, and particularly in natural language processing (NLP), have opened new ways for extracting meaningful information from unstructured data sources. Once constructed, KGs can support a variety of applications, such as improved information retrieval, semantic search functions and expert systems [11] . In addition, the relationships between entities enable a range of applications, including predictive modeling [12] , drug discovery [13] , drug repurposing [13] , decision support [14] , patient stratification [15] , [16] , and precision medicine, and the extraction of genotype-phenotype correlations [17] . A particular example would be their use in molecular tumor boards [18] . Structured knowledge representations in the form of ontologies and KGs also open up important areas in the field of machine learning, where KGs can form a semantic framework for tasks such as relation extraction [19] , question answering [20] and improving large language models (LLMs) [21] , [22] , [23] , [24] . In the area of recommendation systems using machine learning methods, ontologies and KGs are increasingly being applied [25] . In this study, the BioKGrapher application is introduced and evaluated for constructing knowledge graphs from large-scale publication data, focusing on biomedical concepts that are prevalent, meaning they are widespread and commonly occurring, in relation to specific medical conditions. Six conditions, including Breast Cancer, Colorectal Cancer, Actinic Keratosis and Cutaneous Squamous Cell Carcinoma of the Skin, Malignant Melanoma, and Adult Soft Tissue Sarcomas, have been selected for evaluation purposes based on corresponding evidence-based clinical practice guidelines. Publications are identified using Medical Subject Headings (MeSH) terms, and a Named Entity Recognition and Linking (NER+NEL) pipeline extracts and ranks biomedical concepts from abstracts, weighted against occurrences in the entire PubMed database. The concept rankings are assessed by aligning them with the reference clinical practice guideline concepts. The study also examines the distribution and relationships of concepts across different semantic types within the KGs. The effectiveness of BioKGrapher is demonstrated through two downstream tasks: document multi-label classification and drug repurposing. A cancer-specific KG, derived from 3.6 million neoplasm-related publications, is used to pre-train BioBERT [26] , SciBERT [27] , and PubMedBERT [28] models by infusing KG triples into the encoder-decoder architectures through a Mixture-of-Partitions [29] approach with Adapter [30] modules, followed by fine-tuning on a multi-label classification task. In the drug repurposing case study, BioKGrapher analyzes publications on two example drugs, re-ranking associated neoplastic processes. Top results are validated against existing literature, highlighting BioKGrapher's potential in a practical application setting. By linking to the Unified Medical Language System (UMLS) [31] , several target mappings to knowledge bases such as Systematized Nomenclature of Medicine and Clinical Terms (SNOMED CT) [32] , National Cancer Institute Thesaurus (NCIt) or MeSH are available for BioKGrapher, whereby individual hierarchies and relations are generated in each case. The BioKGrapher application code is available online and requires a UMLS license. 3 The remainder of this work is structured as follows. Section 2 reviews existing methodologies for automatic knowledge graph construction in the biomedical domain, highlighting recent advancements and challenges. Section 3 details the datasets used, including the PubMed corpus and clinical practice guidelines. Section 4 describes the BioKGrapher application pipeline, covering NER+NEL, concept weighting, re-ranking, and the construction of hierarchical knowledge graphs with relation triples. Section 5 presents the outcomes of the approach, including semantic type distribution analysis, concept re-ranking, clinical practice guideline alignment, and performance in multi-label document classification and drug repurposing downstream tasks. Finally, Sections 7 and 8 discuss the findings, state limitations, and suggest future directions.

The authors have no relevant interests to disclose.