A methodology for building a medical ontology a with a limited domain experts’ involvement

A methodology for building a medical ontology a with a limited domain experts’ involvement | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article A methodology for building a medical ontology a with a limited domain experts’ involvement Sabrina Azzi This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5305559/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Ontology development is a multidisciplinary work involving domain experts and knowledge engineers. Bringing together such a team to develop an ontology of quality is not easy. Therefore, ontologies are often created with limited expertise either in the medical domain or in ontology engineering. Unfortunately, the existing methodologies do not provide much guidance on how the different steps of ontology development should be performed, particularly in the case of reduced involvement of domain experts. This challenge is getting more difficult when there is a multitude of medical knowledge sources and ontologies covering parts of the domain, and often each having a different representation of the same concept. This research presents a methodology for creating a medical ontology of quality with limited involvement of the domain experts. The latter are only consulted in the domain definition and evaluation phases. We combine building an ontology from codified knowledge and ontology reuse to enhance reusability and interoperability. The methodology is inspired by METHONTOLOGY for which we make several improvements, especially in the ontology reuse phase. We provide a proof of concept of the proposed methodology with a case study involving the development of the pneumonia diagnosis ontology (PNADO). Medical ontology engineering Ontology reuse Clinical practice guidelines OBO Foundry Competency questions Pneumonia diagnosis. Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 1. Introduction The contribution of ontologies to understanding, sharing, and integration of knowledge is well established. The usage of ontologies in medicine gains ground thanks to their ability to represent the medical domain knowledge and their expressive power formally [ 1 ]. An ontology can assure the interoperability and improve the functionality of a clinical decision support system by providing a standard vocabulary to help to integrate heterogeneous biomedical data sources [ 2 ]. The ontology building process is composed of several steps. Each step requires a set of comprehensive instructions and techniques of how it should be carried out, and together they form an ontology building methodology. The completeness and the degree of details of the latter are very important to build an ontology of quality. Medical ontologies have significantly grown over the last decade, but they have quality and content problems [ 2 , 3 ]. This is due, among others, to the following causes: 1) medical ontologies are created with limited expertise either in the medical domain or in ontology engineering, for example, Coronavirus Infectious Disease Ontology 1 is created by a medical expert whereas COVID-19 Ontology 2 is created by a knowledge engineer who confuses signs with symptoms; 2) they are often built from scratch and do not take advantage of reusing existing ontologies that have already been validated to save time and improve the interoperability; 3) the applied ontology building methodology lacks some important steps like evaluation or validation, or during the building process some steps are skipped. Many methodologies have been proposed to build an ontology from a corpus of text and/or by reusing elements of existing ontologies, which is called ontology reuse. The latter is the process of selecting existing ontologies and manipulating them in some way to meet the requirements and needs defined in the conception [ 4 ]. The main shortcoming of methodologies of building from text is that they do not guide how the different steps should be conducted. This makes them difficult to use, time-consuming, possibly error-prone, and costly [ 5 ], especially in situations where the involvement of domain experts is limited. Also, even they recognize ontology reuse as part of the overall ontology construction process, they address it briefly. On the other hand, ontology reuse reduces the ontology development time and required effort since the knowledge engineer can focus on new content specific to the ontology being created rather than developing content that has already been created and validated [ 6 ]. Besides, the reuse of ontologies increases interoperability [ 7 ], reduces redundancy [ 8 ], and prevents the random proliferation of ontologies. However, ontology reuse has the drawback of reproducing the errors of the reused ontologies. During the reuse process, inconsistencies and conflicts between concepts differently represented in different ontologies have to be fixed. Even though ontology reuse is recommended [ 9 ], to the best of our knowledge, there are currently no methodologies that provide adequate support for the ontology reuse process and resolve the problems identified above. Another important problem related to the reuse is the lack of documentation explaining the conceptual model of the ontology and/or the design criteria that were followed to build and evaluate the ontology [ 10 ]. This problem concerns most of the published medical ontologies. In this paper, we propose a methodology designed for a knowledge engineer for building a medical ontology that deals with the interoperability and conflicts between concept representations, with minimal involvement of medical experts. Depending on the ontology domain, the methodology relies on the ability to access already somehow codified knowledge. Examples of such knowledge for a medical domain include clinical practice guidelines, systematic reviews, and gold standard frameworks. Our methodology guides the knowledge engineer step by step by providing the necessary instructions. It consists of seven phases: 1) definition of ontology domain and scope; 2) building of knowledge corpus and extraction of terms; 3) building of preliminary ontology by applying ARCHONTE methodology on the terms previously identified; 4) reuse of ontologies and resolution of conflicts using competency questions; 5) evaluation of the ontology; 6) documentation and; 7) maintenance and evolution. The whole process is executed with limited involvement of the medical experts. We illustrate our ideas with a case study of building pneumonia diagnosis ontology (PNADO). Pneumonia is one of the diseases associated with many diagnostic errors [ 11 ]. To the best of our knowledge, PNADO is the first comprehensive ontology of pneumonia diagnosis. Several of the existing medical ontologies, such as Human Disease Ontology (DOID 3 ), SNOMED-CT 4 and Human Phenotype Ontology (HPO 5 ), contain concepts related to the diagnosis of pneumonia, however, they provide different representations for some concepts and they do not capture the entire domain knowledge. 2. State of the art Most of the first ontology engineering methodologies were developed as a by-product of ontology development for some specific projects. ENTERPRISE [ 12 , 13 ] and TOVE [ 14 ] (Toronto Virtual Enterprise) are examples of projects that relate to the domain of enterprise modeling, developed in 1995. TOVE concentrates on modeling the internal workflow of a company. The methodology principle is that the authors address ontology completeness based on competency questions. ENTERPRISE methodology also called Uschold & King’s methodology is based on the experience of developing the Enterprise Ontology. The four phases that are recommended in the methodology are: 1) identify the purpose, scope, and scenarios for its use; 2) build the ontology including ontology capture, coding, and integrating existing ontologies; 3) document the ontology; 4) evaluate the ontology. A refined version of this methodology was proposed in 1996 by Uschold and Gruninger [ 15 ]. In the same year, a well-structured methodology called METHONTOLOGY [ 16 ] was developed in the Laboratory of Artificial Intelligence at the Polytechnic University of Madrid. The phases proposed in METHONTOLOGY are: 1) specification that defines the purpose and scope of the ontology; 2) knowledge acquisition; 3) conceptualization that consists in structuring the domain knowledge in a conceptual model that describes the problem and its solution according to the domain vocabulary identified in the specification phase; 4) implementation; 5) integration; 6) evaluation; 7) documentation. The main shortcoming of these methodologies is that they consider texts as the only sources of knowledge, and they do not provide guidance on how the different steps should be conducted. An analysis of these methodologies for building ontologies was conducted in [ 10 ]. The study demonstrated that none of them are fully mature comparing them with IEEE standards [ 17 ]. However, METHONTOLOGY is the most mature but some steps require details. 101 method provides guidelines to manually develop domain ontologies [ 18 ]. It provides strategies to assist in making design decisions during ontology construction. It declares the possibility of reusing ontologies without specifying how. Also, the methodology focuses on defining class hierarchies, properties of classes and instances, and omits an important step which is the evaluation. Since developing an ontology from text is difficult and time-consuming, frameworks and tools were also proposed to support the user in the process of building. They use machine learning, linguistic, logic, and statistics techniques [ 19 ]. ASIUM [ 20 ], TextToOnto [ 21 ], Text2Onto [ 22 ], HASTI[ 23 ], SYnthesis of DIstributed Knowledge Acquired from TExts (SYNDIKATE)[ 24 ], ProMine[ 25 ], OntoLearn [ 26 ], OntoLT [ 27 ], OntoGen [ 28 ], Terminae [ 29 ] are examples of these tools. However, all these tools suffer from several shortcomings such as the difficulty of extracting relations and axioms. Authors of [ 19 ] recommend using deep learning rather than shallow learning for automatic ontology construction. While ontology reuse is supposed to facilitate and speed up the development process, it remains a very important issue especially in the biomedical domain where large resources are available. Several methodologies have been proposed to deal with this issue: MetROn was suggested to build a reusable ontology by reusing the existing ontologies and other resources such as classifications, vocabularies, terminologies, and standards [ 30 ]. Top-down, bottom-up, middle-out approaches, and even a combination of them are proposed to be used for defining classes; A methodology for biomedical ontology reuse was proposed in [ 8 ] and applied for Abdominal Ultrasound Ontology development. It provides steps and tools for finding and choosing relevant ontologies to reuse as well as concepts mapping; A methodology was proposed in [ 31 ] to build a multilingual domain ontology by reusing termino-ontological resources and text corpora. The methodology was applied to building Alzheimer’s disease ontology; Authors in [ 32 ] proposed seven steps to develop an ontology for guiding appropriate antibiotic prescribing intended for a clinical decision support system. These steps are: 1) define the domain ontology and scope; 2) review existing ontologies for reuse; 3) create classes and properties; 4) create a conceptual model; 5) select and implement upper ontology; 6) implement the ontology in a formal representation; 7) evaluate the ontology. The methodology evaluates the resulted ontology using Semantic Web Rule Language (SWRL) rules. All these methodologies lack guidance in carrying out the steps and not raise the problem of conflicts between concepts representations and how to resolve them. 3. Material and Methods In this section, we present a methodology inspired by METHONTOLOGY [ 16 ] to which we made several improvements. Figure 1 presents the seven phases of the methodology and shows that domain experts are only involved in the definition of the ontology domain and scope phase and the evaluation phase. 3.1 Phase 1: Ontology domain and scope definition This phase consists of establishing the domain of interest of the ontology to model and its scope. This can be done either by consulting domain experts (physicians) or by finding a gap in the literature. First, two fundamental questions must be treated: 1. What is the domain that the ontology will cover? 2. For what purpose are we going to create this ontology? Second, functional requirements for the ontology or questions that an ontology should be able to answer are defined. To support this, competency questions (CQs) are established by physicians and/or extracted from codified knowledge sources (section 3.2 ). The CQs have the form of informal queries that a particular community of users think that the ontology should answer [ 14 ]. They are used in many methodologies for ontology building to specify requirements [ 15 , 16 , 18 ]. The CQs are a sketch and do not have to be exhaustive. At the end of this phase, the knowledge engineer has a clear vision of what the ontology will cover, for what the ontology will be used, and what information it will contain. 3.2 Phase 2: Corpus building and term extraction In this phase, the knowledge engineer builds the corpus of knowledge extracts and identifies relevant terms to the ontology domain defined in phase 1. The use of codified knowledge makes this task easier for the knowledge engineer since this type of source is more reliable and comprehensible. Since the quality of ontology depends on the quality of the knowledge corpus, we recommend that for a medical domain, the knowledge corpus be composed of the most reliable and complete sources such as clinical practice guidelines (CPGs) or systematic reviews related to the medical domain for which ontology is being built. Another reason to use codified knowledge is that there are no available annotated texts that could be used as a base to create a medical ontology. Systematic reviews investigate and summarize the scientific literature in a systematic and methodologically rigorous way. Each of them focuses on a specific question and uses methods to study and summarize the findings of similar but separate studies. They are considered to be the most important type of scientific review as they are fundamental to evidence-based practice [ 33 ]. They are the basis of the CPGs. The latter are medical documents resuming the state of the art in the management of pathologies and clinical cases, which we call basic recommendations. They are consulted by healthcare practitioners to guide diagnostic and management decisions. They are based on the concept of Evidence-Based Medicine (EBM). In [ 34 ], EBM was defined as « the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients.». The CPGs are found in libraries such as Cochrane 6 , National Institute for Health and Care Excellence (NICE 7 ) and other libraries that are more specific to a specific condition. In fact, given the high degree of completeness of the CPGs regarding the most important concepts, they can be integrated into clinical decision support systems, thus providing recommendations that can be adapted to the patient's profile at the time of decision making [ 35 ]. We distinguish between a term and a concept. According to ISO 1087:2019, a term is a designation that represents a general concept by linguistic means whereas a concept is a unit of knowledge created by a unique combination of characteristics. We represent concepts corresponding to the extracted terms in the ontology in phase 3 (section 3.3 ). Term extraction can be manual or automatic. Some of the tools presented in section 2 could be used for automatic term extraction from a corpus of knowledge constituted of CPGs. Unfortunately, not all of them are available to use anymore. There are also powerful tools for clinical text analysis such as cTAKES 8 . They could be used for term extraction; however, their installation and usage are quite heavy. We suggest using Text2Onto [ 22 ] which is more suitable for this task. It is available, user friendly and its installation and usage are simple. Text2Onto has been developed to support the acquisition of ontologies from textual documents. It is composed of modules that extract terms, relationships (equivalence relationship, hierarchical, etc.), and instances. It offers extraction algorithms calculating the following measures: Relative Term Frequency (RTF), Term Frequency Inverted Document Frequency (TFIDF), Entropy, and the C-value/NC-value. We recommend using TFIDF when there is more than one document because it evaluates how relevant a term is to a document in a collection of documents. The extracted terms that should be analyzed are nouns, proper nouns, verbs, adjectives, adverbs, and phrases. These terms represent the concepts that will be treated in the next phase. It should be noted that the extraction of relations and axioms is not obvious and most of the existing learning systems do not infer new relations or axioms [ 19 ]. According to [ 19 ], deep learning techniques could be very useful for learning and inferring relations and axioms. 3.3 Phase 3: Building of preliminary ontology This phase is dedicated to the building of a preliminary medical ontology. This preliminary ontology represents the intended models of the requirements defined in phase 1. Usually, an ontology can have many interpretations. To limit the number of possible models, it uses axioms. Nevertheless, it is usually impossible to create an ideal ontology whose models coincide with the intended ones. An intended model represents all the requirements and only them [ 36 ]. We propose to follow the ARCHONTE methodology to build a preliminary ontology [ 37 ]. The strong point of this methodology is the recommendation of semantic commitment in the design process. In other words, to explain the meaning of each of the ontology concepts represented by the terms already selected in phase 2, the knowledge engineer expresses, in natural language, the similarities and differences that the concept has with those close to it. The ontology structure is similar to a tree, which facilitates the determination of the meaning that a concept has according to its position. ARCHONTE is composed of three steps: 1) semantic normalization; 2) knowledge formalization; 3) toward a computational ontology. They are elaborated below. 3.3.1 Step 1: Semantic normalization The objective of this step is to reach a semantic agreement about the meaning of the concept labels. The knowledge engineer applies differential principles on the set of candidate concepts chosen previously. According to the differential paradigm proposed by Bachimont [ 37 ], the meaning of a concept (or a node, since an ontology is structured as a tree) is determined by its closest neighbors: parent and its siblings, i.e ., its position in the ontological hierarchy based on terminological structure. Similarities and differences of each concept concerning its parent and its siblings are expressed in natural language. Four differential principles are distinguished: 1) similarity with each parent; 2) similarity with each sibling; 3) difference with each sibling; 4) difference with each parent. The result of this step is a so-called differential ontology. A differential ontology is an ontology where each concept and relationship must be defined according to its similarities and differences with its parents and siblings. 3.3.2 Step 2: Knowledge formalization In this step, the referential ontology is considered from an extensional semantics point of view. Each concept is linked to a set of objects that allows defining new concepts and relations through set operations. During this step, the ontology engineer has to precise the arity and domains of the relations, in correspondence with the intended models. He can also add some logical axioms to constrain the domains of the relations. The result of this step is a so-called referential ontology. A referential ontology is a formal ontology that covers the concepts, relations, instances, and axioms for a domain. 3.3.3 Step 3: Towards a computational ontology This last step of ARCHONTE methodology is implementing the referential ontology in an operational language of knowledge representation such as OWL. This is achieved by adopting, on the one hand, a formalism of representation (conceptual graphs, description logic), and secondly, by adapting the representation of the ontology to the objectives fixed in phase 2. This step marks the transition to a computational ontology. For interoperability and reusability purposes, we recommend following OBO Foundry principles. An upper medical ontology that provides the most general concepts in the medical domain can be reused to enhance interoperability with other medical ontologies. BFO (Basic Formal Ontology) is widely used as an upper ontology in OBO Foundry and the biomedical domain [ 38 ]. This makes BFO the best choice for building any medical ontology to enhance interoperability. It aims to model the basic structures of reality and provides classes that categorize, at a high level, real-world entities. BFO is designed to support information retrieval, analysis, and integration in several domains. It has two basic types of entities: (1) continuants that are entities that continue or persist through time, such as objects, qualities, and functions, and (2) occurrents that are events or happenings in which continuants participate [ 39 ]. Also, there is usually an upper ontology for a specific domain that includes general classes related to this domain. OGMS (Ontology for General Medical Science) is one of these ontologies [ 40 ]. OGMS enriches BFO by including the concepts of general medical science. This consists of approximately 100 terms that describe the fundamental aspects of medicine such as disorder , diagnosis , disease , disease course , laboratory test , sign , symptom , and syndrome . At the development level, Protégé ontology editor, which offers the possibility to visualize the different aspects of the ontology, can be used. For further enrichment of the ontology, UMLS 9 (Unified Medical Language System) can be used to extract definitions, synonyms, acronyms, and other annotations. Concepts in the ontology can be mapped to the referenced resource, UMLS concept unique identifiers (CUIs), where possible. Mapping the ontology to UMLS enables the ontology to be linked to many other relevant biomedical resources such as LOINC and SNOMED-CT. MetaMap [ 40 ] is a tool designed to identify biomedical concepts from free-text and maps them into concepts from the UMLS. Every concept has, when possible, CUI code, preferred label, definitions, synonyms, and external references of the reused ontologies. This step overlaps with the ontology reuse phase. 3.4 Phase 4: Ontology reuse and enrichment In this phase, we finalize the preliminary ontology by adding and detailing concepts. For this purpose, we reuse some terminological resources that we consider relevant to the ontology domain. The reuse can be hard, i.e. , one imports the complete ontology to reuse, or soft, i.e. , one imports the reused ontology concepts. The ontologies to be reused have to be evaluated for the suitability of their content coverage and the depth of knowledge, as well as the potential to support the inference process. The phase is composed of three steps: 1) finding ontologies; 2) choosing relevant ontologies; 3) resolving conflicts. 3.4.1 Step 1: Finding ontologies In this step, we select an ontology for reuse when we conjecture that its axioms could be used to respond to some of the requirements defined in phase 1. We use four main search criteria to identify candidate ontologies for reuse: i) both the natural and formal languages of the candidate ontology are the same as those of the ontology we are building; ii) there is a mapping from a non-empty subset of the new ontology requirements (defined in phase 1) to a robust set of concepts of the ontology candidate; iii) the candidate ontology is accepted within its user community. This acceptance is assessed by using some metrics such as the number of community members that endorse the ontology, the number of times the ontology has been reused, etc.; iv) the candidate ontology has been published in a peer-reviewed publication. Ontology repositories, such as NCBO BioPortal 10 , OBO Foundry 11 , Protégé Wiki 12 , Swoogle 13 (Ontology Lookup Service from the European Bioinformatics Institute), UMLS that integrates many terminologies and coding standards, are used to find relevant ontologies. The tool of BioPortal, Ontology Recommender 14 , which uses among others, coverage and acceptance metrics, could be used to find ontologies in this repository. 3.4.2 Step 2: Choosing relevant ontologies This step involves selecting the most relevant ontologies to be reused. For each significant topic found in the requirements defined in phase 1, we are looking for ontologies that deal with this topic, so-called relevant ontologies. Examples of such topics might be symptoms or laboratory tests. The preference criteria for choosing relevant ontologies are accuracy and precision orderings [ 41 ]. To compare the accuracy and precision of ontology candidates, two kinds of models are considered: Superfluous models (SUP) that correspond to a situation where some aspect of the candidate is weaker than required, i.e ., they prevent some aspects of the requirements from being satisfied (see Fig. 2 ) [ 41 ]; for example, Radiology Lexicon (RadLex) covers several unnecessary concepts for pneumonia diagnosis. Omitted models (OM) that correspond to a situation where some aspect of the candidate is stronger than is required, i.e ., this candidate ontology exceeds what was specified in the requirements (see Fig. 2 ); for example, CIDO 15 (Coronavirus Infectious Disease) is weaker than PNADO intended model, because it does not satisfy some ontology requirements defined in phase 1 ( i.e ., some concepts defined in PNADO are not covered by CIDO). If there are two candidate ontologies T 1 and T 2 that deal with the same topic, we choose them as follows: Case 1 If the set of superfluous models of T 1 is included in the set of superfluous models of T 2 and the set of omitted models of T 1 is included in the set of omitted models of T 2, then T 1 is more accurate than the candidate ontology T 2 . We recommend choosing T 1 . If the set of superfluous models of T 1 is empty, hard reuse should be considered. Case 2 If T 1 and T 2 have no superfluous models then both are precise and we reuse both of them. If the intersection of the sets of intended models is empty, then hard reuse should be considered. If not, hard reuse can be considered only if there are no conflicts between the representations of shared concepts. Case 3 If T 1 has superfluous models and T 2 has not, then regardless of omitted models, T 2 is more precise , we reuse T 2 and hard reuse should be considered. Case 4 If both T 1 and T 2 have superfluous models then they are incomparable using the accuracy and precision orderings, and reuse is problematic. If so, soft reuse should be considered. If there are more than two candidate ontologies, we apply the same algorithm by comparing two ontologies at a time. 3.4.3 Step 3: Resolving conflicts Once the ontologies for reuse have been selected, we proceed to manually identify the concepts that will enrich our original ontology. We are interested in adding new concepts and completing the concepts already represented in our ontology. A concept may be differently represented in different ontologies. In this case, there is a conflict and we have to decide which representation should be used. To resolve the conflict, we use conflict resolution questions (CRQs) defined by the knowledge engineer. These CRQs are related to the meaning and the structure of the ontology concept tree, i.e ., the hierarchy of classes, and other description logic (DL) concepts like intersection and union of classes, equivalent classes, universal classes, universal and existential quantification, has-value restriction, and cardinality restriction. We apply the principle that the concept axiomatization model is semantically correct if it does include an ontology intended model. Here are a few examples of CRQs: Does the concept have a definition (in the form of annotation)? If yes, does its definition correspond to an intended model of the ontology? Does the concept have a superclass? If yes, is it relevant to the ontology? Does the concept have children? If yes, is each of them relevant to the ontology? Does the concept have synonyms? Does the concept have alternative names? Does the concept have related names? The knowledge engineer should propose and use any CRQ that he considers suitable to decide on the most appropriate concept representation. To complete an already existing concept with annotations or add a new concept, we proceed as follows (see Fig. 3 ): If the concerned concept is not found in any ontology, then we propose some relevant annotations that can be obtained from UMLS, such as definitions, synonyms, or concept unique identifier (CUI). If the concept is not found in UMLS, no annotation is added. If the concept is represented in only one ontology, then we reuse it if relevant, i.e ., its representation fits the requirements, with its identifier and annotations. If not, we propose a new identifier. If the concept is found in UMLS, we propose to enrich it with annotations. If the concept is found in more than one ontology, then there is a conflict and we have to choose the ontology that provides the best concept representation for the domain, i.e. , the one that provides answers to the CRQs and corresponds to an intended model. If the chosen ontology is SNOMED-CT where concepts can have multiple parents and multi-hierarchies [ 42 ] that often cause user uncertainty in the selection of concepts and a messy situation in their classification [ 43 ], then we check if this is the case for the concerned concept. If it has more than one hierarchy, then we identify the most relevant hierarchy or combine one or more hierarchies to build a new one. If not, we reuse the concept with its hierarchy. If the chosen ontology is not SNOMED-CT, reuse the concept as it is represented in the ontology. 3.5 Phase 5: Ontology evaluation Ontology evaluation is a process of assessing the quality of an ontology involving a set of evaluation criteria and assessing its adequacy for being used in a specific context for a specific goal. It is also referred to as “quality assurance”, or “auditing” when it is conducted by a third party. Brank et al. [ 44 ] grouped existing evaluation approaches into four broad categories: 1) approaches that use the target ontology in an application and evaluate the application results [ 45 ]; 2) approaches that compare the target ontology to a gold standard [ 46 ]; 3) those that use data sources about a specific domain [ 47 ]; 4) approaches that recommend a manual assessment by domain experts according to a set of criteria [ 48 ]. The evaluation process consists of several tasks dealing with the evaluation of different aspects of the ontology. The tasks can be divided into verification methods that examine the structure of the ontology (they answer if the ontology was built the right way), and validation methods that examine its applicability in the real-world (they answer if the right ontology was built) [ 49 , 50 ]. Let’s observe that Brank’s evaluation approaches correspond to validation methods. More specific evaluation criteria have been proposed by [ 50 , 51 ], see Table 1 . Most of them correspond to verification methods. Table 1 Ontology evaluation criteria. Criteria Definition Accuracy Does the asserted knowledge in the ontology agree with the expert’s knowledge, which is often measured in terms of precision and recall? Completeness Is the domain of interest appropriately covered? Conciseness Does the ontology include irrelevant or redundant axioms? Consistency Does the ontology include or allow for contradictions? Computational efficiency How fast can the reasoners work with the ontology? Adaptability How easy or difficult is it to use the ontology in different contexts? Clarity Does the ontology communicate effectively the intended meaning of the defined terms? Several authors have suggested that automated tools are needed to ensure that high-quality ontologies are developed [ 52 , 53 ]. Authors in [ 1 ] emphasized the lack of tools for ontology evaluation. Most of the tools discussed in the literature are prototypes or proposals. To our knowledge, OOPS! 16 , OAF 17 and OntoMetrics 18 are the only available evaluation tools that can be used currently. To evaluate each criterion listed in Table 1 , we suggest using additional data sources. The evaluation of some criteria can be in part automatized, and for some others, it requires the involvement of the knowledge engineer and domain experts. The following sections go through those criteria. 3.5.1 Consistency Internal consistency relates to the adherence of the ontological model to the rules of the description logic. It can be checked using reasoners in PROTÉGÉ and OOPS! 3.5.2 Accuracy and coverage (completeness) The evaluation of accuracy and completeness usually relies on access to a gold standard built by domain experts. The building task is difficult to achieve for several reasons: difficulties to keep up with the published work in a given domain, variety of annotation standards, diversity of biomedical sources, high cost of annotation by medical experts [ 1 ]. To calculate precision and recall, two different approaches can be used. Precision is evaluated by domain experts that assess how well the ontology meets the set of predefined requirements. Laddering interviewing technique [ 54 ] and competency questions can be used. WebProtégé 19 provides extensive collaboration features that allow sharing the ontology with domain experts. We recommend writing a document to explain the ontology, its objectives, and what is expected of the experts. This document should include questions using the laddering technique. To calculate the recall, we suggest building an evaluation corpus of knowledge composed of documents that substantially cover a given domain and include CPGs, systematic reviews, and patient health records. The goal of using CPGs is to verify that all the concepts identified in phase 3 are covered by ontology, including relationships. We manually annotate a number of relevant text fragments from the corpus. Then we link all concepts that concern the ontology domain to concepts in the ontology. This step allows us to find the ratio of the covered concepts. 3.5.3 Clarity Clarity concerns the meaning of the defined concepts and their independence of any social or computational context. It should be evaluated by domain experts. This step should be preceded by checking the readability of each concept, i.e ., the existence of human-readable descriptions such as labels, definitions, or synonyms. This can be done by using tools such as OntoMetrics and OOPS! 3.5.4 Conciseness Unnecessary concepts with regards to the domain to be covered can be reported by domain experts when evaluating. Redundancy is also evaluated at this stage. It occurs when an axiom can be inferred from already defined axioms. Two types of redundancies are defined in [ 49 ]: 1) concepts (classes, properties, or instances) having more than one subsumption relation between them; 2) identical formal definitions of concepts with possibly different labels. The two kinds of redundancy can be detected by reasoners and OOPS! [ 55 ]. 3.5.5 Computational efficiency The size of an ontology and the complexity of its axioms can slow down processing this ontology by tools. Computational efficiency measures how fast can the used tools, in particular reasoners, work with the ontology. Unfortunately, ontologies are often unnecessarily complex due to a lack of respect for modular structure. Waiting forever for a response and excessive space requirements make these ontologies unusable for the deployment process and feasible maintenance. Since the metrics of processing time and required space are difficult to evaluate, we suggest simple verifications using tools like Protégé, OOPS!, or OntoMetrics. 3.5.6 Adaptability Adaptability measures: 1) how well the ontology anticipates its uses in different contexts; 2) whether it constitutes a reliable foundation for other ontologies; 3) whether it is flexible enough to react predictably to small changes and to allow extensions without any need to remove axioms. Adaptability is strongly related to the ontology modular design. “An ontology-module is a re-employable segment of a bigger or more tedious ontology, which is self-sufficient but holds relationships with the other modules of the ontology that also encompasses the initial non-modularized ontology” [ 56 ]. A modular design offers numerous benefits, such as simple reuse of single components in other ontologies, feasible reasoning performance, facilitating ontology understanding, reduced complexity, making the central ontology less vulnerable to changes, and others. The quality of modular design can be assessed using semantic metrics such as coupling and cohesion. Coupling refers to the interdependencies that exist between ontology modules, i.e ., to the number of shared symbols between axioms in different modules, while cohesion refers to the degree to which the elements in a module belong together. During the reuse process, the ontology engineer looks for an ontology with the maximal cohesion and minimal coupling. There have been many attempts to define a strategy selecting “the best modular ontology” [ 57 ]. Unfortunately, currently, no tools are supporting the creating of modular ontologies. A possible alternative can be a maximal reduction of the ontology scope; a similar approach has been applied to the development of the CORE subset of SNOMED CT. Adaptability is also strongly related to the respect of the community standards. It includes usage of commonly accepted upper ontologies such as BFO, following OBO principles 20 , reusing relations from RO, specification of ontology version, and providing well-written documentation. 3.6 Phase 6: Ontology documentation An ontology must be documented for better understanding and using by future users on one hand, on the other hand, its maintenance and update by any knowledge engineer will be easier. The documentation process starts in the first phase when the domain and scope of the ontology are defined. At the end of each phase, a document detailing its progress and deliverables is established. The documentation of the ontology includes its classes, relations, properties, instances, axioms, and annotations. OWLDoc 21 and LODE 22 (Live OWL Documentation Environment) are tools that can be used to generate readable documentation presented to the user in the form of an HTML page with embedded links for easy navigation. 3.7 Phase 7: Maintenance and evolution Medical knowledge is evolving and this evolution has to be reflected during the maintenance process. It includes integrating the missing knowledge and maintaining ontology consistency and coherence. If there are changes that occur in the ontology domain and scope definition (phase 1), all the phases of the ontology building should be carried out. If the updates concern only a few concepts in the ontology that need to be added, then follow the steps of adding concepts and resolving conflicts approach as presented in Fig. 3 . If the updates concern reused concepts that were modified in the reused ontologies, then reflect the modification in the concept if relevant. If not, keep the old version. The changes must be consistent with the domain and scope of the ontology. Once the changes are applied to the ontology, the ontology must be evaluated for consistency. Reused ontologies are usually updated by their authors. To the best of our knowledge, there are no ontology managers tracking the updates, and checking for the new versions has to be done manually. 4. Results We present in this section the results of applying the proposed methodology on building pneumonia diagnosis ontology (PNADO). 4.1 Ontology domain and scope definition We defined the ontology domain and scope after meetings with physicians from the Gatineau Hospital, who were interested in reducing erroneous pneumonia diagnoses. Functional requirements are defined by a set of CQs established by physicians and coming from the CPGs covering pneumonia diagnosis. The following questions, among others, have to be answered by physicians to diagnose pneumonia correctly: What are the symptoms and clinical signs of pneumonia? What are the types of pneumonia? How is pneumonia diagnosed? What are the pathogens of pneumonia? What is the clinical history of the patient? What are the laboratory tests to diagnose pneumonia? What are the results of the physical examination of the patient? What is the result of lung imaging of the patient? What are the results of the lab tests of the patient? Once the diagnosis of pneumonia has been made, what complications should the physician look for? 4.2 Corpus building and term extraction The corpus of knowledge is constituted of 13 publicly available CPGs from national and international repositories that cover most of the pneumonia diagnosis knowledge. They are Cochrane 23 , NICE 24 , Infectious Diseases Society of America (IDSA 25 ), European Society of Clinical Microbiology and Infectious Diseases (ESCMID 26 ), Australian Society for Infectious Diseases (ASID 27 ), Canadian Respiratory Guidelines (CTS 28 ), Pulmonary & Critical Care Medicine (PulmCCM 29 ), American Thoracic Society (ATS 30 ), and British Thoracic Society (BTS 31 ). The language of all the CPGs is English. To build our ontology, we select the terms that are relevant to pneumonia diagnosis. These terms include symptoms, clinical signs, imaging, laboratory tests and their results, pathogens (agents and others), antecedents, types of pneumonia, differential diagnosis, and complications. Then, in the set of found terms, we examine adjectives, adverbs, nouns, and proper nouns, verbs, and phrases because we believe that they convey the knowledge necessary to identify concepts in the pneumonia diagnosis domain. 4.3 Building of preliminary ontology 4.3.1 Semantic normalization To develop a differential ontology, we have to articulate the candidate terms selected previously by specifying the differential principles that characterize them. For example, bacterial pneumonia, fungal pneumonia, infective pneumonia acquired prenatally, pneumonia due to parasitic infestation, and viral pneumonia are sibling concepts because they have the similarity to be an infective disease. This step is the most time-consuming. 4.3.2 Knowledge formalization The following example illustrates adding an axiom. In the following text, ontological concepts are written in italic. Sign (also called clinical sign ) is a concept in PNADO. It is defined as the ′′quality of a patient, a material entity that is part of a patient, or a processual entity that a patient participates in, any one of which is observed in a physical examination and is deemed by the clinician to be of clinical significance′′. Symptom is another concept in PNADO; it is defined as ′′quality of a patient that is observed by the patient or a processual entity experienced by the patient, either of which is hypothesized by the patient to be a realization of a disease′′. We add an axiom to indicate that the two concepts are disjoint. 4.3.3 Towards a computational ontology During that process, using Protégé, we operationalize the ontology in the OWL language because it meets our needs in terms of expressiveness and manageability. We use the upper medical ontology OGMS. The choice of OGMS will be explained in the next phase. For further enrichment of PNADO, we extract definitions, synonyms, acronyms, and other annotations from UMLS by using MetaMap. At the end of this step, we obtain the preliminary PNADO. 4.4 Ontology reuse and enrichment 4.4.1 Finding ontologies We focus on two open content repositories of biomedical ontologies: Open Biomedical Ontology (OBO) Foundry and BioPortal. The English language and the OWL format are required. Using the search criteria, we found 26 candidate ontologies that could be reused. 4.4.2 Choosing relevant ontologies We identify ontologies to be reused by using accuracy and precision orderings. No precise ontology has been found. Choosing between Basic Formal Ontology (BFO 32 ) and General Formal Ontology (GFO 33 ): both are designed to be upper ontologies. BFO has fewer superfluous and fewer omitted models than GFO, then it is more accurate. We reuse BFO as an upper ontology (hard reuse). Choosing between Ontology of General Medical Science (OGMS 34 ) and DIAGONT 35 : these ontologies cover the most important concepts for diagnostic. Both have superfluous and omitted models that are incomparable. Since OGMS is built upon BFO, we choose to reuse it (hard reuse). Hence, PNADO follows OGMS representation. Choosing between SYMP 36 and Clinical Signs and Symptoms Ontology (CSSO 37 ): SYMP has fewer superfluous and fewer omitted models than CSSO, it is more accurate. We reuse SYMP that covers mostly symptoms and clinical signs. Choosing between SNOMED-CT 38 and NCBITaxon 39 : SNOMED-CT is the most widely used healthcare terminology and has the most comprehensive coverage of concepts for representing clinical knowledge [ 58 , 59 ]. SNOMED-CT is used as an ontology rather than a terminology, it is based on DL, and it can be reasoned over [ 60 ]. NCBITaxon is a classification and nomenclature of all the organisms in the public sequence databases. Both have superfluous and omitted models that are not comparable. Both are interesting to make soft reuse. SNOMED-CT covers other clinical aspects in addition to organisms that also can be reused. Choosing between GAMUTS 40 and Radiological Lexicon (RadLex 41 ): both are comprehensive terminologies for radiology and both have superfluous and omitted models but Radlex has fewer. We decide to reuse Radlex (soft reuse). Choosing between Logical Observation Identifiers Names and Codes (LOINC 42 ) for medical laboratory observations identification and clinical LABoratory Ontology (LABO 43 ): both have superfluous and omitted models that are not comparable. We choose to reuse LOINC because it is the most used by clinicians. Choosing between the International Classification of Diseases (ICD10 44 ) and Human Disease Ontology (DOID 45 ) that is a representation of human diseases organized by etiology: both have superfluous and omitted models that are not comparable. DOID provides more concepts related to PNADO than ICD10. We reuse DOID (soft reuse). We also reuse the following four ontologies dealing with important topics that are not covered by other ontologies chosen in the previous step: Relation Ontology (RO 46 ) containing relations between entities; Computer-Based Patient Record Ontology (CPRO 47 ) for patient profile description; Human Phenotype Ontology (HPO 48 ) providing a structured and controlled vocabulary for the phenotypic features in human hereditary and other diseases; Infectious Disease Ontology (IDO 49 ) covers the infectious disease domain. The reused ontologies use different naming conventions. In SNOMED-CT, the name of each concept begins with a capital letter whereas in SYMP it begins with a lowercase letter. In PNADO, names begin with a capital letter. In this paper, each concept is written according to the convention used by the ontology in question. 4.4.3 Resolving conflicts Here we illustrate resolving concept conflicts with a few examples. The concept of hypoxemia is found in HPO, SYMP, and SNOMED-CT. To choose the best representation of this concept, we proceed as follows: CRQ 1: “Does hypoxemia have a definition in each ontology?” Answer: Only HPO provides a definition. CRQ 2: “Does its definition correspond to the PNADO requirements?” Answer: the definition given in HPO is relevant to PNADO. CRQ 3: “Does hypoxemia have children?” Answer: Hypoxemia has children in HPO and SNOMED-CT but none in SYMP. CRQ 4: “Is each found child relevant to the PNADO requirements?” Answer: Hypoxemia ’s children found in HPO are relevant to the PNADO requirements. Analyze: Hypoxemia presented in HPO seems to have the best representation for PNADO since it provides a relevant definition and relevant children that correspond to the PNADO requirements. The concept of patient is represented differently in CPRO and SNOMED-CT. We use the following CRQs to choose the most relevant representation for this concept: CRQ 1: “Does patient have a definition in each ontology?” Answer: No ontology provides a definition. CRQ 2: “What is the parent concept of patient ?” Answer: In CPRO, the parent concept is organism that is provided by OGMS. In SNOMED-CT, the parent is social context that does not fit the PNADO requirements. CRQ 3: “Is any person a patient?” Answer in CPRO is the axiom “ Human/Person and ( 'Plays Role' some Patient role ) and ( 'Participates_in' some Clinical act )”. No answer is found in SNOMED-CT. Analyze: CPRO provides a better representation for patient than SNOMED-CT. It specifies with an axiom in which case a person can be a patient and it is a subclass of organism defined by OGMS. Two other examples of resolving conflicts are presented in Table 2 . Table 2 Examples of inconsistencies resolved with CRQs. Concept Ontologies CRQs CRQs responses Hypotension HPO SNOMED-CT SYMP Is hypotension a symptom or a vital sign ? - HPO: hypotension is a phenotypic abnormality. - SNOMED-CT: hypotension ( low blood pressure ) is considered a disorder of the cardiovascular system . - SYMP: hypotension is considered a hemic system symptom . What is the synonym of hypotension ? - Hypotension has two synonyms in HPO, three synonyms in SNOMED-CT, and no synonym in SYMP. Are there any hypotension 's children? - Hypotension has three children in HPO, twelve children in SNOMED-CT, and one child in SYMP. Is each child relevant to the diagnosis of pneumonia? - Only hypotension 's children provided by HPO and SYMP are relevant for the diagnosis of pneumonia. Analyze : blood pressure is a measurement of the pressure in arteries. We consider it in PNADO as a vital sign . Hypotension 's children in HPO combined with SYMP's child correspond to PNADO intended model. We reuse hypotension 's synonyms of SNOMED-CT. Respiratory rates SNOMED-CT LOINC Is respiratory rate a symptom or a vital sign ? - In both SNOMED-CT and LOINC, it is mentioned that the scale type of respiratory rate is quantitative. Are there any respiratory rate 's children? - SNOMED-CT: the respiratory rate has two children. - LOINC: the respiratory rate has no children. Is each child relevant to the diagnosis of pneumonia? - Yes, the respiratory rate 's children provided by SNOMED-CT are relevant for the diagnosis of pneumonia. Is there any measurement unit? - No measurement for respiratory rate is given in SNOMED-CT. - LOINC provides "{breaths}/min" as an example of the respiratory rate measurement unit. Are there any synonyms? Are there any related names? - SNOMED-CT provides two synonyms and no related names. - LOINC doesn't provide any synonym for respiratory rate but provides 13 related names. Analyze : The respiratory rate is represented as a vital sign in PNADO. We reuse SNOMED-CT's children because they respond to the PNADO scope and its intended model. We reuse the unit measurement of LOINC and some of the related names. We also reuse the synonyms of SNOMED-CT. 4.5 Ontology evaluation For the evaluation of PNADO, we used tools, a clinical dataset that contains multiple cases of pneumonia, and a clinical corpus of knowledge captured in the CPGs and systematic reviews. Domain experts were also involved in the process. 4.5.1 Consistency We used the Pellet reasoner integrated with PROTÉGÉ and the tool OOPS! to check the internal consistency of PNADO. OOPS! reported minor pitfalls regarding annotations. The missing license for PNADO was reported as an important pitfall. No critical pitfalls were reported. 4.5.2 Accuracy and coverage (completeness) To calculate recall, we constructed an evaluation knowledge corpus constituted of 1) the 13 CPGs chosen in section 4.2 ; 2 ) 43 systematic reviews that treat pneumonia diagnosis from national and international repositories containing evidence-based medical documents; and 3) the clinical dataset MIMIC-III 50 (Multi-parameter Intelligent Monitoring in Intensive Care III) [ 61 ]. First, we treated the CPGs and systematic reviews. We manually extracted 710 different terms related to pneumonia diagnosis from 336 text fragments (each fragment had twenty lines on average) and tried to link them to concepts in PNADO. Linking extracted concepts to concepts in PNADO, we found a match for 570 concepts giving a value of recall of 80%. The remaining 140 concepts that could not be matched were added to PNADO. Here is an example (see Table 3 ) of annotation of a fragment of text from the clinical guideline “Management of Community-Acquired Pneumonia in Adults” [ 62 ]. A chest radiograph is required for the routine evaluation of patients who are likely to have pneumonia , to establish the diagnosis and to aid in differentiating CAP from other common causes of cough and fever , such as acute bronchitis . Table 3 Example of annotation of clinical guideline paragraph with PNADO. Concepts from the text Equivalences in the PNADO Class Object property ID Parent entity in PNADO chest radiograph Chest radiography PNADO:0000783 Imaging of lung patients likely to have pneumonia Has a diagnosis (patients, pneumonia) PNADO:0001291 diagnosis diagnosis OGMS:0000073 data item differentiating CAP from…acute bronchitis Differential diagnosis of(CAP, Acute bronchitis) PNADO:0001157 CAP Community-acquired pneumonia SNOMEDCT_US:385093006 Pneumonia cough Cough SNOMEDCT_US:49727002 Respiratory system and chest symptom fever fever SYMP:0000613 neurological and physiological symptom Second, we treated records from MIMIC III that consists of 38,597 distinct and de-identified adult patients and 49,785 hospital admissions. It includes, among others, laboratory data, radiology reports, vital signs, therapeutic intervention profiles, ventilator settings, nursing notes, diagnostic codes, discharge summaries, and provider order entry data. It comprises 26 tables linked by identifiers such as HADM_ID referring to unique hospital admission and SUBJECT_ID referring to a unique patient. Among those tables, we considered the ‘diagnoses_icd’ table that contains ICD-9 diagnoses for patients and the ‘note events’ table that contains all notes about patients from their admissions to their discharges, including nursing and physician notes, ECG reports, radiology reports, and discharge summaries. We found 7702 cases of pneumonia diagnosis in the ‘diagnoses_icd’ table. As shown in Fig. 4 , there were 32 types of pneumonia. We were interested in making sure that PNADO covered all the cases captured in MIMIC-III. The number of pneumonia cases with code 860 was more than half of all cases. Consequently, we used the logarithm function to reduce the number of cases to include in the evaluation for each type of pneumonia, 43 cases in total. The next task was the extraction of the records from the ′noteevents′ table by using HADM_ID already obtained in the previous task. We randomly selected cases according to the cardinality of each type and completed the annotation. We manually annotated terms related to pneumonia diagnosis and verified whether they were covered by PNADO or not. For those not covered, we analyzed if they were synonyms for the existing concepts or they were new concepts. At the end of the evaluation of PNADO with the MIMIC III database, 36 concepts of 988 concepts found in the electronic health records were not covered by PNADO. The analysis of each concept revealed that 16 concepts were synonyms and 20 concepts were subclasses of the existing concepts. Here are some examples of synonyms: pulmonary embolus, rhonchi sound, rhonchus sound, pneumococci, pneumonic infiltrates, MSSA PNA, MSSA pneumonia, MRSA PNA, MRSA pneumonia, and trouble breathing. The new subclasses include types of pneumonia such as polymicrobial pneumonia ; physical examination findings such as respiratory failure; symptoms such as recurrent cough and occasional cough ; and clinical histories such as tracheostomy . The recall ratio was 96%. To calculate the precision of PNADO, we involved a pneumologist from Charles-Le Moyne Hospital 51 , an emergency physician from the Gatineau Hospital 52 , and a family doctor from the University of Quebec in Outaouais medical clinic 53 . We prepared a document to explain the ontology and its objective. We also prepared a document containing the CQs used for the determination of the domain and scope of PNADO (section 4.1 Ontology domain and scope definition, and questions using the laddering technique [ 54 ]. Here are a few examples of these questions: What are the symptoms of pneumonia? Is the classification of symptoms according to the different systems (digestive system, cardiovascular system, etc.) correct? Is each symptom well classified? The participating physicians created accounts in WebProtégé, and we shared PNADO with them. Once they had familiarized themselves with the environment, they evaluated each concept and commented directly in WebProtégé. They verified the relevance of each concept in PNADO to pneumonia diagnosis, its position, and its relations with the other concepts. For each concept, they provided a mention of evaluation (relevance and suggestions) in WebProtégé. They suggested several changes including adding radiological signs of consolidation : air bronchograms , ill-defined , fluffy opacities , air alveologram , patchy opacities , acinar , preserved lung volume , extension to pleural surface , CT angiogram sign , silhouette sign ; moving concepts of fever to general symptom, purulent tracheobronchial secretions to respiratory system and chest symptom , headache to neurological and physical symptom , arterial blood gas to procedure ; and removing the concepts transitory tachypnea of the newborn, sputum eosinophilia and lithoptysis. The precision ratio was 96%. 4.5.3 Clarity We used OntoMetrics and OOPS! tools to evaluate readability. OntoMetrics was not able to check the readability of all the concepts. OOPS! detected that some of the new concepts added to PNADO did not have definitions. In fact, these concepts have no definitions in UMLS. 663 concepts without definition (mostly UMLS concepts) were reported. 4.5.4 Conciseness OOPS! and Pellet reasoner were used to detecting redundancies in PNADO. OOPS! reported two classes that contain the same labels. We found that the two classes have labels that only differ by one character ( legionella pneumophila serogroup 1 , legionella pneumophila serogroup 2 ). 4.5.5 Computational efficiency Time processing of PNADO using Pellet reasoner, OOPS! and OntoMetrics was short (1–2 sec) with one exception: OntoMetrics was not able to check the criterion of readability when providing the entire ontology. 4.5.6 Adaptability PNADO uses the BFO upper ontology that enhances reusability and interoperability. It also follows OBO Foundry principles and reuses relations from RO. We specify the PNADO version and provide documentation (see section 4.5.6 Adaptability). PNADO does not have a modular structure. 4.5.7 Characteristics of PNADO PNADO is published in BioPortal repository and can be viewed and explored at https://bioportal.bioontology.org/ontologies/PNADO . We present in this section the results obtained by applying the proposed methodology to the PNADO building. PNADO contains 1598 classes (1448 are reused and 150 are new classes), 42 object properties (27 are reused and 15 are new), 1591 logical axioms, and 83 annotation properties. Table 4 presents statistics on reused concepts and resolved conflicts. We noticed that SNOMED-CT was frequently involved in conflict resolution, as it overlaps with most existing medical ontologies. PNADO is built according to the principles of OBO Foundry, using OGMS that follows the BFO paradigm. OGMS provides a set of general reference classes related to diseases and diagnoses. PNADO is only focused on diagnosis. Therefore, only the OGMS concepts related to diagnosis are used. Other high-level OGMS terms may be used in future extensions. Table 4 Ontology reuse in PNADO. Ontology Usage in PNADO Classes Relations Total Conflicts Hard reuse 1 BFO Upper ontology 35 5 40 0 2 OGMS Upper domain ontology 149 0 149 0 Soft reuse 1 SYMP Symptoms 57 0 57 22 2 NCBITAXON Virus and bacteria 184 0 184 0 3 RO Relations 0 22 22 0 4 CPRO Roles 4 0 4 2 5 LOINC Laboratory 26 0 26 1 6 SNOMED-CT Diseases, symptoms, and clinical signs 796 0 796 115 7 RADLEX Imaging 55 0 55 5 8 DOID Disease 35 0 35 13 9 HPO Phenotype 96 0 96 73 10 IDO Pathogens 11 0 11 0 Figure 5 shows an example from PNADO demonstrating the hierarchy of the class infective pneumonia and, in particular, the class viral pneumonia. Figure 6 shows the representation of the healthcare process assay class. 4.6 Ontology documentation We annotated every concept in PNADO and used OWLDoc to document the ontology. Every phase of the building process is described in this work. 4.7 Maintenance and evolution Within the COVID pandemic context, two CPGs treating pneumonia diagnosis were found in NICE and CEBM 54 (the Centre for Evidence-Based Medicine) repositories. We extracted 16 new concepts relevant to PNADO. We also frequently check the parts of the reused ontologies. Recently, OGMS has been subject to changes that affect PNADO significantly. For example, symptom was moved to process class, vital sign was moved to material entity class, sign class was deleted, etc. 5. Discussion 5.1 Generality of the methodology The techniques proposed in each phase are independent of any specific domain and can therefore be used for any domain of any field. However, the corpus of knowledge must be tailored to the targeted ontology domain in order to apply the methodology. The same goes for the ontologies to reuse, they must be related to the same field. The use of codified knowledge is very useful for a knowledge engineer, he can easily understand the domain and there are fewer chances of getting confused. 5.2 Ontology reuse Ontology reuse in the methodology is used to enrich the preliminary ontology with new concepts and annotations, and to enhance ontology interoperability and reusability. Since the methodology is designed for a knowledge engineer and the involvement of the domain experts is limited, ontology reuse cannot be performed at the beginning of the building process because the knowledge engineer does not know which concepts are relevant to the ontology domain. Ontology reuse is the most tricking phase in the methodology due to the interoperability issues among the reused ontologies. Performing this phase during the PNADO building revealed some findings: Choosing relevant ontologies to reuse is very important and determinant for the rest of the reuse process. Some candidate ontologies can be easily excluded while others require careful analysis. Even if the chosen ontology is of good quality, the representation of a concept may not be relevant to the requirements of the ontology that is being built. We also found out during the PNADO building that some ontologies have bad representations for some concepts. For example, SYMP represents diseases as symptoms. An example of a relation found in RO that is not relevant for PNADO is discussed in section 5.3 . Each ontology is chosen according to a significant topic, as discussed in section 3.4.2 , but in the process of reuse, the chosen ontology may contain concepts that deal with other topics. For example, DOID was chosen for the topic of diseases but it was involved in conflicts regarding concepts related to symptoms and clinical signs. A concept can be differently presented in different ontologies and to choose the best representation to reuse requires reflection about the relevant CRQs. 5.3 Upper domain ontologies As mentioned in section 5.2 , there are ontologies designed to cover a specific domain which also cover concepts belonging to other domains. These ontologies do not respect the principles of modular design [ 9 ]. The unwanted concepts should be usually covered by upper domain ontologies. We notice that it is often useful to design several levels of upper domain ontologies to get a better module coupling and cohesion, and at the same time, to reduce the reuse and interoperability issues. OGMS, which covers diagnosis and treatment of disease, was used as an upper domain ontology for PNADO. Many concepts (classes and relations) required for the diagnostic process are missing in OGMS. PNADO aims to cover the concepts related to pneumonia diagnosis and the concepts related to the diagnostic process should be available in an upper domain ontology. A different example would be an upper domain ontology describing patient. Currently, several domain ontologies differently model the concept of patient. Having one upper ontology representing patient would be very useful for reuse and enhancing interoperability. 5.4 PNADO new concepts In the following paragraphs, we discuss some of the reused object properties and classes that required major modifications. Complication of A complication in medicine is a medical problem that occurs during or after a disease, procedure, treatment, or function (pregnancy, for example). This concept is represented as a class in many ontologies such as SNOMED-CT or Radlex. In SNOMED-CT, the Complication is a subclass of Disease and represents complications of disease such as Complications due to diabetes mellitus , or Complication due to Crohn's disease. Diabetes mellitus and Crohn’s disease are subclasses of disease. The diseases concerned by complications are also represented as subclasses of the class disease . Such a representation creates a heritage chain disease → complication → disease and does not cover all cases of complications due to disease , procedure , treatment , or function . We propose to represent complication as an object property named complication of with four sub-properties: 1) complication of disease that associates disease to disease ; 2) complication of procedure that associates procedure to disease ; 3) complication of treatment that associates treatment to disease, and 4) complication of function that associates function to disease. Axioms are progressively added when needed to represent complications. For example, in PNADO, axioms are added to illustrate pneumonia complications. This way of representing complication of enhances the interoperability and reusability. Differential diagnosis of Differential diagnosis is the distinguishing of disease from others that present similar clinical features (signs and symptoms). It is represented in SNOMED-CT and LOINC as a class. We notice that there is no representation of pneumonia differential diagnosis in both ontologies. The presence of differential diagnosis could help to avoid diagnosis errors. We propose a new object property Differential diagnosis of that holds between a diagnosis “A” and a diagnosis “B” if they have some similar clinical features. Recall that a diagnosis is the representation of a conclusion of a diagnostic process that can be a disease or a syndrome. Axioms are added to define the relation. An example of such an axiom: pneumonia ′ Differential diagnosis of′ some bronchitis . Has symptom Has symptom is an object property of RO where it is defined as "a relation that holds between a disease or an organism and a phenotype". We notice that it is a sub-property of the object property that has phenotype. The latter associates the domains generically dependent continuant or material anatomical entity or disease to the range phenotype. The class phenotype , which is reused by RO from Combined Phenotype Ontology (UPHENO), is defined "as a defect or loss of some anatomical structure or a biological process to wild-type". The class phenotype in OGMS and reused in PNADO is defined as "a (combination of) quality(ies) of an organism determined by the interaction of its genetic make-up and environment that differentiates specific instances of a species from other instances of the same species". It is obvious that the two definitions are different, the one provided by UPHENO does not correspond to PNADO intended model, and subsequently, we do not reuse an object property associating this class to any other class. We create a new object property Has a symptom that associates Patient to symptom . symptom symptom is defined in OGMS as "a quality of a patient that is observed by the patient or a processual entity experienced by the patient, either of which is hypothesized by the patient to be a realization of a disease". It is also defined in SYMP as "a perceived change in function, sensation, loss, disturbance or appearance reported by a patient indicative of a disease". Nevertheless, during the reuse of SYMP, we noticed that it also contains diseases and clinical signs when it was intended to contain only symptoms. For instance, hypotension , arrhythmia , bronchitis , endocarditis , and shock are defined in SYMP as symptoms, whereas they are diseases. Indeed, there is often confusion between clinical signs, diseases, and symptoms in SYMP and other ontologies. In PNADO, the class symptom , thanks to the ontological commitment, includes only what is reported by a patient. Pathogen Before discussing Pathogen , we will briefly review the Organism since both concepts are connected. Organism is an interesting concept needed in PNADO, and it can be found in other ontologies such as CPRO, IDO, or SNOMED-CT. In SNOMED-CT, it is represented as an independent entity that includes bacteria , virus , archaea , eukarya , and prion . Since no definition is given to this concept, its meaning (not precise) can only be deduced from its sub-classes. CPRO represents organism as a subclass of object whereas IDO represents it as a sibling to object . An object according to the definition given by BFO "is a material entity that is: 1) spatially extended in three dimensions; 2) causally unified, meaning its parts are tied together by relations of connection in such a way that if one part of the object is moved in space, then its other parts will likely be moved also; and 3) maximally self-connected". The definitions given by the three ontologies are almost similar and they mean that an organism is an individual living system that may be unicellular or made up of many billions of cells like humans. This leads us to conclude that an Organism is definitely an object , contrary to the IDO representation. Pathogen in CPRO is a subclass of organism and it is defined as "any virus, microorganism, or other substance causing disease". This definition is limited because, in the broadest sense, pathogen is anything that can produce disease. SNOMED-CT provides pathogenic organism rather than organism , as a subclass of navigational concept , separate branch in the class hierarchy. IDO provides a more accurate definition and representation of pathogen. pathogen is a subclass of material entity with a pathogenic disposition . The latter is a subclass of disposition defined in IDO as "a disposition to initiate processes that result in a disorder". It means that some organisms in some circumstances of their lives play the role of pathogens towards other organisms and cause diseases. We create pathogen as a subclass of material entity and pathogen role as a subclass of role which is already defined in BFO. We add the axiom pathogen is a material entity and (′ has disposition′ some pathogenic disposition ) and (′ has role′ some pathogen role) . has disposition is an object property defined in RO as "a relation between an independent continuant (the bearer) and a disposition, in which the disposition specifically depends on the bearer for its existence". has role is also an object property which is defined in RO as "a relation between an independent continuant (the bearer) and a role, in which the role specifically depends on the bearer for its existence". 6. Conclusions The main objective of this work was to show how to build a medical ontology of quality. The involvement of domain experts in all development steps contributes to getting the required quality but bringing together such a team is not easy. Providing the knowledge engineer with building guidelines to compensate for the domain experts’ absence in some steps would be very helpful. It allows limiting the domain experts’ involvement in the ontology domain and scope definition phase and the evaluation one. We propose a methodology inspired by METHONTOLOGY [ 16 ] to which we made the following improvements: 1) in the corpus building and term extraction phase, we recommend the construction of knowledge corpus by using the most reliable source of knowledge that consists of a codified one; 2) in the building of preliminary ontology phase, we recommend the application of differential principles to explain the meaning of each ontology concept; the knowledge engineer expresses, in natural language, the similarities and differences that the concept has with those close to it in the concept hierarchy; 3) in the ontology reuse and enrichment phase, we recommend that the knowledge engineer use conflict resolution questions to resolve conflicts between concepts representations; 4) the methodology also provides guidance for the documentation (annotations, conceptual model of the ontology), the maintenance and evolution (versioning, dependency management), and the evaluation. The methodology combines ontology engineering from scratch using codified knowledge and reusing ontologies to enhance interoperability and reusability. The methodology can be used for other medical domains. For example, gold standard frameworks could be used as a source of codified knowledge to represent a prognosis domain. The suggested methodology could be very well applied to the building of ontologies covering the domains of other, not necessarily medical fields. In this case, one should identify a source of codified knowledge. A few examples of such sources are best practice guidelines for pollution prevention and waste minimization, billing norms in the industry, industrial standards, etc. We validate our methodology on pneumonia diagnosis by building PNADO. PNADO has been evaluated by physicians. The evaluation results showed that our methodology is effectual. PNADO is the first reported ontology developed to represent different pneumonia diagnosis aspects. During the PNADO building, in the ontology reuse phase, we suggested some improvements to the OGMS, SYMP, and CPRO ontologies. In future work, we will consider the modularity aspect in the building process of a medical ontology. Abbreviations BFO Basic Formal Ontology CIDO Coronavirus Infectious Disease CPRO Computer-Based Patient Record Ontology CPGs Clinical Practice Guidelines CRQs conflict resolution questions CSSO Clinical Signs and Symptoms Ontology CUI concept unique identifiers DL description logic DOID Human Disease Ontology GFO General Formal Ontology HPO Human Phenotype Ontology ICD10 International Classification of Diseases LOINC Logical Observation Identifiers Names and Code NCBO National Center for Biomedical Ontology OGMS Ontology for General Medical Science OM Omitted models OWL Web Ontology Language PNADO Pneumonia Diagnostic Ontology RadLex Radiology Lexicon RO Relation Ontology RTF Relative Term Frequency SNOMED-CT Systematized Nomenclature of Medicine Clinical Terms SUP Superfluous models TFIDF Term Frequency Inverted Document Frequency UMLS Unified Medical Language System. Declarations Competing interest Author declares that there is no conflict of interest statements. Authors’ contributions The author contributed all aspects to the paper. The author(s) read and approved the final manuscript. Funding The author declares that she has not received project funding for this work. Ethics approval and consent to participate Not applicable. Consent for publication Not applicable Acknowledgment We are indebted to the CISSSO (Centre Intégré de la Santé et des Services Sociaux de l'Outaouais) and specially Dr Sylvain Croteau, emergency physician at Gatineau, Qc. Hospital, Yasmine Lisa Rebaine, a pneumologist in Charles-Le Moyne Hospital, and Serge Chartrand, a family doctor at UQO medical clinic. Last but not least, we would like to express our gratitude to Wojtek Michalowski, a professor at Ottawa University, for his support. Supplementary material https://bioportal.bioontology.org/ontologies/PNADO References Patel A, Debnath NC. A Comprehensive Overview of Ontology: Fundamental and Research Directions. Curr Mater Science: Formerly: Recent Pat Mater Sci. 2024;17(1):2–20. Amith M, et al. Assessing the practice of biomedical ontology evaluation: Gaps and opportunities. J Biomed Inf. 2018;80:1–13. Amith M, et al. Architecture and usability of OntoKeeper, an ontology evaluation tool. BMC Med Inf Decis Mak. 2019;19(4):152. Katsumi M, Grüninger M. What is ontology reuse? in FOIS . 2016. Al-Arfaj A, Al-Salman A. Ontology construction from text: challenges and trends. Int J Artif Intell Expert Syst (IJAE). 2015;6(2):15–26. Pathak J, Johnson TM, Chute CG. Survey of modular ontology techniques and their applications in the biomedical domain. Integr computer-aided Eng. 2009;16(3):225–42. Simperl E. Reusing ontologies on the Semantic Web: A feasibility study. Data Knowl Eng. 2009;68(10):905–25. Zulkarnain NZ, Meziane F, Crofts G. A methodology for biomedical ontology reuse . in International conference on applications of natural language to information systems . 2016. Springer. Ochs C, et al. An empirical analysis of ontology reuse in BioPortal. J Biomed Inform. 2017;71:165–77. Fernández-López M, Gómez-Pérez A. Overview and analysis of methodologies for building ontologies. Knowl Eng Rev. 2002;17(2):129. Brendish NJ, et al. Hospitalised adults with pneumonia are frequently misclassified as another diagnosis. Respir Med. 2019;150:81–4. Uschold M, King M. Towards a methodology for building ontologies. Citeseer; 1995. Uschold M, et al. The enterprise ontology. Knowl Eng Rev. 1998;13(1):31–89. Grüninger M, Fox MS. Methodology for the design and evaluation of ontologies. 1995. Uschold M, Gruninger M. Ontologies: Principles, methods and applications. Knowl Eng Rev. 1996;11(2):93–136. Fernández-López M, Gómez-Pérez A, Juristo N. Methontology: from ontological art towards ontological engineering. 1997. Schultz DJ. IEEE standard for developing software life cycle processes. IEEE Std, 1997: pp. 1074–997. Noy NF, McGuinness DL. Ontology development 101: A guide to creating your first ontology . 2001, Stanford knowledge systems laboratory technical report KSL-01-05 and Stanford medical informatics technical report SMI-2001-0880, Stanford, CA. Al-Aswadi FN, Chan HY, Gan KH. Automatic ontology construction from text: a review from shallow to deep learning trend. Artif Intell Rev, 2019: pp. 1–28. Faure D, Nédellec. C. A corpus-based conceptual clustering method for verb frames and ontology acquisition. LREC workshop on adapting lexical and corpus resources to sublanguages and applications. Citeseer; 1998. Maedche A, Staab S. Ontology learning. Handbook on ontologies. Springer; 2004. pp. 173–90. Cimiano P, Völker J. text2onto . in International conference on application of natural language to information systems . 2005. Springer. Shamsfard M, Barforoush AA. Learning ontologies from natural language texts. Int J Hum Comput Stud. 2004;60(1):17–63. Hahn U, Romacker M. The SYNDIKATE Text Knowledge Base Generator . in Proceedings of the first international conference on Human language technology research . 2001. Gillani Andleeb S. From text mining to knowledge mining: An integrated framework of concept extraction and categorization for domain ontology. Budapesti Corvinus Egyetem; 2015. Navigli R, Velardi P, Gangemi A. Ontology learning and its application to automated terminology translation. IEEE Intell Syst. 2003;18(1):22–31. Buitelaar P, Olejnik D, Sintek M. A protégé plug-in for ontology extraction from text based on linguistic analysis . in European Semantic Web Symposium . 2004. Springer. Fortuna B, Grobelnik M, Mladenic D. Semi-automatic data-driven ontology construction system . in Proceedings of the 9th International multi-conference Information Society IS-2006, Ljubljana, Slovenia . 2006. Biébow B, Szulman S. Terminae: une approche terminologique pour la construction d’ontologies du domaine à partir de textes. Actes de RFIA2000, Reconnaissances des Formes et Intelligence Artificielle, 2000. Trokanas N, Koo L, Cecelja F. Towards a Methodology for Reusable Ontology Engineering: Application to the Process Engineering Domain , in Computer Aided Chemical Engineering . Elsevier; 2018. pp. 471–6. Dramé K, et al. Reuse of termino-ontological resources and text corpora for building a multilingual domain ontology: an application to Alzheimer’s disease. J Biomed Inform. 2014;48:171–82. Bright TJ, et al. Development and evaluation of an ontology for guiding appropriate antibiotic prescribing. J Biomed Inform. 2012;45(1):120–8. Hulshof CT. 1710d Systematic reviews and evidence-based guidelines, two of a different kind? BMJ Publishing Group Ltd.; 2018. Sackett DL, et al. Evidence based medicine: what it is and what it isn't. British Medical Journal Publishing Group; 1996. Gillois P, et al. From paper-based to electronic guidelines: application to French guidelines. Stud Health Technol Inf. 2001;84(Pt 1):196–200. Guarino N, Oberle D, Staab S. What is an ontology? Handbook on ontologies. Springer; 2009. pp. 1–17. Bachimont B, Isaac A, Troncy R. Semantic commitment for designing ontologies: a proposal . in International Conference on Knowledge Engineering and Knowledge Management . 2002. Springer. Grenon P, Smith B, Goldberg L. Biodynamic ontology: applying BFO in the biomedical domain. Studies in health technology and informatics, 2004: pp. 20–38. Arp R, Smith B. Function, role and disposition in basic formal ontology. Nat Precedings, 2008: pp. 1–1. Aronson AR. Metamap: Mapping text to the umls metathesaurus. Bethesda, MD: NLM, NIH, DHHS,; 2006. pp. 1–26. Katsumi M, Grüninger M. Choosing ontologies for reuse. Appl Ontology. 2017;12(3–4):195–221. Cui L, et al. Mining non-lattice subgraphs for detecting missing hierarchical relations and concepts in SNOMED CT. J Am Med Inform Assoc. 2017;24(4):788–98. Yamagata Y, et al. An ontological modeling approach for abnormal states and its application in the medical domain. J Biomedical Semant. 2014;5(1):23. Brank J, Grobelnik M, Mladenic D. A survey of ontology evaluation techniques . in Proceedings of the conference on data mining and data warehouses (SiKDD 2005) . 2005. Citeseer Ljubljana, Slovenia. Porzel R, Malaka R. A task-based approach for ontology evaluation. ECAI Workshop on Ontology Learning and Population, Valencia, Spain. Citeseer; 2004. Maedche A, Staab S. Measuring similarity between ontologies . in International Conference on Knowledge Engineering and Knowledge Management . 2002. Springer. Brewster C et al. Data driven ontology evaluation. 2004. Lozano-Tello A, Gómez-Pérez A. Ontometric: A method to choose the appropriate ontology. J Database Manage (JDM). 2004;15(2):1–18. Gómez-Pérez A. Evaluation of ontologies. Int J Intell Syst. 2001;16(3):391–409. Vrandečić D. Ontology evaluation. Handbook on ontologies. Springer; 2009. pp. 293–313. Zhu X, et al. A review of auditing methods applied to the content of controlled biomedical terminologies. J Biomed Inform. 2009;42(3):413–25. Burton-Jones A, et al. A semiotic metrics suite for assessing the quality of ontologies. Data Knowl Eng. 2005;55(1):84–102. Aruna T, Saranya K, Bhandari C. A survey on ontology evaluation tools . in 2011 International Conference on Process Automation, Control and Computing . 2011. IEEE. Corbridge C, et al. Laddering: technique and tool use in knowledge acquisition. Knowl Acquisition. 1994;6(3):315–41. Poveda-Villalón M, Gómez-Pérez A. Suárez-Figueroa, Oops!(ontology pitfall scanner!): An on-line tool for ontology evaluation . Int J Semantic Web Inform Syst (IJSWIS). 2014;10(2):7–34. Doran P, Tamma V, Iannone L. Ontology module extraction for ontology reuse: an ontology engineering perspective . in Proceedings of the sixteenth ACM conference on Conference on information and knowledge management . 2007. Kumar S, Baliyan N. Quality Evaluation of Ontologies , in Semantic Web-Based Systems . Springer; 2018. pp. 19–50. Halland K, Britz K. Investigations into the use of SNOMED CT to enhance an OpenMRS health information system. South Afr Comput J. 2011;47(1):33–45. El-Sappagh S, et al. SNOMED CT standard ontology based on the ontology for general medical science. BMC Med Inf Decis Mak. 2018;18(1):76. Shah T et al. A guiding framework for ontology reuse in the biomedical domain . in System Sciences (HICSS) , 2014 47th Hawaii International Conference on . 2014. IEEE. Johnson AE, et al. MIMIC-III, a freely accessible critical care database. Sci Data. 2016;3:160035. Metlay JP, et al. Diagnosis and Treatment of Adults with Community-acquired Pneumonia. An Official Clinical Practice Guideline of the American Thoracic Society and Infectious Diseases Society of America. Am J Respir Crit Care Med. 2019;200(7):e45–67. Footnotes https://bioportal.bioontology.org/ontologies/CIDO , last access 08/08/2024. https://bioportal.bioontology.org/ontologies/COVID-19 , last access 08/08/2024. http://www.obofoundry.org/ontology/doid.html , last access 08/08/2024. https://browser.ihtsdotools.org/? , last access 08/08/2024. http://www.obofoundry.org/ontology/hp.html , last access 08/08/2024. https://www.cochranelibrary.com/ , last access 08/07/2024. https://www.nice.org.uk/guidance , last access 08/07/2024. https://ctakes.apache.org/index.html , last access 08/07/2024. https://www.nlm.nih.gov/research/umls/index.html , last access 08/08/2024 https://bioportal.bioontology.org/ , last access 08/08/2024. http://www.obofoundry.org/ , last access 08/08/2024. http://www.protegewiki.stanford.edu/ , last access 08/08/2024. http://www.swoogle.umbc.edu/ , last access 08/08/2024. https://bioportal.bioontology.org/recommender , last access 08/08/2024. https://bioportal.bioontology.org/ontologies/CIDO/?p=summary , last access 08/08/2024. http://oops.linkeddata.es/ , last access 08/08/2024. https://njitsaboc.github.io/ , last access 08/08/2024. https://ontometrics.informatik.uni-rostock.de/ontologymetrics/index.jsp , last access 08/08/2024. https://webprotege.stanford.edu/ , last access 08/08/2024. http://www.obofoundry.org/principles/fp-000-summary.html , last access 08/08/2024. https://protegewiki.stanford.edu/wiki/OWLDoc , last access 08/08/2024. https://essepuntato.it/lode/ , last access 08/08/2024. https://www.cochranelibrary.com/ , last access 08/08/2024. https://www.nice.org.uk/ , last access 08/08/2024. https://www.idsociety.org/ , last access 08/08/2024. https://www.escmid.org/ , last access 08/08/2024. https://www.asid.net.au/ , last access 08/08/2024. https://cts-sct.ca/ , last access 08/08/2024. https://pulmccm.org/ , last access 08/08/2024. https://www.thoracic.org/ , last access 08/08/2024. https://www.brit-thoracic.org.uk/ , last access 08/08/2024. http://www.obofoundry.org/ontology/bfo.html , last access 08/08/2024. https://bioportal.bioontology.org/ontologies/GFO , last access 08/08/2024. https://bioportal.bioontology.org/ontologies/OGMS , last access 08/08/2024. https://bioportal.bioontology.org/ontologies/DIAGONT , last access 08/08/2024. http://www.obofoundry.org/ontology/symp.html , last access 08/08/2024. https://bioportal.bioontology.org/ontologies/CSSO , last access 08/08/2024. https://browser.ihtsdotools.org/? , last access 08/08/2024. http://www.obofoundry.org/ontology/ncbitaxon.html , last access 08/08/2024. https://bioportal.bioontology.org/ontologies/GAMUTS , last access 08/08/2024. http://radlex.org/ , last access 08/08/2024. https://loinc.org/ , last access 08/08/2024. https://bioportal.bioontology.org/ontologies/LABO , last access 08/08/2024. https://bioportal.bioontology.org/ontologies/ICD10 , last access 08/08/2024. http://www.obofoundry.org/ontology/doid.html , last access 08/08/2024. http://www.obofoundry.org/ontology/ro.html , last access 08/08/2024. https://bioportal.bioontology.org/ontologies/CPRO , last access 08/08/2024. http://www.obofoundry.org/ontology/hp.html , last access 08/08/2024. http://www.obofoundry.org/ontology/ido.html , last access 08/08/2024. http://physionet.org/ , last access 08/08/2024. https://santemonteregie.qc.ca/installations/hopital-charles-le-moyne , last access 08/08/2024. https://cisss-outaouais.gouv.qc.ca/ , last access 08/08/2024. https://ssuqo.ca/ , last access 08/08/2024. https://www.cebm.net/ , last access 08/08/2024. Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5305559","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":373701219,"identity":"c520c763-40bb-4a32-bc7c-5233f9382832","order_by":0,"name":"Sabrina Azzi","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA6klEQVRIie2RLQvCUBSGz7gwy4XVJfcXJiaD7q9sDExqtBhcuhZhdcEfYrzjwFam1sGKQzANWdSkV/Ej7Wo03CcdDjyc9+UAKBR/CQVwnyOv31v+oxJHPysvyGeWKNZik9TlegBGmJbYZ+hAC/eEZs2KnU38yMt8MDl3cczQC+jQJjSXKDDqgseIKBJwoRSi10gkrJsVK6zuyhws1ALsscIBo5IrkD+uINgJAdRYoQXm/YosWH7yhZLSTqZDvNxePWYe7XglqW+FY9QubNZu79JDfZ4OHcPwy32VSII9+XxEhy+PVCgUCsV3bt0NUZI1y3ZIAAAAAElFTkSuQmCC","orcid":"","institution":"University of the West of Scotland","correspondingAuthor":true,"submittingAuthor":false,"prefix":"","firstName":"Sabrina","middleName":"","lastName":"Azzi","suffix":""}],"badges":[],"createdAt":"2024-10-21 15:08:19","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5305559/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5305559/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":68696060,"identity":"6f3f9716-c860-4539-93e0-b69ec2a54968","added_by":"auto","created_at":"2024-11-11 06:52:49","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":113988,"visible":true,"origin":"","legend":"\u003cp\u003eMethodology phases\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-5305559/v1/c8aace977af378865e927511.png"},{"id":68697698,"identity":"77f74576-2820-44a7-becc-fa5e81616968","added_by":"auto","created_at":"2024-11-11 07:00:49","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":83020,"visible":true,"origin":"","legend":"\u003cp\u003eThe relationship between intended models for an ontology and models of the ontology’s axioms (based on [41]).\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-5305559/v1/7a2348e54a8f4a51e11029f0.png"},{"id":68696062,"identity":"71b2bbfd-918c-44eb-8121-6a039b228afc","added_by":"auto","created_at":"2024-11-11 06:52:49","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":197317,"visible":true,"origin":"","legend":"\u003cp\u003eOntology enrichment and resolving conflicts approach\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-5305559/v1/e2b5e6345c566109e03ee4eb.png"},{"id":68696061,"identity":"67fbe4a7-8789-4053-9104-6c4cac004c73","added_by":"auto","created_at":"2024-11-11 06:52:49","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":53300,"visible":true,"origin":"","legend":"\u003cp\u003eMIMIC-III pneumonia cases\u003c/p\u003e","description":"","filename":"4.png","url":"https://assets-eu.researchsquare.com/files/rs-5305559/v1/cea3238e4bf7aed1a8fe53b5.png"},{"id":68696065,"identity":"45e0c5b7-d086-4fab-bd99-d192f82c0c2f","added_by":"auto","created_at":"2024-11-11 06:52:49","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":168562,"visible":true,"origin":"","legend":"\u003cp\u003eThe disease class with infective pneumonia as an example of the subclass.\u003c/p\u003e","description":"","filename":"5.png","url":"https://assets-eu.researchsquare.com/files/rs-5305559/v1/d5f487a71dde6d22a4011b6e.png"},{"id":68696064,"identity":"4d07b5d9-7d77-474d-8f6b-ce1c26e89fac","added_by":"auto","created_at":"2024-11-11 06:52:49","extension":"png","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":304261,"visible":true,"origin":"","legend":"\u003cp\u003eRepresentation of \u003cem\u003ehealthcare process assay\u003c/em\u003e class in Protégé.\u003c/p\u003e","description":"","filename":"6.png","url":"https://assets-eu.researchsquare.com/files/rs-5305559/v1/01c81d8e0acc6541dc09f725.png"},{"id":75435209,"identity":"20fadcbe-51ce-4ad0-9cc6-cf05cfa40acc","added_by":"auto","created_at":"2025-02-04 14:16:59","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2631240,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5305559/v1/be66f1c6-0b18-45d1-9528-6a5c388985da.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"A methodology for building a medical ontology a with a limited domain experts’ involvement","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eThe contribution of ontologies to understanding, sharing, and integration of knowledge is well established. The usage of ontologies in medicine gains ground thanks to their ability to represent the medical domain knowledge and their expressive power formally [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]. An ontology can assure the interoperability and improve the functionality of a clinical decision support system by providing a standard vocabulary to help to integrate heterogeneous biomedical data sources [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThe ontology building process is composed of several steps. Each step requires a set of comprehensive instructions and techniques of how it should be carried out, and together they form an ontology building methodology. The completeness and the degree of details of the latter are very important to build an ontology of quality.\u003c/p\u003e \u003cp\u003eMedical ontologies have significantly grown over the last decade, but they have quality and content problems [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e, \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. This is due, among others, to the following causes: 1) medical ontologies are created with limited expertise either in the medical domain or in ontology engineering, for example, Coronavirus Infectious Disease Ontology\u003csup\u003e1\u003c/sup\u003e is created by a medical expert whereas COVID-19 Ontology\u003csup\u003e2\u003c/sup\u003e is created by a knowledge engineer who confuses signs with symptoms; 2) they are often built from scratch and do not take advantage of reusing existing ontologies that have already been validated to save time and improve the interoperability; 3) the applied ontology building methodology lacks some important steps like evaluation or validation, or during the building process some steps are skipped.\u003c/p\u003e \u003cp\u003eMany methodologies have been proposed to build an ontology from a corpus of text and/or by reusing elements of existing ontologies, which is called ontology reuse. The latter is the process of selecting existing ontologies and manipulating them in some way to meet the requirements and needs defined in the conception [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThe main shortcoming of methodologies of building from text is that they do not guide how the different steps should be conducted. This makes them difficult to use, time-consuming, possibly error-prone, and costly [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e], especially in situations where the involvement of domain experts is limited. Also, even they recognize ontology reuse as part of the overall ontology construction process, they address it briefly.\u003c/p\u003e \u003cp\u003eOn the other hand, ontology reuse reduces the ontology development time and required effort since the knowledge engineer can focus on new content specific to the ontology being created rather than developing content that has already been created and validated [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e]. Besides, the reuse of ontologies increases interoperability [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e], reduces redundancy [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e], and prevents the random proliferation of ontologies. However, ontology reuse has the drawback of reproducing the errors of the reused ontologies. During the reuse process, inconsistencies and conflicts between concepts differently represented in different ontologies have to be fixed. Even though ontology reuse is recommended [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e], to the best of our knowledge, there are currently no methodologies that provide adequate support for the ontology reuse process and resolve the problems identified above. Another important problem related to the reuse is the lack of documentation explaining the conceptual model of the ontology and/or the design criteria that were followed to build and evaluate the ontology [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. This problem concerns most of the published medical ontologies.\u003c/p\u003e \u003cp\u003eIn this paper, we propose a methodology designed for a knowledge engineer for building a medical ontology that deals with the interoperability and conflicts between concept representations, with minimal involvement of medical experts. Depending on the ontology domain, the methodology relies on the ability to access already somehow codified knowledge. Examples of such knowledge for a medical domain include clinical practice guidelines, systematic reviews, and gold standard frameworks.\u003c/p\u003e \u003cp\u003eOur methodology guides the knowledge engineer step by step by providing the necessary instructions. It consists of seven phases: 1) definition of ontology domain and scope; 2) building of knowledge corpus and extraction of terms; 3) building of preliminary ontology by applying ARCHONTE methodology on the terms previously identified; 4) reuse of ontologies and resolution of conflicts using competency questions; 5) evaluation of the ontology; 6) documentation and; 7) maintenance and evolution. The whole process is executed with limited involvement of the medical experts.\u003c/p\u003e \u003cp\u003eWe illustrate our ideas with a case study of building pneumonia diagnosis ontology (PNADO). Pneumonia is one of the diseases associated with many diagnostic errors [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]. To the best of our knowledge, PNADO is the first comprehensive ontology of pneumonia diagnosis. Several of the existing medical ontologies, such as Human Disease Ontology (DOID\u003csup\u003e3\u003c/sup\u003e), SNOMED-CT\u003csup\u003e4\u003c/sup\u003e and Human Phenotype Ontology (HPO\u003csup\u003e5\u003c/sup\u003e), contain concepts related to the diagnosis of pneumonia, however, they provide different representations for some concepts and they do not capture the entire domain knowledge.\u003c/p\u003e"},{"header":"2. State of the art","content":"\u003cp\u003eMost of the first ontology engineering methodologies were developed as a by-product of ontology development for some specific projects. ENTERPRISE [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e, \u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e] and TOVE [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e] (Toronto Virtual Enterprise) are examples of projects that relate to the domain of enterprise modeling, developed in 1995. TOVE concentrates on modeling the internal workflow of a company. The methodology principle is that the authors address ontology completeness based on competency questions. ENTERPRISE methodology also called Uschold \u0026amp; King\u0026rsquo;s methodology is based on the experience of developing the Enterprise Ontology. The four phases that are recommended in the methodology are: 1) identify the purpose, scope, and scenarios for its use; 2) build the ontology including ontology capture, coding, and integrating existing ontologies; 3) document the ontology; 4) evaluate the ontology. A refined version of this methodology was proposed in 1996 by Uschold and Gruninger [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]. In the same year, a well-structured methodology called METHONTOLOGY [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e] was developed in the Laboratory of Artificial Intelligence at the Polytechnic University of Madrid. The phases proposed in METHONTOLOGY are: 1) specification that defines the purpose and scope of the ontology; 2) knowledge acquisition; 3) conceptualization that consists in structuring the domain knowledge in a conceptual model that describes the problem and its solution according to the domain vocabulary identified in the specification phase; 4) implementation; 5) integration; 6) evaluation; 7) documentation.\u003c/p\u003e \u003cp\u003eThe main shortcoming of these methodologies is that they consider texts as the only sources of knowledge, and they do not provide guidance on how the different steps should be conducted. An analysis of these methodologies for building ontologies was conducted in [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. The study demonstrated that none of them are fully mature comparing them with IEEE standards [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. However, METHONTOLOGY is the most mature but some steps require details.\u003c/p\u003e \u003cp\u003e101 method provides guidelines to manually develop domain ontologies [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e]. It provides strategies to assist in making design decisions during ontology construction. It declares the possibility of reusing ontologies without specifying how. Also, the methodology focuses on defining class hierarchies, properties of classes and instances, and omits an important step which is the evaluation.\u003c/p\u003e \u003cp\u003eSince developing an ontology from text is difficult and time-consuming, frameworks and tools were also proposed to support the user in the process of building. They use machine learning, linguistic, logic, and statistics techniques [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e]. ASIUM [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e], TextToOnto [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e], Text2Onto [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e], HASTI[\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e], SYnthesis of DIstributed Knowledge Acquired from TExts (SYNDIKATE)[\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e], ProMine[\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e], OntoLearn [\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e], OntoLT [\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e], OntoGen [\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e], Terminae [\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e] are examples of these tools. However, all these tools suffer from several shortcomings such as the difficulty of extracting relations and axioms. Authors of [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e] recommend using deep learning rather than shallow learning for automatic ontology construction.\u003c/p\u003e \u003cp\u003eWhile ontology reuse is supposed to facilitate and speed up the development process, it remains a very important issue especially in the biomedical domain where large resources are available. Several methodologies have been proposed to deal with this issue:\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003eMetROn was suggested to build a reusable ontology by reusing the existing ontologies and other resources such as classifications, vocabularies, terminologies, and standards [\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e]. Top-down, bottom-up, middle-out approaches, and even a combination of them are proposed to be used for defining classes;\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eA methodology for biomedical ontology reuse was proposed in [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e] and applied for Abdominal Ultrasound Ontology development. It provides steps and tools for finding and choosing relevant ontologies to reuse as well as concepts mapping;\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eA methodology was proposed in [\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e] to build a multilingual domain ontology by reusing termino-ontological resources and text corpora. The methodology was applied to building Alzheimer\u0026rsquo;s disease ontology;\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eAuthors in [\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e] proposed seven steps to develop an ontology for guiding appropriate antibiotic prescribing intended for a clinical decision support system. These steps are: 1) define the domain ontology and scope; 2) review existing ontologies for reuse; 3) create classes and properties; 4) create a conceptual model; 5) select and implement upper ontology; 6) implement the ontology in a formal representation; 7) evaluate the ontology. The methodology evaluates the resulted ontology using Semantic Web Rule Language (SWRL) rules.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003cp\u003eAll these methodologies lack guidance in carrying out the steps and not raise the problem of conflicts between concepts representations and how to resolve them.\u003c/p\u003e"},{"header":"3. Material and Methods","content":"\u003cp\u003eIn this section, we present a methodology inspired by METHONTOLOGY [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e] to which we made several improvements. Figure\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e presents the seven phases of the methodology and shows that domain experts are only involved in the definition of the ontology domain and scope phase and the evaluation phase.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e3.1 Phase 1: Ontology domain and scope definition\u003c/h2\u003e \u003cp\u003eThis phase consists of establishing the domain of interest of the ontology to model and its scope. This can be done either by consulting domain experts (physicians) or by finding a gap in the literature. First, two fundamental questions must be treated:\u003c/p\u003e \u003c/div\u003e\n\u003ch3\u003e1. What is the domain that the ontology will cover?\u003c/h3\u003e\n\n\u003ch3\u003e2. For what purpose are we going to create this ontology?\u003c/h3\u003e\n\u003cp\u003eSecond, functional requirements for the ontology or questions that an ontology should be able to answer are defined. To support this, competency questions (CQs) are established by physicians and/or extracted from codified knowledge sources (section \u003cspan refid=\"Sec7\" class=\"InternalRef\"\u003e3.2\u003c/span\u003e). The CQs have the form of informal queries that a particular community of users think that the ontology should answer [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e]. They are used in many methodologies for ontology building to specify requirements [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e, \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e, \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e]. The CQs are a sketch and do not have to be exhaustive.\u003c/p\u003e \u003cp\u003eAt the end of this phase, the knowledge engineer has a clear vision of what the ontology will cover, for what the ontology will be used, and what information it will contain.\u003c/p\u003e \u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003e3.2 Phase 2: Corpus building and term extraction\u003c/h2\u003e \u003cp\u003eIn this phase, the knowledge engineer builds the corpus of knowledge extracts and identifies relevant terms to the ontology domain defined in phase 1. The use of codified knowledge makes this task easier for the knowledge engineer since this type of source is more reliable and comprehensible. Since the quality of ontology depends on the quality of the knowledge corpus, we recommend that for a medical domain, the knowledge corpus be composed of the most reliable and complete sources such as clinical practice guidelines (CPGs) or systematic reviews related to the medical domain for which ontology is being built. Another reason to use codified knowledge is that there are no available annotated texts that could be used as a base to create a medical ontology.\u003c/p\u003e \u003cp\u003eSystematic reviews investigate and summarize the scientific literature in a systematic and methodologically rigorous way. Each of them focuses on a specific question and uses methods to study and summarize the findings of similar but separate studies. They are considered to be the most important type of scientific review as they are fundamental to evidence-based practice [\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e]. They are the basis of the CPGs. The latter are medical documents resuming the state of the art in the management of pathologies and clinical cases, which we call basic recommendations. They are consulted by healthcare practitioners to guide diagnostic and management decisions. They are based on the concept of Evidence-Based Medicine (EBM). In [\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e], EBM was defined as \u003cem\u003e\u0026laquo; the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients.\u0026raquo;.\u003c/em\u003e The CPGs are found in libraries such as Cochrane\u003csup\u003e6\u003c/sup\u003e, National Institute for Health and Care Excellence (NICE\u003csup\u003e7\u003c/sup\u003e) and other libraries that are more specific to a specific condition. In fact, given the high degree of completeness of the CPGs regarding the most important concepts, they can be integrated into clinical decision support systems, thus providing recommendations that can be adapted to the patient's profile at the time of decision making [\u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eWe distinguish between a term and a concept. According to ISO 1087:2019, a term is a designation that represents a general concept by linguistic means whereas a concept is a unit of knowledge created by a unique combination of characteristics. We represent concepts corresponding to the extracted terms in the ontology in phase 3 (section \u003cspan refid=\"Sec8\" class=\"InternalRef\"\u003e3.3\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eTerm extraction can be manual or automatic. Some of the tools presented in section \u003cspan refid=\"Sec2\" class=\"InternalRef\"\u003e2\u003c/span\u003e could be used for automatic term extraction from a corpus of knowledge constituted of CPGs. Unfortunately, not all of them are available to use anymore. There are also powerful tools for clinical text analysis such as cTAKES\u003csup\u003e8\u003c/sup\u003e. They could be used for term extraction; however, their installation and usage are quite heavy. We suggest using Text2Onto [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e] which is more suitable for this task. It is available, user friendly and its installation and usage are simple. Text2Onto has been developed to support the acquisition of ontologies from textual documents. It is composed of modules that extract terms, relationships (equivalence relationship, hierarchical, etc.), and instances. It offers extraction algorithms calculating the following measures: Relative Term Frequency (RTF), Term Frequency Inverted Document Frequency (TFIDF), Entropy, and the C-value/NC-value. We recommend using TFIDF when there is more than one document because it evaluates how relevant a term is to a document in a collection of documents.\u003c/p\u003e \u003cp\u003eThe extracted terms that should be analyzed are nouns, proper nouns, verbs, adjectives, adverbs, and phrases. These terms represent the concepts that will be treated in the next phase.\u003c/p\u003e \u003cp\u003eIt should be noted that the extraction of relations and axioms is not obvious and most of the existing learning systems do not infer new relations or axioms [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e]. According to [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e], deep learning techniques could be very useful for learning and inferring relations and axioms.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003e3.3 Phase 3: Building of preliminary ontology\u003c/h2\u003e \u003cp\u003eThis phase is dedicated to the building of a preliminary medical ontology. This preliminary ontology represents the intended models of the requirements defined in phase 1. Usually, an ontology can have many interpretations. To limit the number of possible models, it uses axioms. Nevertheless, it is usually impossible to create an ideal ontology whose models coincide with the intended ones. An intended model represents all the requirements and only them [\u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eWe propose to follow the ARCHONTE methodology to build a preliminary ontology [\u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e]. The strong point of this methodology is the recommendation of semantic commitment in the design process. In other words, to explain the meaning of each of the ontology concepts represented by the terms already selected in phase 2, the knowledge engineer expresses, in natural language, the similarities and differences that the concept has with those close to it. The ontology structure is similar to a tree, which facilitates the determination of the meaning that a concept has according to its position. ARCHONTE is composed of three steps: 1) semantic normalization; 2) knowledge formalization; 3) toward a computational ontology. They are elaborated below.\u003c/p\u003e \u003cdiv id=\"Sec9\" class=\"Section3\"\u003e \u003ch2\u003e3.3.1 Step 1: Semantic normalization\u003c/h2\u003e \u003cp\u003eThe objective of this step is to reach a semantic agreement about the meaning of the concept labels. The knowledge engineer applies differential principles on the set of candidate concepts chosen previously. According to the differential paradigm proposed by Bachimont [\u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e], the meaning of a concept (or a node, since an ontology is structured as a tree) is determined by its closest neighbors: parent and its siblings, \u003cem\u003ei.e\u003c/em\u003e., its position in the ontological hierarchy based on terminological structure. Similarities and differences of each concept concerning its parent and its siblings are expressed in natural language. Four differential principles are distinguished: 1) similarity with each parent; 2) similarity with each sibling; 3) difference with each sibling; 4) difference with each parent. The result of this step is a so-called differential ontology. A differential ontology is an ontology where each concept and relationship must be defined according to its similarities and differences with its parents and siblings.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec10\" class=\"Section3\"\u003e \u003ch2\u003e3.3.2 Step 2: Knowledge formalization\u003c/h2\u003e \u003cp\u003eIn this step, the referential ontology is considered from an extensional semantics point of view. Each concept is linked to a set of objects that allows defining new concepts and relations through set operations. During this step, the ontology engineer has to precise the arity and domains of the relations, in correspondence with the intended models. He can also add some logical axioms to constrain the domains of the relations. The result of this step is a so-called referential ontology. A referential ontology is a formal ontology that covers the concepts, relations, instances, and axioms for a domain.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec11\" class=\"Section3\"\u003e \u003ch2\u003e3.3.3 Step 3: Towards a computational ontology\u003c/h2\u003e \u003cp\u003eThis last step of ARCHONTE methodology is implementing the referential ontology in an operational language of knowledge representation such as OWL. This is achieved by adopting, on the one hand, a formalism of representation (conceptual graphs, description logic), and secondly, by adapting the representation of the ontology to the objectives fixed in phase 2. This step marks the transition to a computational ontology. For interoperability and reusability purposes, we recommend following OBO Foundry principles. An upper medical ontology that provides the most general concepts in the medical domain can be reused to enhance interoperability with other medical ontologies. BFO (Basic Formal Ontology) is widely used as an upper ontology in OBO Foundry and the biomedical domain [\u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e38\u003c/span\u003e]. This makes BFO the best choice for building any medical ontology to enhance interoperability. It aims to model the basic structures of reality and provides classes that categorize, at a high level, real-world entities. BFO is designed to support information retrieval, analysis, and integration in several domains. It has two basic types of entities: (1) continuants that are entities that continue or persist through time, such as objects, qualities, and functions, and (2) occurrents that are events or happenings in which continuants participate [\u003cspan citationid=\"CR39\" class=\"CitationRef\"\u003e39\u003c/span\u003e]. Also, there is usually an upper ontology for a specific domain that includes general classes related to this domain. OGMS (Ontology for General Medical Science) is one of these ontologies [\u003cspan citationid=\"CR40\" class=\"CitationRef\"\u003e40\u003c/span\u003e]. OGMS enriches BFO by including the concepts of general medical science. This consists of approximately 100 terms that describe the fundamental aspects of medicine such as \u003cem\u003edisorder\u003c/em\u003e, \u003cem\u003ediagnosis\u003c/em\u003e, \u003cem\u003edisease\u003c/em\u003e, \u003cem\u003edisease course\u003c/em\u003e, \u003cem\u003elaboratory test\u003c/em\u003e, \u003cem\u003esign\u003c/em\u003e, \u003cem\u003esymptom\u003c/em\u003e, and \u003cem\u003esyndrome\u003c/em\u003e.\u003c/p\u003e \u003cp\u003eAt the development level, Prot\u0026eacute;g\u0026eacute; ontology editor, which offers the possibility to visualize the different aspects of the ontology, can be used.\u003c/p\u003e \u003cp\u003eFor further enrichment of the ontology, UMLS\u003csup\u003e9\u003c/sup\u003e (Unified Medical Language System) can be used to extract definitions, synonyms, acronyms, and other annotations. Concepts in the ontology can be mapped to the referenced resource, UMLS concept unique identifiers (CUIs), where possible. Mapping the ontology to UMLS enables the ontology to be linked to many other relevant biomedical resources such as LOINC and SNOMED-CT. MetaMap [\u003cspan citationid=\"CR40\" class=\"CitationRef\"\u003e40\u003c/span\u003e] is a tool designed to identify biomedical concepts from free-text and maps them into concepts from the UMLS. Every concept has, when possible, CUI code, preferred label, definitions, synonyms, and external references of the reused ontologies. This step overlaps with the ontology reuse phase.\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003e3.4 Phase 4: Ontology reuse and enrichment\u003c/h2\u003e \u003cp\u003eIn this phase, we finalize the preliminary ontology by adding and detailing concepts. For this purpose, we reuse some terminological resources that we consider relevant to the ontology domain. The reuse can be hard, \u003cem\u003ei.e.\u003c/em\u003e, one imports the complete ontology to reuse, or soft, \u003cem\u003ei.e.\u003c/em\u003e, one imports the reused ontology concepts. The ontologies to be reused have to be evaluated for the suitability of their content coverage and the depth of knowledge, as well as the potential to support the inference process. The phase is composed of three steps: 1) finding ontologies; 2) choosing relevant ontologies; 3) resolving conflicts.\u003c/p\u003e \u003cdiv id=\"Sec13\" class=\"Section3\"\u003e \u003ch2\u003e3.4.1 Step 1: Finding ontologies\u003c/h2\u003e \u003cp\u003eIn this step, we select an ontology for reuse when we conjecture that its axioms could be used to respond to some of the requirements defined in phase 1. We use four main search criteria to identify candidate ontologies for reuse:\u003c/p\u003e \u003cp\u003e \u003cem\u003ei)\u003c/em\u003e both the natural and formal languages of the candidate ontology are the same as those of the ontology we are building;\u003c/p\u003e \u003cp\u003e \u003cem\u003eii)\u003c/em\u003e there is a mapping from a non-empty subset of the new ontology requirements (defined in phase 1) to a robust set of concepts of the ontology candidate;\u003c/p\u003e \u003cp\u003e \u003cem\u003eiii)\u003c/em\u003e the candidate ontology is accepted within its user community. This acceptance is assessed by using some metrics such as the number of community members that endorse the ontology, the number of times the ontology has been reused, etc.;\u003c/p\u003e\u003cp\u003e \u003cem\u003eiv)\u003c/em\u003e the candidate ontology has been published in a peer-reviewed publication.\u003c/p\u003e\u003cp\u003eOntology repositories, such as NCBO BioPortal\u003csup\u003e10\u003c/sup\u003e, OBO Foundry\u003csup\u003e11\u003c/sup\u003e, Prot\u0026eacute;g\u0026eacute; Wiki\u003csup\u003e12\u003c/sup\u003e, Swoogle\u003csup\u003e13\u003c/sup\u003e (Ontology Lookup Service from the European Bioinformatics Institute), UMLS that integrates many terminologies and coding standards, are used to find relevant ontologies.\u003c/p\u003e \u003cp\u003eThe tool of BioPortal, Ontology Recommender\u003csup\u003e14\u003c/sup\u003e, which uses among others, coverage and acceptance metrics, could be used to find ontologies in this repository.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec14\" class=\"Section3\"\u003e \u003ch2\u003e3.4.2 Step 2: Choosing relevant ontologies\u003c/h2\u003e \u003cp\u003eThis step involves selecting the most relevant ontologies to be reused. For each significant topic found in the requirements defined in phase 1, we are looking for ontologies that deal with this topic, so-called relevant ontologies. Examples of such topics might be symptoms or laboratory tests.\u003c/p\u003e \u003cp\u003eThe preference criteria for choosing relevant ontologies are accuracy and precision orderings [\u003cspan citationid=\"CR41\" class=\"CitationRef\"\u003e41\u003c/span\u003e]. To compare the accuracy and precision of ontology candidates, two kinds of models are considered:\u003c/p\u003e \u003cp\u003e \u003col style=\"list-style-type:lower-roman;\"\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eSuperfluous models (SUP) that correspond to a situation where some aspect of the candidate is weaker than required, \u003cem\u003ei.e\u003c/em\u003e., they prevent some aspects of the requirements from being satisfied (see Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e) [\u003cspan citationid=\"CR41\" class=\"CitationRef\"\u003e41\u003c/span\u003e]; for example, Radiology Lexicon (RadLex) covers several unnecessary concepts for pneumonia diagnosis.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eOmitted models (OM) that correspond to a situation where some aspect of the candidate is stronger than is required, \u003cem\u003ei.e\u003c/em\u003e., this candidate ontology exceeds what was specified in the requirements (see Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e); for example, CIDO\u003csup\u003e15\u003c/sup\u003e (Coronavirus Infectious Disease) is weaker than PNADO intended model, because it does not satisfy some ontology requirements defined in phase 1 (\u003cem\u003ei.e\u003c/em\u003e., some concepts defined in PNADO are not covered by CIDO).\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003c/ol\u003e \u003c/p\u003e \u003cp\u003eIf there are two candidate ontologies T\u003csub\u003e1\u003c/sub\u003e and T\u003csub\u003e2\u003c/sub\u003e that deal with the same topic, we choose them as follows:\u003c/p\u003e \u003cp\u003e \u003cstrong\u003eCase 1\u003c/strong\u003e \u003cp\u003eIf the set of superfluous models of T\u003csub\u003e1\u003c/sub\u003e is included in the set of superfluous models of T\u003csub\u003e2\u003c/sub\u003e and the set of omitted models of T\u003csub\u003e1\u003c/sub\u003e is included in the set of omitted models of T\u003csub\u003e2,\u003c/sub\u003e then T\u003csub\u003e1\u003c/sub\u003e is more \u003cem\u003eaccurate\u003c/em\u003e than the candidate ontology T\u003csub\u003e2\u003c/sub\u003e. We recommend choosing T\u003csub\u003e1\u003c/sub\u003e. If the set of superfluous models of T\u003csub\u003e1\u003c/sub\u003e is empty, hard reuse should be considered.\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eCase 2\u003c/strong\u003e \u003cp\u003eIf T\u003csub\u003e1\u003c/sub\u003e and T\u003csub\u003e2\u003c/sub\u003e have no superfluous models then both are \u003cem\u003eprecise\u003c/em\u003e and we reuse both of them. If the intersection of the sets of intended models is empty, then hard reuse should be considered. If not, hard reuse can be considered only if there are no conflicts between the representations of shared concepts.\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eCase 3\u003c/strong\u003e \u003cp\u003eIf T\u003csub\u003e1\u003c/sub\u003e has superfluous models and T\u003csub\u003e2\u003c/sub\u003e has not, then regardless of omitted models, T\u003csub\u003e2\u003c/sub\u003e is more \u003cem\u003eprecise\u003c/em\u003e, we reuse T\u003csub\u003e2\u003c/sub\u003e and hard reuse should be considered.\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eCase 4\u003c/strong\u003e \u003cp\u003eIf both T\u003csub\u003e1\u003c/sub\u003e and T\u003csub\u003e2\u003c/sub\u003e have superfluous models then they are incomparable using the accuracy and precision orderings, and reuse is problematic. If so, soft reuse should be considered.\u003c/p\u003e \u003c/p\u003e \u003cp\u003eIf there are more than two candidate ontologies, we apply the same algorithm by comparing two ontologies at a time.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec15\" class=\"Section3\"\u003e \u003ch2\u003e3.4.3 Step 3: Resolving conflicts\u003c/h2\u003e \u003cp\u003eOnce the ontologies for reuse have been selected, we proceed to manually identify the concepts that will enrich our original ontology. We are interested in adding new concepts and completing the concepts already represented in our ontology.\u003c/p\u003e \u003cp\u003eA concept may be differently represented in different ontologies. In this case, there is a conflict and we have to decide which representation should be used. To resolve the conflict, we use conflict resolution questions (CRQs) defined by the knowledge engineer. These CRQs are related to the meaning and the structure of the ontology concept tree, \u003cem\u003ei.e\u003c/em\u003e., the hierarchy of classes, and other description logic (DL) concepts like intersection and union of classes, equivalent classes, universal classes, universal and existential quantification, has-value restriction, and cardinality restriction. We apply the principle that the concept axiomatization model is semantically correct if it does include an ontology intended model. Here are a few examples of CRQs:\u003c/p\u003e \n\u003col\u003e\n \u003cli\u003eDoes the concept have a definition (in the form of annotation)? If yes, does its definition correspond to an intended model of the ontology?\u003c/li\u003e\n \u003cli\u003eDoes the concept have a superclass? If yes, is it relevant to the ontology?\u003c/li\u003e\n \u003cli\u003eDoes the concept have children? If yes, is each of them relevant to the ontology?\u003c/li\u003e\n \u003cli\u003eDoes the concept have synonyms?\u003c/li\u003e\n \u003cli\u003eDoes the concept have alternative names?\u003c/li\u003e\n \u003cli\u003eDoes the concept have related names?\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eThe knowledge engineer should propose and use any CRQ that he considers suitable to decide on the most appropriate concept representation.\u003c/p\u003e \u003cp\u003eTo complete an already existing concept with annotations or add a new concept, we proceed as follows (see Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e):\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003eIf the concerned concept is not found in any ontology, then we propose some relevant annotations that can be obtained from UMLS, such as definitions, synonyms, or concept unique identifier (CUI). If the concept is not found in UMLS, no annotation is added.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eIf the concept is represented in only one ontology, then we reuse it if relevant, \u003cem\u003ei.e\u003c/em\u003e., its representation fits the requirements, with its identifier and annotations. If not, we propose a new identifier. If the concept is found in UMLS, we propose to enrich it with annotations.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eIf the concept is found in more than one ontology, then there is a conflict and we have to choose the ontology that provides the best concept representation for the domain, \u003cem\u003ei.e.\u003c/em\u003e, the one that provides answers to the CRQs and corresponds to an intended model.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eIf the chosen ontology is SNOMED-CT where concepts can have multiple parents and multi-hierarchies [\u003cspan citationid=\"CR42\" class=\"CitationRef\"\u003e42\u003c/span\u003e] that often cause user uncertainty in the selection of concepts and a messy situation in their classification [\u003cspan citationid=\"CR43\" class=\"CitationRef\"\u003e43\u003c/span\u003e], then we check if this is the case for the concerned concept. If it has more than one hierarchy, then we identify the most relevant hierarchy or combine one or more hierarchies to build a new one. If not, we reuse the concept with its hierarchy.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eIf the chosen ontology is not SNOMED-CT, reuse the concept as it is represented in the ontology.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cdiv id=\"Sec21\" class=\"Section2\"\u003e \u003ch2\u003e3.5 Phase 5: Ontology evaluation\u003c/h2\u003e \u003cp\u003eOntology evaluation is a process of assessing the quality of an ontology involving a set of evaluation criteria and assessing its adequacy for being used in a specific context for a specific goal. It is also referred to as \u0026ldquo;quality assurance\u0026rdquo;, or \u0026ldquo;auditing\u0026rdquo; when it is conducted by a third party. Brank et al. [\u003cspan citationid=\"CR44\" class=\"CitationRef\"\u003e44\u003c/span\u003e] grouped existing evaluation approaches into four broad categories: 1) approaches that use the target ontology in an application and evaluate the application results [\u003cspan citationid=\"CR45\" class=\"CitationRef\"\u003e45\u003c/span\u003e]; 2) approaches that compare the target ontology to a gold standard [\u003cspan citationid=\"CR46\" class=\"CitationRef\"\u003e46\u003c/span\u003e]; 3) those that use data sources about a specific domain [\u003cspan citationid=\"CR47\" class=\"CitationRef\"\u003e47\u003c/span\u003e]; 4) approaches that recommend a manual assessment by domain experts according to a set of criteria [\u003cspan citationid=\"CR48\" class=\"CitationRef\"\u003e48\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThe evaluation process consists of several tasks dealing with the evaluation of different aspects of the ontology. The tasks can be divided into verification methods that examine the structure of the ontology (they answer if the ontology was built the right way), and validation methods that examine its applicability in the real-world (they answer if the right ontology was built) [\u003cspan citationid=\"CR49\" class=\"CitationRef\"\u003e49\u003c/span\u003e, \u003cspan citationid=\"CR50\" class=\"CitationRef\"\u003e50\u003c/span\u003e]. Let\u0026rsquo;s observe that Brank\u0026rsquo;s evaluation approaches correspond to validation methods. More specific evaluation criteria have been proposed by [\u003cspan citationid=\"CR50\" class=\"CitationRef\"\u003e50\u003c/span\u003e, \u003cspan citationid=\"CR51\" class=\"CitationRef\"\u003e51\u003c/span\u003e], see Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e. Most of them correspond to verification methods.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eOntology evaluation criteria.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"2\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCriteria\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eDefinition\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eDoes the asserted knowledge in the ontology agree with the expert\u0026rsquo;s knowledge, which is often measured in terms of precision and recall?\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCompleteness\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eIs the domain of interest appropriately covered?\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eConciseness\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eDoes the ontology include irrelevant or redundant axioms?\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eConsistency\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eDoes the ontology include or allow for contradictions?\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eComputational efficiency\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHow fast can the reasoners work with the ontology?\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAdaptability\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHow easy or difficult is it to use the ontology in different contexts?\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eClarity\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eDoes the ontology communicate effectively the intended meaning of the defined terms?\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eSeveral authors have suggested that automated tools are needed to ensure that high-quality ontologies are developed [\u003cspan citationid=\"CR52\" class=\"CitationRef\"\u003e52\u003c/span\u003e, \u003cspan citationid=\"CR53\" class=\"CitationRef\"\u003e53\u003c/span\u003e]. Authors in [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e] emphasized the lack of tools for ontology evaluation. Most of the tools discussed in the literature are prototypes or proposals. To our knowledge, OOPS!\u003csup\u003e16\u003c/sup\u003e, OAF\u003csup\u003e17\u003c/sup\u003e and OntoMetrics\u003csup\u003e18\u003c/sup\u003e are the only available evaluation tools that can be used currently.\u003c/p\u003e \u003cp\u003eTo evaluate each criterion listed in Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e, we suggest using additional data sources. The evaluation of some criteria can be in part automatized, and for some others, it requires the involvement of the knowledge engineer and domain experts. The following sections go through those criteria.\u003c/p\u003e \u003cdiv id=\"Sec22\" class=\"Section3\"\u003e \u003ch2\u003e3.5.1 Consistency\u003c/h2\u003e \u003cp\u003eInternal consistency relates to the adherence of the ontological model to the rules of the description logic. It can be checked using reasoners in PROT\u0026Eacute;G\u0026Eacute; and OOPS!\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec23\" class=\"Section3\"\u003e \u003ch2\u003e3.5.2 Accuracy and coverage (completeness)\u003c/h2\u003e \u003cp\u003eThe evaluation of accuracy and completeness usually relies on access to a gold standard built by domain experts. The building task is difficult to achieve for several reasons: difficulties to keep up with the published work in a given domain, variety of annotation standards, diversity of biomedical sources, high cost of annotation by medical experts [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eTo calculate precision and recall, two different approaches can be used. Precision is evaluated by domain experts that assess how well the ontology meets the set of predefined requirements. Laddering interviewing technique [\u003cspan citationid=\"CR54\" class=\"CitationRef\"\u003e54\u003c/span\u003e] and competency questions can be used. WebProt\u0026eacute;g\u0026eacute;\u003csup\u003e19\u003c/sup\u003e provides extensive collaboration features that allow sharing the ontology with domain experts. We recommend writing a document to explain the ontology, its objectives, and what is expected of the experts. This document should include questions using the laddering technique.\u003c/p\u003e \u003cp\u003eTo calculate the recall, we suggest building an evaluation corpus of knowledge composed of documents that substantially cover a given domain and include CPGs, systematic reviews, and patient health records. The goal of using CPGs is to verify that all the concepts identified in phase 3 are covered by ontology, including relationships.\u003c/p\u003e \u003cp\u003eWe manually annotate a number of relevant text fragments from the corpus. Then we link all concepts that concern the ontology domain to concepts in the ontology. This step allows us to find the ratio of the covered concepts.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec24\" class=\"Section3\"\u003e \u003ch2\u003e3.5.3 Clarity\u003c/h2\u003e \u003cp\u003eClarity concerns the meaning of the defined concepts and their independence of any social or computational context. It should be evaluated by domain experts. This step should be preceded by checking the readability of each concept, \u003cem\u003ei.e\u003c/em\u003e., the existence of human-readable descriptions such as labels, definitions, or synonyms. This can be done by using tools such as OntoMetrics and OOPS!\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec25\" class=\"Section3\"\u003e \u003ch2\u003e3.5.4 Conciseness\u003c/h2\u003e \u003cp\u003eUnnecessary concepts with regards to the domain to be covered can be reported by domain experts when evaluating. Redundancy is also evaluated at this stage. It occurs when an axiom can be inferred from already defined axioms. Two types of redundancies are defined in [\u003cspan citationid=\"CR49\" class=\"CitationRef\"\u003e49\u003c/span\u003e]: 1) concepts (classes, properties, or instances) having more than one subsumption relation between them; 2) identical formal definitions of concepts with possibly different labels. The two kinds of redundancy can be detected by reasoners and OOPS! [\u003cspan citationid=\"CR55\" class=\"CitationRef\"\u003e55\u003c/span\u003e].\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec26\" class=\"Section3\"\u003e \u003ch2\u003e3.5.5 Computational efficiency\u003c/h2\u003e \u003cp\u003eThe size of an ontology and the complexity of its axioms can slow down processing this ontology by tools. Computational efficiency measures how fast can the used tools, in particular reasoners, work with the ontology. Unfortunately, ontologies are often unnecessarily complex due to a lack of respect for modular structure. Waiting forever for a response and excessive space requirements make these ontologies unusable for the deployment process and feasible maintenance. Since the metrics of processing time and required space are difficult to evaluate, we suggest simple verifications using tools like Prot\u0026eacute;g\u0026eacute;, OOPS!, or OntoMetrics.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec27\" class=\"Section3\"\u003e \u003ch2\u003e3.5.6 Adaptability\u003c/h2\u003e \u003cp\u003eAdaptability measures: 1) how well the ontology anticipates its uses in different contexts; 2) whether it constitutes a reliable foundation for other ontologies; 3) whether it is flexible enough to react predictably to small changes and to allow extensions without any need to remove axioms. Adaptability is strongly related to the ontology modular design. \u0026ldquo;An ontology-module is a re-employable segment of a bigger or more tedious ontology, which is self-sufficient but holds relationships with the other modules of the ontology that also encompasses the initial non-modularized ontology\u0026rdquo; [\u003cspan citationid=\"CR56\" class=\"CitationRef\"\u003e56\u003c/span\u003e]. A modular design offers numerous benefits, such as simple reuse of single components in other ontologies, feasible reasoning performance, facilitating ontology understanding, reduced complexity, making the central ontology less vulnerable to changes, and others. The quality of modular design can be assessed using semantic metrics such as coupling and cohesion. Coupling refers to the interdependencies that exist between ontology modules, \u003cem\u003ei.e\u003c/em\u003e., to the number of shared symbols between axioms in different modules, while cohesion refers to the degree to which the elements in a module belong together. During the reuse process, the ontology engineer looks for an ontology with the maximal cohesion and minimal coupling. There have been many attempts to define a strategy selecting \u0026ldquo;the best modular ontology\u0026rdquo; [\u003cspan citationid=\"CR57\" class=\"CitationRef\"\u003e57\u003c/span\u003e]. Unfortunately, currently, no tools are supporting the creating of modular ontologies. A possible alternative can be a maximal reduction of the ontology scope; a similar approach has been applied to the development of the CORE subset of SNOMED CT.\u003c/p\u003e \u003cp\u003eAdaptability is also strongly related to the respect of the community standards. It includes usage of commonly accepted upper ontologies such as BFO, following OBO principles\u003csup\u003e20\u003c/sup\u003e, reusing relations from RO, specification of ontology version, and providing well-written documentation.\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv id=\"Sec28\" class=\"Section2\"\u003e \u003ch2\u003e3.6 Phase 6: Ontology documentation\u003c/h2\u003e \u003cp\u003eAn ontology must be documented for better understanding and using by future users on one hand, on the other hand, its maintenance and update by any knowledge engineer will be easier. The documentation process starts in the first phase when the domain and scope of the ontology are defined. At the end of each phase, a document detailing its progress and deliverables is established. The documentation of the ontology includes its classes, relations, properties, instances, axioms, and annotations. OWLDoc\u003csup\u003e21\u003c/sup\u003e and LODE\u003csup\u003e22\u003c/sup\u003e (Live OWL Documentation Environment) are tools that can be used to generate readable documentation presented to the user in the form of an HTML page with embedded links for easy navigation.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec29\" class=\"Section2\"\u003e \u003ch2\u003e3.7 Phase 7: Maintenance and evolution\u003c/h2\u003e \u003cp\u003eMedical knowledge is evolving and this evolution has to be reflected during the maintenance process. It includes integrating the missing knowledge and maintaining ontology consistency and coherence.\u003c/p\u003e \u003cp\u003eIf there are changes that occur in the ontology domain and scope definition (phase 1), all the phases of the ontology building should be carried out.\u003c/p\u003e \u003cp\u003eIf the updates concern only a few concepts in the ontology that need to be added, then follow the steps of adding concepts and resolving conflicts approach as presented in Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e.\u003c/p\u003e \u003cp\u003eIf the updates concern reused concepts that were modified in the reused ontologies, then reflect the modification in the concept if relevant. If not, keep the old version.\u003c/p\u003e \u003cp\u003eThe changes must be consistent with the domain and scope of the ontology. Once the changes are applied to the ontology, the ontology must be evaluated for consistency.\u003c/p\u003e \u003cp\u003eReused ontologies are usually updated by their authors. To the best of our knowledge, there are no ontology managers tracking the updates, and checking for the new versions has to be done manually.\u003c/p\u003e \u003c/div\u003e"},{"header":"4. Results","content":"\u003cp\u003eWe present in this section the results of applying the proposed methodology on building pneumonia diagnosis ontology (PNADO).\u003c/p\u003e \u003cdiv id=\"Sec31\" class=\"Section2\"\u003e \u003ch2\u003e4.1 Ontology domain and scope definition\u003c/h2\u003e \u003cp\u003eWe defined the ontology domain and scope after meetings with physicians from the Gatineau Hospital, who were interested in reducing erroneous pneumonia diagnoses. Functional requirements are defined by a set of CQs established by physicians and coming from the CPGs covering pneumonia diagnosis. The following questions, among others, have to be answered by physicians to diagnose pneumonia correctly:\u003c/p\u003e \u003cp\u003e \u003col\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eWhat are the symptoms and clinical signs of pneumonia?\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eWhat are the types of pneumonia?\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eHow is pneumonia diagnosed?\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eWhat are the pathogens of pneumonia?\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eWhat is the clinical history of the patient?\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eWhat are the laboratory tests to diagnose pneumonia?\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eWhat are the results of the physical examination of the patient?\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eWhat is the result of lung imaging of the patient?\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eWhat are the results of the lab tests of the patient?\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eOnce the diagnosis of pneumonia has been made, what complications should the physician look for?\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003c/ol\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec32\" class=\"Section2\"\u003e \u003ch2\u003e4.2 Corpus building and term extraction\u003c/h2\u003e \u003cp\u003eThe corpus of knowledge is constituted of 13 publicly available CPGs from national and international repositories that cover most of the pneumonia diagnosis knowledge. They are Cochrane\u003csup\u003e23\u003c/sup\u003e, NICE\u003csup\u003e24\u003c/sup\u003e, Infectious Diseases Society of America (IDSA\u003csup\u003e25\u003c/sup\u003e), European Society of Clinical Microbiology and Infectious Diseases (ESCMID\u003csup\u003e26\u003c/sup\u003e), Australian Society for Infectious Diseases (ASID\u003csup\u003e27\u003c/sup\u003e), Canadian Respiratory Guidelines (CTS\u003csup\u003e28\u003c/sup\u003e), Pulmonary \u0026amp; Critical Care Medicine (PulmCCM\u003csup\u003e29\u003c/sup\u003e), American Thoracic Society (ATS\u003csup\u003e30\u003c/sup\u003e), and British Thoracic Society (BTS\u003csup\u003e31\u003c/sup\u003e). The language of all the CPGs is English.\u003c/p\u003e \u003cp\u003eTo build our ontology, we select the terms that are relevant to pneumonia diagnosis. These terms include symptoms, clinical signs, imaging, laboratory tests and their results, pathogens (agents and others), antecedents, types of pneumonia, differential diagnosis, and complications. Then, in the set of found terms, we examine adjectives, adverbs, nouns, and proper nouns, verbs, and phrases because we believe that they convey the knowledge necessary to identify concepts in the pneumonia diagnosis domain.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec33\" class=\"Section2\"\u003e \u003ch2\u003e4.3 Building of preliminary ontology\u003c/h2\u003e \u003cdiv id=\"Sec34\" class=\"Section3\"\u003e \u003ch2\u003e4.3.1 Semantic normalization\u003c/h2\u003e \u003cp\u003eTo develop a differential ontology, we have to articulate the candidate terms selected previously by specifying the differential principles that characterize them. For example, bacterial pneumonia, fungal pneumonia, infective pneumonia acquired prenatally, pneumonia due to parasitic infestation, and viral pneumonia are sibling concepts because they have the similarity to be an infective disease. This step is the most time-consuming.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec35\" class=\"Section3\"\u003e \u003ch2\u003e4.3.2 Knowledge formalization\u003c/h2\u003e \u003cp\u003eThe following example illustrates adding an axiom. In the following text, ontological concepts are written in italic. \u003cem\u003eSign\u003c/em\u003e (also called \u003cem\u003eclinical sign\u003c/em\u003e) is a concept in PNADO. It is defined as the \u0026prime;\u0026prime;quality of a patient, a material entity that is part of a patient, or a processual entity that a patient participates in, any one of which is observed in a physical examination and is deemed by the clinician to be of clinical significance\u0026prime;\u0026prime;. \u003cem\u003eSymptom\u003c/em\u003e is another concept in PNADO; it is defined as \u0026prime;\u0026prime;quality of a patient that is observed by the patient or a processual entity experienced by the patient, either of which is hypothesized by the patient to be a realization of a disease\u0026prime;\u0026prime;. We add an axiom to indicate that the two concepts are disjoint.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec36\" class=\"Section3\"\u003e \u003ch2\u003e4.3.3 Towards a computational ontology\u003c/h2\u003e \u003cp\u003eDuring that process, using Prot\u0026eacute;g\u0026eacute;, we operationalize the ontology in the OWL language because it meets our needs in terms of expressiveness and manageability. We use the upper medical ontology OGMS. The choice of OGMS will be explained in the next phase. For further enrichment of PNADO, we extract definitions, synonyms, acronyms, and other annotations from UMLS by using MetaMap. At the end of this step, we obtain the preliminary PNADO.\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv id=\"Sec37\" class=\"Section2\"\u003e \u003ch2\u003e4.4 Ontology reuse and enrichment\u003c/h2\u003e \u003cdiv id=\"Sec38\" class=\"Section3\"\u003e \u003ch2\u003e4.4.1 Finding ontologies\u003c/h2\u003e \u003cp\u003eWe focus on two open content repositories of biomedical ontologies: Open Biomedical Ontology (OBO) Foundry and BioPortal. The English language and the OWL format are required. Using the search criteria, we found 26 candidate ontologies that could be reused.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec39\" class=\"Section3\"\u003e \u003ch2\u003e4.4.2 Choosing relevant ontologies\u003c/h2\u003e \u003cp\u003eWe identify ontologies to be reused by using accuracy and precision orderings. No precise ontology has been found.\u003c/p\u003e \u003cp\u003eChoosing between Basic Formal Ontology (BFO\u003csup\u003e32\u003c/sup\u003e) and General Formal Ontology (GFO\u003csup\u003e33\u003c/sup\u003e): both are designed to be upper ontologies. BFO has fewer superfluous and fewer omitted models than GFO, then it is more accurate. We reuse BFO as an upper ontology (hard reuse).\u003c/p\u003e \u003cp\u003eChoosing between Ontology of General Medical Science (OGMS\u003csup\u003e34\u003c/sup\u003e) and DIAGONT\u003csup\u003e35\u003c/sup\u003e: these ontologies cover the most important concepts for diagnostic. Both have superfluous and omitted models that are incomparable. Since OGMS is built upon BFO, we choose to reuse it (hard reuse). Hence, PNADO follows OGMS representation.\u003c/p\u003e \u003cp\u003eChoosing between SYMP\u003csup\u003e36\u003c/sup\u003e and Clinical Signs and Symptoms Ontology (CSSO\u003csup\u003e37\u003c/sup\u003e): SYMP has fewer superfluous and fewer omitted models than CSSO, it is more accurate. We reuse SYMP that covers mostly symptoms and clinical signs.\u003c/p\u003e \u003cp\u003eChoosing between SNOMED-CT\u003csup\u003e38\u003c/sup\u003e and NCBITaxon\u003csup\u003e39\u003c/sup\u003e: SNOMED-CT is the most widely used healthcare terminology and has the most comprehensive coverage of concepts for representing clinical knowledge [\u003cspan citationid=\"CR58\" class=\"CitationRef\"\u003e58\u003c/span\u003e, \u003cspan citationid=\"CR59\" class=\"CitationRef\"\u003e59\u003c/span\u003e]. SNOMED-CT is used as an ontology rather than a terminology, it is based on DL, and it can be reasoned over [\u003cspan citationid=\"CR60\" class=\"CitationRef\"\u003e60\u003c/span\u003e]. NCBITaxon is a classification and nomenclature of all the organisms in the public sequence databases. Both have superfluous and omitted models that are not comparable. Both are interesting to make soft reuse. SNOMED-CT covers other clinical aspects in addition to organisms that also can be reused.\u003c/p\u003e \u003cp\u003eChoosing between GAMUTS\u003csup\u003e40\u003c/sup\u003e and Radiological Lexicon (RadLex\u003csup\u003e41\u003c/sup\u003e): both are comprehensive terminologies for radiology and both have superfluous and omitted models but Radlex has fewer. We decide to reuse Radlex (soft reuse).\u003c/p\u003e \u003cp\u003eChoosing between Logical Observation Identifiers Names and Codes (LOINC\u003csup\u003e42\u003c/sup\u003e) for medical laboratory observations identification and clinical LABoratory Ontology (LABO\u003csup\u003e43\u003c/sup\u003e): both have superfluous and omitted models that are not comparable. We choose to reuse LOINC because it is the most used by clinicians.\u003c/p\u003e \u003cp\u003eChoosing between the International Classification of Diseases (ICD10\u003csup\u003e44\u003c/sup\u003e) and Human Disease Ontology (DOID\u003csup\u003e45\u003c/sup\u003e) that is a representation of human diseases organized by etiology: both have superfluous and omitted models that are not comparable. DOID provides more concepts related to PNADO than ICD10. We reuse DOID (soft reuse).\u003c/p\u003e \u003cp\u003eWe also reuse the following four ontologies dealing with important topics that are not covered by other ontologies chosen in the previous step: Relation Ontology (RO\u003csup\u003e46\u003c/sup\u003e) containing relations between entities; Computer-Based Patient Record Ontology (CPRO\u003csup\u003e47\u003c/sup\u003e) for patient profile description; Human Phenotype Ontology (HPO\u003csup\u003e48\u003c/sup\u003e) providing a structured and controlled vocabulary for the phenotypic features in human hereditary and other diseases; Infectious Disease Ontology (IDO\u003csup\u003e49\u003c/sup\u003e) covers the infectious disease domain.\u003c/p\u003e \u003cp\u003eThe reused ontologies use different naming conventions. In SNOMED-CT, the name of each concept begins with a capital letter whereas in SYMP it begins with a lowercase letter. In PNADO, names begin with a capital letter. In this paper, each concept is written according to the convention used by the ontology in question.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec40\" class=\"Section3\"\u003e \u003ch2\u003e4.4.3 Resolving conflicts\u003c/h2\u003e \u003cp\u003eHere we illustrate resolving concept conflicts with a few examples.\u003c/p\u003e \u003cp\u003eThe concept of \u003cem\u003ehypoxemia\u003c/em\u003e is found in HPO, SYMP, and SNOMED-CT. To choose the best representation of this concept, we proceed as follows:\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003eCRQ 1: \u0026ldquo;Does \u003cem\u003ehypoxemia\u003c/em\u003e have a definition in each ontology?\u0026rdquo; Answer: Only HPO provides a definition.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eCRQ 2: \u0026ldquo;Does its definition correspond to the PNADO requirements?\u0026rdquo; Answer: the definition given in HPO is relevant to PNADO.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eCRQ 3: \u0026ldquo;Does \u003cem\u003ehypoxemia\u003c/em\u003e have children?\u0026rdquo; Answer: \u003cem\u003eHypoxemia\u003c/em\u003e has children in HPO and SNOMED-CT but none in SYMP.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eCRQ 4: \u0026ldquo;Is each found child relevant to the PNADO requirements?\u0026rdquo; Answer: \u003cem\u003eHypoxemia\u003c/em\u003e\u0026rsquo;s children found in HPO are relevant to the PNADO requirements.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eAnalyze: \u003cem\u003eHypoxemia\u003c/em\u003e presented in HPO seems to have the best representation for PNADO since it provides a relevant definition and relevant children that correspond to the PNADO requirements.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003cp\u003eThe concept of \u003cem\u003epatient\u003c/em\u003e is represented differently in CPRO and SNOMED-CT. We use the following CRQs to choose the most relevant representation for this concept:\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003eCRQ 1: \u0026ldquo;Does \u003cem\u003epatient\u003c/em\u003e have a definition in each ontology?\u0026rdquo; Answer: No ontology provides a definition.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eCRQ 2: \u0026ldquo;What is the parent concept of \u003cem\u003epatient\u003c/em\u003e?\u0026rdquo; Answer: In CPRO, the parent concept is \u003cem\u003eorganism\u003c/em\u003e that is provided by OGMS. In SNOMED-CT, the parent is \u003cem\u003esocial context\u003c/em\u003e that does not fit the PNADO requirements.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eCRQ 3: \u0026ldquo;Is any person a patient?\u0026rdquo; Answer in CPRO is the axiom \u0026ldquo;\u003cem\u003eHuman/Person\u003c/em\u003e \u003cb\u003eand\u003c/b\u003e (\u003cem\u003e'Plays Role'\u003c/em\u003e \u003cb\u003esome\u003c/b\u003e \u003cem\u003ePatient role\u003c/em\u003e) \u003cb\u003eand\u003c/b\u003e (\u003cem\u003e'Participates_in'\u003c/em\u003e \u003cb\u003esome\u003c/b\u003e \u003cem\u003eClinical act\u003c/em\u003e)\u0026rdquo;. No answer is found in SNOMED-CT.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eAnalyze: CPRO provides a better representation for \u003cem\u003epatient\u003c/em\u003e than SNOMED-CT. It specifies with an axiom in which case a \u003cem\u003eperson\u003c/em\u003e can be a \u003cem\u003epatient\u003c/em\u003e and it is a subclass of \u003cem\u003eorganism\u003c/em\u003e defined by OGMS.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003cp\u003eTwo other examples of resolving conflicts are presented in Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eExamples of inconsistencies resolved with CRQs.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eConcept\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eOntologies\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCRQs\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eCRQs responses\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"4\" rowspan=\"5\"\u003e \u003cp\u003e\u003cem\u003eHypotension\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\" morerows=\"4\" rowspan=\"5\"\u003e \u003cp\u003eHPO\u003c/p\u003e \u003cp\u003eSNOMED-CT SYMP\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eIs \u003cem\u003ehypotension\u003c/em\u003e\u0026nbsp;a \u003cem\u003esymptom\u003c/em\u003e or a \u003cem\u003evital sign\u003c/em\u003e?\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e- HPO: \u003cem\u003ehypotension\u003c/em\u003e is a phenotypic abnormality.\u003c/p\u003e \u003cp\u003e-\u0026nbsp;SNOMED-CT: \u003cem\u003ehypotension\u003c/em\u003e (\u003cem\u003elow blood pressure\u003c/em\u003e) is considered a \u003cem\u003edisorder of the cardiovascular system\u003c/em\u003e.\u003c/p\u003e\u003cp\u003e- SYMP: \u003cem\u003ehypotension\u003c/em\u003e is considered a \u003cem\u003ehemic system symptom\u003c/em\u003e.\u003c/p\u003e\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eWhat is the synonym of \u003cem\u003ehypotension\u003c/em\u003e?\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e-\u0026nbsp;\u003cem\u003eHypotension\u003c/em\u003e has two synonyms in HPO, three synonyms in SNOMED-CT, and no synonym in SYMP.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAre there any \u003cem\u003ehypotension\u003c/em\u003e's children?\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e- \u003cem\u003eHypotension\u003c/em\u003e has three children in HPO, twelve children in SNOMED-CT, and one child in SYMP.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eIs each child relevant to the diagnosis of pneumonia?\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e-\u0026nbsp;Only \u003cem\u003ehypotension\u003c/em\u003e's children provided by HPO and SYMP are relevant for the diagnosis of pneumonia.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c4\" namest=\"c3\"\u003e \u003cp\u003e\u003cb\u003eAnalyze\u003c/b\u003e: \u003cem\u003eblood pressure\u003c/em\u003e is a measurement of the pressure in arteries. We consider it in PNADO as a \u003cem\u003evital sign\u003c/em\u003e. \u003cem\u003eHypotension\u003c/em\u003e's children in HPO combined with SYMP's child correspond to PNADO intended model. We reuse \u003cem\u003ehypotension\u003c/em\u003e's synonyms of SNOMED-CT.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\" morerows=\"5\" rowspan=\"6\"\u003e \u003cp\u003e\u003cem\u003eRespiratory rates\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\" morerows=\"5\" rowspan=\"6\"\u003e \u003cp\u003eSNOMED-CT\u003c/p\u003e \u003cp\u003eLOINC\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eIs \u003cem\u003erespiratory rate\u003c/em\u003e a \u003cem\u003esymptom\u003c/em\u003e or a \u003cem\u003evital sign\u003c/em\u003e?\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e-\u0026nbsp;In both SNOMED-CT and LOINC, it is mentioned that the scale type of \u003cem\u003erespiratory rate\u003c/em\u003e is quantitative.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAre there any \u003cem\u003erespiratory rate\u003c/em\u003e's children?\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e-\u0026nbsp;SNOMED-CT: \u003cem\u003ethe respiratory rate\u003c/em\u003e has two children.\u003c/p\u003e \u003cp\u003e-\u0026nbsp;LOINC: the \u003cem\u003erespiratory rate\u003c/em\u003e has no children.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eIs each child relevant to the diagnosis of pneumonia?\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e-\u0026nbsp;Yes, \u003cem\u003ethe respiratory rate\u003c/em\u003e's children provided by SNOMED-CT are relevant for the diagnosis of pneumonia.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eIs there any measurement unit?\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e-\u0026nbsp;No measurement for \u003cem\u003erespiratory rate\u003c/em\u003e is given in SNOMED-CT.\u003c/p\u003e \u003cp\u003e-\u0026nbsp;LOINC provides \"{breaths}/min\" as an example of the \u003cem\u003erespiratory rate\u003c/em\u003e measurement unit.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAre there any synonyms?\u003c/p\u003e \u003cp\u003eAre there any related names?\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e-\u0026nbsp;SNOMED-CT provides two synonyms and no related names.\u003c/p\u003e \u003cp\u003e-\u0026nbsp;LOINC doesn't provide any synonym for \u003cem\u003erespiratory rate\u003c/em\u003e but provides 13 related names.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"2\" nameend=\"c4\" namest=\"c3\"\u003e \u003cp\u003e\u003cb\u003eAnalyze\u003c/b\u003e: The \u003cem\u003erespiratory rate\u003c/em\u003e is represented as a \u003cem\u003evital sign\u003c/em\u003e in PNADO. We reuse SNOMED-CT's children because they respond to the PNADO scope and its intended model. We reuse the unit measurement of LOINC and some of the related names. We also reuse the synonyms of SNOMED-CT.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv id=\"Sec41\" class=\"Section2\"\u003e \u003ch2\u003e4.5 Ontology evaluation\u003c/h2\u003e \u003cp\u003eFor the evaluation of PNADO, we used tools, a clinical dataset that contains multiple cases of pneumonia, and a clinical corpus of knowledge captured in the CPGs and systematic reviews. Domain experts were also involved in the process.\u003c/p\u003e \u003cdiv id=\"Sec42\" class=\"Section3\"\u003e \u003ch2\u003e4.5.1 Consistency\u003c/h2\u003e \u003cp\u003eWe used the Pellet reasoner integrated with PROT\u0026Eacute;G\u0026Eacute; and the tool OOPS! to check the internal consistency of PNADO. OOPS! reported minor pitfalls regarding annotations. The missing license for PNADO was reported as an important pitfall. No critical pitfalls were reported.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec43\" class=\"Section3\"\u003e \u003ch2\u003e4.5.2 Accuracy and coverage (completeness)\u003c/h2\u003e \u003cp\u003eTo calculate recall, we constructed an evaluation knowledge corpus constituted of 1) the 13 CPGs chosen in section \u003cspan refid=\"Sec32\" class=\"InternalRef\"\u003e4.2\u003c/span\u003e; \u003cspan refid=\"Sec2\" class=\"InternalRef\"\u003e2\u003c/span\u003e) 43 systematic reviews that treat pneumonia diagnosis from national and international repositories containing evidence-based medical documents; and 3) the clinical dataset MIMIC-III\u003csup\u003e50\u003c/sup\u003e (Multi-parameter Intelligent Monitoring in Intensive Care III) [\u003cspan citationid=\"CR61\" class=\"CitationRef\"\u003e61\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eFirst, we treated the CPGs and systematic reviews. We manually extracted 710 different terms related to pneumonia diagnosis from 336 text fragments (each fragment had twenty lines on average) and tried to link them to concepts in PNADO. Linking extracted concepts to concepts in PNADO, we found a match for 570 concepts giving a value of recall of 80%. The remaining 140 concepts that could not be matched were added to PNADO.\u003c/p\u003e \u003cp\u003eHere is an example (see Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e) of annotation of a fragment of text from the clinical guideline \u0026ldquo;Management of Community-Acquired Pneumonia in Adults\u0026rdquo; [\u003cspan citationid=\"CR62\" class=\"CitationRef\"\u003e62\u003c/span\u003e].\u003cdiv class=\"BlockQuote\"\u003e\u003cp\u003eA \u003cb\u003echest radiograph\u003c/b\u003e is required for the routine evaluation of \u003cb\u003epatients\u003c/b\u003e who are \u003cb\u003elikely to have pneumonia\u003c/b\u003e, to establish the \u003cb\u003ediagnosis\u003c/b\u003e and to aid in \u003cb\u003edifferentiating CAP\u003c/b\u003e from other \u003cb\u003ecommon causes\u003c/b\u003e of \u003cb\u003ecough\u003c/b\u003e and \u003cb\u003efever\u003c/b\u003e, such as \u003cb\u003eacute bronchitis\u003c/b\u003e.\u003c/p\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eExample of annotation of clinical guideline paragraph with PNADO.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eConcepts from the text\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"4\" nameend=\"c5\" namest=\"c2\"\u003e \u003cp\u003eEquivalences in the PNADO\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eClass\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eObject property\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eID\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eParent entity in PNADO\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003echest radiograph\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cem\u003eChest radiography\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003ePNADO:0000783\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cem\u003eImaging of lung\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003epatients likely to have pneumonia\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003eHas a diagnosis (patients, pneumonia)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003ePNADO:0001291\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ediagnosis\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cem\u003ediagnosis\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eOGMS:0000073\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cem\u003edata item\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003edifferentiating CAP from\u0026hellip;acute bronchitis\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003eDifferential diagnosis of(CAP, Acute bronchitis)\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003ePNADO:0001157\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCAP\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cem\u003eCommunity-acquired pneumonia\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eSNOMEDCT_US:385093006\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cem\u003ePneumonia\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ecough\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cem\u003eCough\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eSNOMEDCT_US:49727002\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cem\u003eRespiratory system and chest symptom\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003efever\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u003cem\u003efever\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eSYMP:0000613\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u003cem\u003eneurological and physiological symptom\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eSecond, we treated records from MIMIC III that consists of 38,597 distinct and de-identified adult patients and 49,785 hospital admissions. It includes, among others, laboratory data, radiology reports, vital signs, therapeutic intervention profiles, ventilator settings, nursing notes, diagnostic codes, discharge summaries, and provider order entry data. It comprises 26 tables linked by identifiers such as HADM_ID referring to unique hospital admission and SUBJECT_ID referring to a unique patient. Among those tables, we considered the \u0026lsquo;diagnoses_icd\u0026rsquo; table that contains ICD-9 diagnoses for patients and the \u0026lsquo;note events\u0026rsquo; table that contains all notes about patients from their admissions to their discharges, including nursing and physician notes, ECG reports, radiology reports, and discharge summaries.\u003c/p\u003e \u003cp\u003eWe found 7702 cases of pneumonia diagnosis in the \u0026lsquo;diagnoses_icd\u0026rsquo; table. As shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e, there were 32 types of pneumonia. We were interested in making sure that PNADO covered all the cases captured in MIMIC-III. The number of pneumonia cases with code 860 was more than half of all cases. Consequently, we used the logarithm function to reduce the number of cases to include in the evaluation for each type of pneumonia, 43 cases in total.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eThe next task was the extraction of the records from the \u0026prime;noteevents\u0026prime; table by using HADM_ID already obtained in the previous task. We randomly selected cases according to the cardinality of each type and completed the annotation. We manually annotated terms related to pneumonia diagnosis and verified whether they were covered by PNADO or not. For those not covered, we analyzed if they were synonyms for the existing concepts or they were new concepts.\u003c/p\u003e \u003cp\u003eAt the end of the evaluation of PNADO with the MIMIC III database, 36 concepts of 988 concepts found in the electronic health records were not covered by PNADO. The analysis of each concept revealed that 16 concepts were synonyms and 20 concepts were subclasses of the existing concepts. Here are some examples of synonyms: pulmonary embolus, rhonchi sound, rhonchus sound, pneumococci, pneumonic infiltrates, MSSA PNA, MSSA pneumonia, MRSA PNA, MRSA pneumonia, and trouble breathing. The new subclasses include types of pneumonia such as \u003cem\u003epolymicrobial pneumonia\u003c/em\u003e; physical examination findings such as \u003cem\u003erespiratory failure;\u003c/em\u003e symptoms such as \u003cem\u003erecurrent cough\u003c/em\u003e and \u003cem\u003eoccasional cough\u003c/em\u003e; and clinical histories such as \u003cem\u003etracheostomy\u003c/em\u003e. The recall ratio was 96%.\u003c/p\u003e \u003cp\u003eTo calculate the precision of PNADO, we involved a pneumologist from Charles-Le Moyne Hospital\u003csup\u003e51\u003c/sup\u003e, an emergency physician from the Gatineau Hospital\u003csup\u003e52\u003c/sup\u003e, and a family doctor from the University of Quebec in Outaouais medical clinic\u003csup\u003e53\u003c/sup\u003e. We prepared a document to explain the ontology and its objective. We also prepared a document containing the CQs used for the determination of the domain and scope of PNADO (section \u003cspan refid=\"Sec31\" class=\"InternalRef\"\u003e4.1\u003c/span\u003e Ontology domain and scope definition, and questions using the laddering technique [\u003cspan citationid=\"CR54\" class=\"CitationRef\"\u003e54\u003c/span\u003e]. Here are a few examples of these questions: What are the symptoms of pneumonia? Is the classification of symptoms according to the different systems (digestive system, cardiovascular system, etc.) correct? Is each symptom well classified?\u003c/p\u003e \u003cp\u003eThe participating physicians created accounts in WebProt\u0026eacute;g\u0026eacute;, and we shared PNADO with them. Once they had familiarized themselves with the environment, they evaluated each concept and commented directly in WebProt\u0026eacute;g\u0026eacute;. They verified the relevance of each concept in PNADO to pneumonia diagnosis, its position, and its relations with the other concepts. For each concept, they provided a mention of evaluation (relevance and suggestions) in WebProt\u0026eacute;g\u0026eacute;. They suggested several changes including adding radiological signs of \u003cem\u003econsolidation\u003c/em\u003e: \u003cem\u003eair bronchograms\u003c/em\u003e, \u003cem\u003eill-defined\u003c/em\u003e, \u003cem\u003efluffy opacities\u003c/em\u003e, \u003cem\u003eair alveologram\u003c/em\u003e, \u003cem\u003epatchy opacities\u003c/em\u003e, \u003cem\u003eacinar\u003c/em\u003e, \u003cem\u003epreserved lung volume\u003c/em\u003e, \u003cem\u003eextension to pleural surface\u003c/em\u003e, \u003cem\u003eCT angiogram sign\u003c/em\u003e, \u003cem\u003esilhouette sign\u003c/em\u003e; moving concepts of \u003cem\u003efever\u003c/em\u003e to \u003cem\u003egeneral symptom, purulent tracheobronchial secretions\u003c/em\u003e to \u003cem\u003erespiratory system and chest symptom\u003c/em\u003e, \u003cem\u003eheadache\u003c/em\u003e to \u003cem\u003eneurological and physical symptom\u003c/em\u003e, \u003cem\u003earterial blood gas\u003c/em\u003e to \u003cem\u003eprocedure\u003c/em\u003e; and removing the concepts \u003cem\u003etransitory tachypnea of the newborn, sputum eosinophilia\u003c/em\u003e and \u003cem\u003elithoptysis.\u003c/em\u003e The precision ratio was 96%.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec44\" class=\"Section3\"\u003e \u003ch2\u003e4.5.3 Clarity\u003c/h2\u003e \u003cp\u003eWe used OntoMetrics and OOPS! tools to evaluate readability. OntoMetrics was not able to check the readability of all the concepts. OOPS! detected that some of the new concepts added to PNADO did not have definitions. In fact, these concepts have no definitions in UMLS. 663 concepts without definition (mostly UMLS concepts) were reported.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec45\" class=\"Section3\"\u003e \u003ch2\u003e4.5.4 Conciseness\u003c/h2\u003e \u003cp\u003eOOPS! and Pellet reasoner were used to detecting redundancies in PNADO. OOPS! reported two classes that contain the same labels. We found that the two classes have labels that only differ by one character (\u003cem\u003elegionella pneumophila serogroup 1\u003c/em\u003e, \u003cem\u003elegionella pneumophila serogroup 2\u003c/em\u003e).\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec46\" class=\"Section3\"\u003e \u003ch2\u003e4.5.5 Computational efficiency\u003c/h2\u003e \u003cp\u003eTime processing of PNADO using Pellet reasoner, OOPS! and OntoMetrics was short (1\u0026ndash;2 sec) with one exception: OntoMetrics was not able to check the criterion of readability when providing the entire ontology.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec47\" class=\"Section3\"\u003e \u003ch2\u003e4.5.6 Adaptability\u003c/h2\u003e \u003cp\u003ePNADO uses the BFO upper ontology that enhances reusability and interoperability. It also follows OBO Foundry principles and reuses relations from RO. We specify the PNADO version and provide documentation (see section \u003cspan refid=\"Sec47\" class=\"InternalRef\"\u003e4.5.6\u003c/span\u003e Adaptability). PNADO does not have a modular structure.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec48\" class=\"Section3\"\u003e \u003ch2\u003e4.5.7 Characteristics of PNADO\u003c/h2\u003e \u003cp\u003ePNADO is published in BioPortal repository and can be viewed and explored at \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://bioportal.bioontology.org/ontologies/PNADO\u003c/span\u003e\u003cspan address=\"https://bioportal.bioontology.org/ontologies/PNADO\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/p\u003e \u003cp\u003eWe present in this section the results obtained by applying the proposed methodology to the PNADO building.\u003c/p\u003e \u003cp\u003ePNADO contains 1598 classes (1448 are reused and 150 are new classes), 42 object properties (27 are reused and 15 are new), 1591 logical axioms, and 83 annotation properties. Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e presents statistics on reused concepts and resolved conflicts. We noticed that SNOMED-CT was frequently involved in conflict resolution, as it overlaps with most existing medical ontologies. PNADO is built according to the principles of OBO Foundry, using OGMS that follows the BFO paradigm. OGMS provides a set of general reference classes related to diseases and diagnoses. PNADO is only focused on diagnosis. Therefore, only the OGMS concepts related to diagnosis are used. Other high-level OGMS terms may be used in future extensions.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab4\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 4\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eOntology reuse in PNADO.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"7\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colspan=\"2\" nameend=\"c2\" namest=\"c1\"\u003e \u003cp\u003eOntology\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eUsage in PNADO\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eClasses\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eRelations\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eTotal\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eConflicts\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003ctr\u003e \u003cth align=\"left\" colspan=\"7\" nameend=\"c7\" namest=\"c1\"\u003e \u003cp\u003eHard reuse\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eBFO\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eUpper ontology\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e35\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e40\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eOGMS\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eUpper domain ontology\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e149\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e149\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"7\" nameend=\"c7\" namest=\"c1\"\u003e \u003cp\u003e\u003cb\u003eSoft reuse\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSYMP\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eSymptoms\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e57\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e57\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e22\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eNCBITAXON\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eVirus and bacteria\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e184\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e184\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eRO\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eRelations\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e22\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e22\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eCPRO\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eRoles\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e2\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eLOINC\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eLaboratory\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e26\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e26\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSNOMED-CT\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eDiseases, symptoms, and clinical signs\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e796\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e796\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e115\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eRADLEX\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eImaging\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e55\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e55\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eDOID\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eDisease\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e35\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e35\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e13\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHPO\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003ePhenotype\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e96\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e96\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e73\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e10\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eIDO\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003ePathogens\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e11\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e11\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eFigure \u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003e shows an example from PNADO demonstrating the hierarchy of the class \u003cem\u003einfective pneumonia\u003c/em\u003e and, in particular, the class \u003cem\u003eviral pneumonia.\u003c/em\u003e Figure\u0026nbsp;\u003cspan refid=\"Fig6\" class=\"InternalRef\"\u003e6\u003c/span\u003e shows the representation of the \u003cem\u003ehealthcare process assay\u003c/em\u003e class.\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv id=\"Sec49\" class=\"Section2\"\u003e \u003ch2\u003e4.6 Ontology documentation\u003c/h2\u003e \u003cp\u003eWe annotated every concept in PNADO and used OWLDoc to document the ontology. Every phase of the building process is described in this work.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec50\" class=\"Section2\"\u003e \u003ch2\u003e4.7 Maintenance and evolution\u003c/h2\u003e \u003cp\u003eWithin the COVID pandemic context, two CPGs treating pneumonia diagnosis were found in NICE and CEBM\u003csup\u003e54\u003c/sup\u003e (the Centre for Evidence-Based Medicine) repositories. We extracted 16 new concepts relevant to PNADO. We also frequently check the parts of the reused ontologies. Recently, OGMS has been subject to changes that affect PNADO significantly. For example, \u003cem\u003esymptom\u003c/em\u003e was moved to \u003cem\u003eprocess\u003c/em\u003e class, \u003cem\u003evital sign\u003c/em\u003e was moved to \u003cem\u003ematerial entity\u003c/em\u003e class, \u003cem\u003esign\u003c/em\u003e class was deleted, etc.\u003c/p\u003e \u003c/div\u003e"},{"header":"5. Discussion","content":"\u003cdiv id=\"Sec52\" class=\"Section2\"\u003e \u003ch2\u003e5.1 Generality of the methodology\u003c/h2\u003e \u003cp\u003eThe techniques proposed in each phase are independent of any specific domain and can therefore be used for any domain of any field. However, the corpus of knowledge must be tailored to the targeted ontology domain in order to apply the methodology. The same goes for the ontologies to reuse, they must be related to the same field. The use of codified knowledge is very useful for a knowledge engineer, he can easily understand the domain and there are fewer chances of getting confused.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec53\" class=\"Section2\"\u003e \u003ch2\u003e5.2 Ontology reuse\u003c/h2\u003e \u003cp\u003eOntology reuse in the methodology is used to enrich the preliminary ontology with new concepts and annotations, and to enhance ontology interoperability and reusability. Since the methodology is designed for a knowledge engineer and the involvement of the domain experts is limited, ontology reuse cannot be performed at the beginning of the building process because the knowledge engineer does not know which concepts are relevant to the ontology domain.\u003c/p\u003e \u003cp\u003eOntology reuse is the most tricking phase in the methodology due to the interoperability issues among the reused ontologies. Performing this phase during the PNADO building revealed some findings:\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003eChoosing relevant ontologies to reuse is very important and determinant for the rest of the reuse process. Some candidate ontologies can be easily excluded while others require careful analysis.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eEven if the chosen ontology is of good quality, the representation of a concept may not be relevant to the requirements of the ontology that is being built. We also found out during the PNADO building that some ontologies have bad representations for some concepts. For example, SYMP represents diseases as symptoms. An example of a relation found in RO that is not relevant for PNADO is discussed in section \u003cspan refid=\"Sec54\" class=\"InternalRef\"\u003e5.3\u003c/span\u003e.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eEach ontology is chosen according to a significant topic, as discussed in section \u003cspan refid=\"Sec14\" class=\"InternalRef\"\u003e3.4.2\u003c/span\u003e, but in the process of reuse, the chosen ontology may contain concepts that deal with other topics. For example, DOID was chosen for the topic of diseases but it was involved in conflicts regarding concepts related to symptoms and clinical signs.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eA concept can be differently presented in different ontologies and to choose the best representation to reuse requires reflection about the relevant CRQs.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec54\" class=\"Section2\"\u003e \u003ch2\u003e5.3 Upper domain ontologies\u003c/h2\u003e \u003cp\u003eAs mentioned in section \u003cspan refid=\"Sec53\" class=\"InternalRef\"\u003e5.2\u003c/span\u003e, there are ontologies designed to cover a specific domain which also cover concepts belonging to other domains. These ontologies do not respect the principles of modular design [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. The unwanted concepts should be usually covered by upper domain ontologies. We notice that it is often useful to design several levels of upper domain ontologies to get a better module coupling and cohesion, and at the same time, to reduce the reuse and interoperability issues.\u003c/p\u003e \u003cp\u003eOGMS, which covers diagnosis and treatment of disease, was used as an upper domain ontology for PNADO. Many concepts (classes and relations) required for the diagnostic process are missing in OGMS. PNADO aims to cover the concepts related to pneumonia diagnosis and the concepts related to the diagnostic process should be available in an upper domain ontology.\u003c/p\u003e \u003cp\u003eA different example would be an upper domain ontology describing patient. Currently, several domain ontologies differently model the concept of patient. Having one upper ontology representing patient would be very useful for reuse and enhancing interoperability.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec55\" class=\"Section2\"\u003e \u003ch2\u003e5.4 PNADO new concepts\u003c/h2\u003e \u003cp\u003eIn the following paragraphs, we discuss some of the reused object properties and classes that required major modifications.\u003c/p\u003e \u003cp\u003e \u003cb\u003eComplication of\u003c/b\u003e \u003c/p\u003e \u003cp\u003eA complication in medicine is a medical problem that occurs during or after a disease, procedure, treatment, or function (pregnancy, for example). This concept is represented as a class in many ontologies such as SNOMED-CT or Radlex. In SNOMED-CT, the \u003cem\u003eComplication\u003c/em\u003e is a subclass of \u003cem\u003eDisease\u003c/em\u003e and represents complications of disease such as \u003cem\u003eComplications due to diabetes mellitus\u003c/em\u003e, or \u003cem\u003eComplication due to Crohn's disease. Diabetes mellitus\u003c/em\u003e and \u003cem\u003eCrohn\u0026rsquo;s disease\u003c/em\u003e are subclasses of \u003cem\u003edisease.\u003c/em\u003e The diseases concerned by complications are also represented as subclasses of the class \u003cem\u003edisease\u003c/em\u003e. Such a representation creates a heritage chain \u003cem\u003edisease \u0026rarr; complication \u0026rarr; disease\u003c/em\u003e and does not cover all cases of complications due to \u003cem\u003edisease\u003c/em\u003e, \u003cem\u003eprocedure\u003c/em\u003e, \u003cem\u003etreatment\u003c/em\u003e, or \u003cem\u003efunction\u003c/em\u003e. We propose to represent complication as an object property named \u003cem\u003ecomplication of\u003c/em\u003e with four sub-properties: 1) \u003cem\u003ecomplication of disease\u003c/em\u003e that associates \u003cem\u003edisease\u003c/em\u003e to \u003cem\u003edisease\u003c/em\u003e; 2) \u003cem\u003ecomplication of procedure\u003c/em\u003e that associates \u003cem\u003eprocedure\u003c/em\u003e to \u003cem\u003edisease\u003c/em\u003e; 3) \u003cem\u003ecomplication of treatment\u003c/em\u003e that associates \u003cem\u003etreatment\u003c/em\u003e to \u003cem\u003edisease, and\u003c/em\u003e 4) \u003cem\u003ecomplication of function\u003c/em\u003e that associates \u003cem\u003efunction\u003c/em\u003e to \u003cem\u003edisease.\u003c/em\u003e Axioms are progressively added when needed to represent complications. For example, in PNADO, axioms are added to illustrate pneumonia complications. This way of representing \u003cem\u003ecomplication of\u003c/em\u003e enhances the interoperability and reusability.\u003c/p\u003e \u003cp\u003e \u003cb\u003eDifferential diagnosis of\u003c/b\u003e \u003c/p\u003e \u003cp\u003eDifferential diagnosis is the distinguishing of disease from others that present similar clinical features (signs and symptoms). It is represented in SNOMED-CT and LOINC as a class. We notice that there is no representation of pneumonia differential diagnosis in both ontologies. The presence of differential diagnosis could help to avoid diagnosis errors. We propose a new object property \u003cem\u003eDifferential diagnosis of\u003c/em\u003e that holds between a \u003cem\u003ediagnosis\u003c/em\u003e \u0026ldquo;A\u0026rdquo; and a \u003cem\u003ediagnosis\u003c/em\u003e \u0026ldquo;B\u0026rdquo; if they have some similar clinical features. Recall that a \u003cem\u003ediagnosis\u003c/em\u003e is the representation of a conclusion of a diagnostic process that can be a disease or a syndrome. Axioms are added to define the relation. An example of such an axiom: \u003cem\u003epneumonia\u003c/em\u003e \u0026prime;\u003cem\u003eDifferential diagnosis of\u0026prime;\u003c/em\u003e \u003cb\u003esome\u003c/b\u003e \u003cem\u003ebronchitis\u003c/em\u003e.\u003c/p\u003e \u003cp\u003e \u003cb\u003eHas symptom\u003c/b\u003e \u003c/p\u003e \u003cp\u003e \u003cem\u003eHas symptom\u003c/em\u003e is an object property of RO where it is defined as \"a relation that holds between a disease or an organism and a phenotype\". We notice that it is a sub-property of the object property that \u003cem\u003ehas phenotype.\u003c/em\u003e The latter associates the domains \u003cem\u003egenerically dependent continuant\u003c/em\u003e or \u003cem\u003ematerial anatomical entity\u003c/em\u003e or \u003cem\u003edisease\u003c/em\u003e to the range \u003cem\u003ephenotype.\u003c/em\u003e The class \u003cem\u003ephenotype\u003c/em\u003e, which is reused by RO from Combined Phenotype Ontology (UPHENO), is defined \"as a defect or loss of some anatomical structure or a biological process to wild-type\". The class \u003cem\u003ephenotype\u003c/em\u003e in OGMS and reused in PNADO is defined as \"a (combination of) quality(ies) of an organism determined by the interaction of its genetic make-up and environment that differentiates specific instances of a species from other instances of the same species\". It is obvious that the two definitions are different, the one provided by UPHENO does not correspond to PNADO intended model, and subsequently, we do not reuse an object property associating this class to any other class. We create a new object property \u003cem\u003eHas a symptom\u003c/em\u003e that associates \u003cem\u003ePatient\u003c/em\u003e to \u003cem\u003esymptom\u003c/em\u003e.\u003c/p\u003e \u003cp\u003e \u003cb\u003esymptom\u003c/b\u003e \u003c/p\u003e \u003cp\u003e \u003cem\u003esymptom\u003c/em\u003e is defined in OGMS as \"a quality of a patient that is observed by the patient or a processual entity experienced by the patient, either of which is hypothesized by the patient to be a realization of a disease\". It is also defined in SYMP as \"a perceived change in function, sensation, loss, disturbance or appearance reported by a patient indicative of a disease\". Nevertheless, during the reuse of SYMP, we noticed that it also contains diseases and clinical signs when it was intended to contain only symptoms. For instance, \u003cem\u003ehypotension\u003c/em\u003e, \u003cem\u003earrhythmia\u003c/em\u003e, \u003cem\u003ebronchitis\u003c/em\u003e, \u003cem\u003eendocarditis\u003c/em\u003e, and \u003cem\u003eshock\u003c/em\u003e are defined in SYMP as symptoms, whereas they are diseases. Indeed, there is often confusion between clinical signs, diseases, and symptoms in SYMP and other ontologies. In PNADO, the class \u003cem\u003esymptom\u003c/em\u003e, thanks to the ontological commitment, includes only what is reported by a patient.\u003c/p\u003e \u003cp\u003e \u003cb\u003ePathogen\u003c/b\u003e \u003c/p\u003e \u003cp\u003eBefore discussing \u003cem\u003ePathogen\u003c/em\u003e, we will briefly review the \u003cem\u003eOrganism\u003c/em\u003e since both concepts are connected.\u003cem\u003eOrganism\u003c/em\u003e is an interesting concept needed in PNADO, and it can be found in other ontologies such as CPRO, IDO, or SNOMED-CT. In SNOMED-CT, it is represented as an independent entity that includes \u003cem\u003ebacteria\u003c/em\u003e, \u003cem\u003evirus\u003c/em\u003e, \u003cem\u003earchaea\u003c/em\u003e, \u003cem\u003eeukarya\u003c/em\u003e, and \u003cem\u003eprion\u003c/em\u003e. Since no definition is given to this concept, its meaning (not precise) can only be deduced from its sub-classes. CPRO represents \u003cem\u003eorganism\u003c/em\u003e as a subclass of \u003cem\u003eobject\u003c/em\u003e whereas IDO represents it as a sibling to \u003cem\u003eobject\u003c/em\u003e. An \u003cem\u003eobject\u003c/em\u003e according to the definition given by BFO \"is a material entity that is: 1) spatially extended in three dimensions; 2) causally unified, meaning its parts are tied together by relations of connection in such a way that if one part of the object is moved in space, then its other parts will likely be moved also; and 3) maximally self-connected\". The definitions given by the three ontologies are almost similar and they mean that an organism is an individual living system that may be unicellular or made up of many billions of cells like humans. This leads us to conclude that an \u003cem\u003eOrganism\u003c/em\u003e is definitely an \u003cem\u003eobject\u003c/em\u003e, contrary to the IDO representation. \u003cem\u003ePathogen\u003c/em\u003e in CPRO is a subclass of \u003cem\u003eorganism\u003c/em\u003e and it is defined as \"any virus, microorganism, or other substance causing disease\". This definition is limited because, in the broadest sense, \u003cem\u003epathogen\u003c/em\u003e is anything that can produce disease. SNOMED-CT provides \u003cem\u003epathogenic organism\u003c/em\u003e rather than \u003cem\u003eorganism\u003c/em\u003e, as a subclass of \u003cem\u003enavigational concept\u003c/em\u003e, separate branch in the class hierarchy. IDO provides a more accurate definition and representation of \u003cem\u003epathogen. pathogen\u003c/em\u003e is a subclass of \u003cem\u003ematerial entity\u003c/em\u003e with a \u003cem\u003epathogenic disposition\u003c/em\u003e. The latter is a subclass of \u003cem\u003edisposition\u003c/em\u003e defined in IDO as \"a disposition to initiate processes that result in a disorder\". It means that some organisms in some circumstances of their lives play the role of pathogens towards other organisms and cause diseases. We create \u003cem\u003epathogen\u003c/em\u003e as a subclass of \u003cem\u003ematerial entity\u003c/em\u003e and \u003cem\u003epathogen role\u003c/em\u003e as a subclass of \u003cem\u003erole\u003c/em\u003e which is already defined in BFO. We add the axiom \u003cem\u003epathogen\u003c/em\u003e is a \u003cem\u003ematerial entity\u003c/em\u003e \u003cb\u003eand\u003c/b\u003e (\u0026prime;\u003cem\u003ehas disposition\u0026prime;\u003c/em\u003e \u003cb\u003esome\u003c/b\u003e \u003cem\u003epathogenic disposition\u003c/em\u003e) \u003cb\u003eand\u003c/b\u003e (\u0026prime;\u003cem\u003ehas role\u0026prime;\u003c/em\u003e \u003cb\u003esome\u003c/b\u003e \u003cem\u003epathogen role)\u003c/em\u003e. \u003cem\u003ehas disposition\u003c/em\u003e is an object property defined in RO as \"a relation between an independent continuant (the bearer) and a disposition, in which the disposition specifically depends on the bearer for its existence\". \u003cem\u003ehas role\u003c/em\u003e is also an object property which is defined in RO as \"a relation between an independent continuant (the bearer) and a role, in which the role specifically depends on the bearer for its existence\".\u003c/p\u003e \u003c/div\u003e"},{"header":"6. Conclusions","content":"\u003cp\u003eThe main objective of this work was to show how to build a medical ontology of quality. The involvement of domain experts in all development steps contributes to getting the required quality but bringing together such a team is not easy. Providing the knowledge engineer with building guidelines to compensate for the domain experts\u0026rsquo; absence in some steps would be very helpful. It allows limiting the domain experts\u0026rsquo; involvement in the ontology domain and scope definition phase and the evaluation one.\u003c/p\u003e \u003cp\u003eWe propose a methodology inspired by METHONTOLOGY [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e] to which we made the following improvements: 1) in the corpus building and term extraction phase, we recommend the construction of knowledge corpus by using the most reliable source of knowledge that consists of a codified one; 2) in the building of preliminary ontology phase, we recommend the application of differential principles to explain the meaning of each ontology concept; the knowledge engineer expresses, in natural language, the similarities and differences that the concept has with those close to it in the concept hierarchy; 3) in the ontology reuse and enrichment phase, we recommend that the knowledge engineer use conflict resolution questions to resolve conflicts between concepts representations; 4) the methodology also provides guidance for the documentation (annotations, conceptual model of the ontology), the maintenance and evolution (versioning, dependency management), and the evaluation.\u003c/p\u003e \u003cp\u003eThe methodology combines ontology engineering from scratch using codified knowledge and reusing ontologies to enhance interoperability and reusability. The methodology can be used for other medical domains. For example, gold standard frameworks could be used as a source of codified knowledge to represent a prognosis domain.\u003c/p\u003e \u003cp\u003eThe suggested methodology could be very well applied to the building of ontologies covering the domains of other, not necessarily medical fields. In this case, one should identify a source of codified knowledge. A few examples of such sources are best practice guidelines for pollution prevention and waste minimization, billing norms in the industry, industrial standards, etc.\u003c/p\u003e \u003cp\u003eWe validate our methodology on pneumonia diagnosis by building PNADO. PNADO has been evaluated by physicians. The evaluation results showed that our methodology is effectual. PNADO is the first reported ontology developed to represent different pneumonia diagnosis aspects.\u003c/p\u003e \u003cp\u003eDuring the PNADO building, in the ontology reuse phase, we suggested some improvements to the OGMS, SYMP, and CPRO ontologies.\u003c/p\u003e \u003cp\u003eIn future work, we will consider the modularity aspect in the building process of a medical ontology.\u003c/p\u003e"},{"header":"Abbreviations","content":"\u003cdiv class=\"DefinitionList\"\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eBFO\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eBasic Formal Ontology\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eCIDO\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eCoronavirus Infectious Disease\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eCPRO\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eComputer-Based Patient Record Ontology\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eCPGs\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eClinical Practice Guidelines\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eCRQs\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003econflict resolution questions\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eCSSO\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eClinical Signs and Symptoms Ontology\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eCUI\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003econcept unique identifiers\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eDL\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003edescription logic\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eDOID\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eHuman Disease Ontology\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eGFO\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eGeneral Formal Ontology\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eHPO\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eHuman Phenotype Ontology\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eICD10\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eInternational Classification of Diseases\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eLOINC\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eLogical Observation Identifiers Names and Code\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eNCBO\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eNational Center for Biomedical Ontology\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eOGMS\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eOntology for General Medical Science\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eOM\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eOmitted models\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eOWL\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eWeb Ontology Language\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003ePNADO\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003ePneumonia Diagnostic Ontology\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eRadLex\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eRadiology Lexicon\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eRO\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eRelation Ontology\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eRTF\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eRelative Term Frequency\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eSNOMED-CT\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eSystematized Nomenclature of Medicine Clinical Terms\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eSUP\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eSuperfluous models\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eTFIDF\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eTerm Frequency Inverted Document Frequency\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eUMLS\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eUnified Medical Language System.\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003c/div\u003e"},{"header":"Declarations","content":"\u003cp\u003eCompeting interest\u003c/p\u003e\n\u003cp\u003eAuthor declares that there is no conflict of interest statements.\u003c/p\u003e\n\u003cp\u003eAuthors\u0026rsquo; contributions\u003c/p\u003e\n\u003cp\u003eThe author contributed all aspects to the paper. The author(s) read and approved the final manuscript.\u003c/p\u003e\n\u003cp\u003eFunding\u003c/p\u003e\n\u003cp\u003eThe author declares that she has not received project funding for this work.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eEthics approval and consent to participate\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eNot applicable.\u003c/p\u003e\n\u003cp\u003eConsent for publication\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eNot applicable\u003c/p\u003e\n\u003cp\u003eAcknowledgment\u003c/p\u003e\n\u003cp\u003eWe are indebted to the CISSSO (Centre Int\u0026eacute;gr\u0026eacute; de la Sant\u0026eacute; et des Services Sociaux de l\u0026apos;Outaouais) and specially Dr Sylvain Croteau, emergency physician at Gatineau, Qc.\u0026nbsp;Hospital, Yasmine Lisa Rebaine, a pneumologist in Charles-Le Moyne Hospital, and Serge Chartrand, a family doctor at UQO medical clinic. Last but not least, we would like to express our gratitude to Wojtek Michalowski, a professor at Ottawa University, for his support.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eSupplementary material\u003c/p\u003e\n\u003cp\u003ehttps://bioportal.bioontology.org/ontologies/PNADO\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003ePatel A, Debnath NC. A Comprehensive Overview of Ontology: Fundamental and Research Directions. Curr Mater Science: Formerly: Recent Pat Mater Sci. 2024;17(1):2\u0026ndash;20.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAmith M, et al. Assessing the practice of biomedical ontology evaluation: Gaps and opportunities. J Biomed Inf. 2018;80:1\u0026ndash;13.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAmith M, et al. Architecture and usability of OntoKeeper, an ontology evaluation tool. BMC Med Inf Decis Mak. 2019;19(4):152.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKatsumi M, Gr\u0026uuml;ninger M. \u003cem\u003eWhat is ontology reuse?\u003c/em\u003e in \u003cem\u003eFOIS\u003c/em\u003e. 2016.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAl-Arfaj A, Al-Salman A. Ontology construction from text: challenges and trends. Int J Artif Intell Expert Syst (IJAE). 2015;6(2):15\u0026ndash;26.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePathak J, Johnson TM, Chute CG. Survey of modular ontology techniques and their applications in the biomedical domain. Integr computer-aided Eng. 2009;16(3):225\u0026ndash;42.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSimperl E. Reusing ontologies on the Semantic Web: A feasibility study. Data Knowl Eng. 2009;68(10):905\u0026ndash;25.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZulkarnain NZ, Meziane F, Crofts G. \u003cem\u003eA methodology for biomedical ontology reuse\u003c/em\u003e. in \u003cem\u003eInternational conference on applications of natural language to information systems\u003c/em\u003e. 2016. Springer.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eOchs C, et al. An empirical analysis of ontology reuse in BioPortal. J Biomed Inform. 2017;71:165\u0026ndash;77.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFern\u0026aacute;ndez-L\u0026oacute;pez M, G\u0026oacute;mez-P\u0026eacute;rez A. Overview and analysis of methodologies for building ontologies. Knowl Eng Rev. 2002;17(2):129.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBrendish NJ, et al. Hospitalised adults with pneumonia are frequently misclassified as another diagnosis. Respir Med. 2019;150:81\u0026ndash;4.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eUschold M, King M. Towards a methodology for building ontologies. Citeseer; 1995.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eUschold M, et al. The enterprise ontology. Knowl Eng Rev. 1998;13(1):31\u0026ndash;89.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGr\u0026uuml;ninger M, Fox MS. \u003cem\u003eMethodology for the design and evaluation of ontologies.\u003c/em\u003e 1995.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eUschold M, Gruninger M. Ontologies: Principles, methods and applications. Knowl Eng Rev. 1996;11(2):93\u0026ndash;136.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFern\u0026aacute;ndez-L\u0026oacute;pez M, G\u0026oacute;mez-P\u0026eacute;rez A, Juristo N. \u003cem\u003eMethontology: from ontological art towards ontological engineering.\u003c/em\u003e 1997.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSchultz DJ. IEEE standard for developing software life cycle processes. IEEE Std, 1997: pp. 1074\u0026ndash;997.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNoy NF, McGuinness DL. \u003cem\u003eOntology development 101: A guide to creating your first ontology\u003c/em\u003e. 2001, Stanford knowledge systems laboratory technical report KSL-01-05 and Stanford medical informatics technical report SMI-2001-0880, Stanford, CA.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAl-Aswadi FN, Chan HY, Gan KH. Automatic ontology construction from text: a review from shallow to deep learning trend. Artif Intell Rev, 2019: pp. 1\u0026ndash;28.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFaure D, N\u0026eacute;dellec. C. A corpus-based conceptual clustering method for verb frames and ontology acquisition. LREC workshop on adapting lexical and corpus resources to sublanguages and applications. Citeseer; 1998.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMaedche A, Staab S. Ontology learning. Handbook on ontologies. Springer; 2004. pp. 173\u0026ndash;90.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCimiano P, V\u0026ouml;lker J. \u003cem\u003etext2onto\u003c/em\u003e. in \u003cem\u003eInternational conference on application of natural language to information systems\u003c/em\u003e. 2005. Springer.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eShamsfard M, Barforoush AA. Learning ontologies from natural language texts. Int J Hum Comput Stud. 2004;60(1):17\u0026ndash;63.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHahn U, Romacker M. \u003cem\u003eThe SYNDIKATE Text Knowledge Base Generator\u003c/em\u003e. in \u003cem\u003eProceedings of the first international conference on Human language technology research\u003c/em\u003e. 2001.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGillani Andleeb S. From text mining to knowledge mining: An integrated framework of concept extraction and categorization for domain ontology. Budapesti Corvinus Egyetem; 2015.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNavigli R, Velardi P, Gangemi A. Ontology learning and its application to automated terminology translation. IEEE Intell Syst. 2003;18(1):22\u0026ndash;31.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBuitelaar P, Olejnik D, Sintek M. \u003cem\u003eA prot\u0026eacute;g\u0026eacute; plug-in for ontology extraction from text based on linguistic analysis\u003c/em\u003e. in \u003cem\u003eEuropean Semantic Web Symposium\u003c/em\u003e. 2004. Springer.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFortuna B, Grobelnik M, Mladenic D. \u003cem\u003eSemi-automatic data-driven ontology construction system\u003c/em\u003e. in \u003cem\u003eProceedings of the 9th International multi-conference Information Society IS-2006, Ljubljana, Slovenia\u003c/em\u003e. 2006.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBi\u0026eacute;bow B, Szulman S. \u003cem\u003eTerminae: une approche terminologique pour la construction d\u0026rsquo;ontologies du domaine \u0026agrave; partir de textes.\u003c/em\u003e Actes de RFIA2000, Reconnaissances des Formes et Intelligence Artificielle, 2000.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTrokanas N, Koo L, Cecelja F. \u003cem\u003eTowards a Methodology for Reusable Ontology Engineering: Application to the Process Engineering Domain\u003c/em\u003e, in \u003cem\u003eComputer Aided Chemical Engineering\u003c/em\u003e. Elsevier; 2018. pp. 471\u0026ndash;6.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDram\u0026eacute; K, et al. Reuse of termino-ontological resources and text corpora for building a multilingual domain ontology: an application to Alzheimer\u0026rsquo;s disease. J Biomed Inform. 2014;48:171\u0026ndash;82.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBright TJ, et al. Development and evaluation of an ontology for guiding appropriate antibiotic prescribing. J Biomed Inform. 2012;45(1):120\u0026ndash;8.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHulshof CT. 1710d Systematic reviews and evidence-based guidelines, two of a different kind? BMJ Publishing Group Ltd.; 2018.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSackett DL, et al. Evidence based medicine: what it is and what it isn't. British Medical Journal Publishing Group; 1996.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGillois P, et al. From paper-based to electronic guidelines: application to French guidelines. Stud Health Technol Inf. 2001;84(Pt 1):196\u0026ndash;200.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGuarino N, Oberle D, Staab S. What is an ontology? Handbook on ontologies. Springer; 2009. pp. 1\u0026ndash;17.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBachimont B, Isaac A, Troncy R. \u003cem\u003eSemantic commitment for designing ontologies: a proposal\u003c/em\u003e. in \u003cem\u003eInternational Conference on Knowledge Engineering and Knowledge Management\u003c/em\u003e. 2002. Springer.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGrenon P, Smith B, Goldberg L. \u003cem\u003eBiodynamic ontology: applying BFO in the biomedical domain.\u003c/em\u003e Studies in health technology and informatics, 2004: pp. 20\u0026ndash;38.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eArp R, Smith B. Function, role and disposition in basic formal ontology. Nat Precedings, 2008: pp. 1\u0026ndash;1.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAronson AR. Metamap: Mapping text to the umls metathesaurus. Bethesda, MD: NLM, NIH, DHHS,; 2006. pp. 1\u0026ndash;26.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKatsumi M, Gr\u0026uuml;ninger M. Choosing ontologies for reuse. Appl Ontology. 2017;12(3\u0026ndash;4):195\u0026ndash;221.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCui L, et al. Mining non-lattice subgraphs for detecting missing hierarchical relations and concepts in SNOMED CT. J Am Med Inform Assoc. 2017;24(4):788\u0026ndash;98.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYamagata Y, et al. An ontological modeling approach for abnormal states and its application in the medical domain. J Biomedical Semant. 2014;5(1):23.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBrank J, Grobelnik M, Mladenic D. \u003cem\u003eA survey of ontology evaluation techniques\u003c/em\u003e. in \u003cem\u003eProceedings of the conference on data mining and data warehouses (SiKDD 2005)\u003c/em\u003e. 2005. Citeseer Ljubljana, Slovenia.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePorzel R, Malaka R. A task-based approach for ontology evaluation. ECAI Workshop on Ontology Learning and Population, Valencia, Spain. Citeseer; 2004.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMaedche A, Staab S. \u003cem\u003eMeasuring similarity between ontologies\u003c/em\u003e. in \u003cem\u003eInternational Conference on Knowledge Engineering and Knowledge Management\u003c/em\u003e. 2002. Springer.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBrewster C et al. \u003cem\u003eData driven ontology evaluation.\u003c/em\u003e 2004.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLozano-Tello A, G\u0026oacute;mez-P\u0026eacute;rez A. Ontometric: A method to choose the appropriate ontology. J Database Manage (JDM). 2004;15(2):1\u0026ndash;18.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eG\u0026oacute;mez-P\u0026eacute;rez A. Evaluation of ontologies. Int J Intell Syst. 2001;16(3):391\u0026ndash;409.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eVrandečić D. Ontology evaluation. Handbook on ontologies. Springer; 2009. pp. 293\u0026ndash;313.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhu X, et al. A review of auditing methods applied to the content of controlled biomedical terminologies. J Biomed Inform. 2009;42(3):413\u0026ndash;25.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBurton-Jones A, et al. A semiotic metrics suite for assessing the quality of ontologies. Data Knowl Eng. 2005;55(1):84\u0026ndash;102.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAruna T, Saranya K, Bhandari C. \u003cem\u003eA survey on ontology evaluation tools\u003c/em\u003e. in 2011 \u003cem\u003eInternational Conference on Process Automation, Control and Computing\u003c/em\u003e. 2011. IEEE.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCorbridge C, et al. Laddering: technique and tool use in knowledge acquisition. Knowl Acquisition. 1994;6(3):315\u0026ndash;41.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePoveda-Villal\u0026oacute;n M, G\u0026oacute;mez-P\u0026eacute;rez A. Su\u0026aacute;rez-Figueroa, \u003cem\u003eOops!(ontology pitfall scanner!): An on-line tool for ontology evaluation\u003c/em\u003e. Int J Semantic Web Inform Syst (IJSWIS). 2014;10(2):7\u0026ndash;34.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDoran P, Tamma V, Iannone L. \u003cem\u003eOntology module extraction for ontology reuse: an ontology engineering perspective\u003c/em\u003e. in \u003cem\u003eProceedings of the sixteenth ACM conference on Conference on information and knowledge management\u003c/em\u003e. 2007.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKumar S, Baliyan N. \u003cem\u003eQuality Evaluation of Ontologies\u003c/em\u003e, in \u003cem\u003eSemantic Web-Based Systems\u003c/em\u003e. Springer; 2018. pp. 19\u0026ndash;50.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHalland K, Britz K. Investigations into the use of SNOMED CT to enhance an OpenMRS health information system. South Afr Comput J. 2011;47(1):33\u0026ndash;45.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eEl-Sappagh S, et al. SNOMED CT standard ontology based on the ontology for general medical science. BMC Med Inf Decis Mak. 2018;18(1):76.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eShah T et al. \u003cem\u003eA guiding framework for ontology reuse in the biomedical domain\u003c/em\u003e. in \u003cem\u003eSystem Sciences (HICSS)\u003c/em\u003e, 2014 \u003cem\u003e47th Hawaii International Conference on\u003c/em\u003e. 2014. IEEE.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJohnson AE, et al. MIMIC-III, a freely accessible critical care database. Sci Data. 2016;3:160035.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMetlay JP, et al. Diagnosis and Treatment of Adults with Community-acquired Pneumonia. An Official Clinical Practice Guideline of the American Thoracic Society and Infectious Diseases Society of America. Am J Respir Crit Care Med. 2019;200(7):e45\u0026ndash;67.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"},{"header":"Footnotes","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://bioportal.bioontology.org/ontologies/CIDO\u003c/span\u003e\u003cspan address=\"https://bioportal.bioontology.org/ontologies/CIDO\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://bioportal.bioontology.org/ontologies/COVID-19\u003c/span\u003e\u003cspan address=\"https://bioportal.bioontology.org/ontologies/COVID-19\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://www.obofoundry.org/ontology/doid.html\u003c/span\u003e\u003cspan address=\"http://www.obofoundry.org/ontology/doid.html\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://browser.ihtsdotools.org/?\u003c/span\u003e\u003cspan address=\"https://browser.ihtsdotools.org/?\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://www.obofoundry.org/ontology/hp.html\u003c/span\u003e\u003cspan address=\"http://www.obofoundry.org/ontology/hp.html\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.cochranelibrary.com/\u003c/span\u003e\u003cspan address=\"https://www.cochranelibrary.com/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/07/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.nice.org.uk/guidance\u003c/span\u003e\u003cspan address=\"https://www.nice.org.uk/guidance\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/07/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://ctakes.apache.org/index.html\u003c/span\u003e\u003cspan address=\"https://ctakes.apache.org/index.html\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/07/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.nlm.nih.gov/research/umls/index.html\u003c/span\u003e\u003cspan address=\"https://www.nlm.nih.gov/research/umls/index.html\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://bioportal.bioontology.org/\u003c/span\u003e\u003cspan address=\"https://bioportal.bioontology.org/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://www.obofoundry.org/\u003c/span\u003e\u003cspan address=\"http://www.obofoundry.org/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://www.protegewiki.stanford.edu/\u003c/span\u003e\u003cspan address=\"http://www.protegewiki.stanford.edu/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://www.swoogle.umbc.edu/\u003c/span\u003e\u003cspan address=\"http://www.swoogle.umbc.edu/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://bioportal.bioontology.org/recommender\u003c/span\u003e\u003cspan address=\"https://bioportal.bioontology.org/recommender\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://bioportal.bioontology.org/ontologies/CIDO/?p=summary\u003c/span\u003e\u003cspan address=\"https://bioportal.bioontology.org/ontologies/CIDO/?p=summary\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://oops.linkeddata.es/\u003c/span\u003e\u003cspan address=\"http://oops.linkeddata.es/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://njitsaboc.github.io/\u003c/span\u003e\u003cspan address=\"https://njitsaboc.github.io/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://ontometrics.informatik.uni-rostock.de/ontologymetrics/index.jsp\u003c/span\u003e\u003cspan address=\"https://ontometrics.informatik.uni-rostock.de/ontologymetrics/index.jsp\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://webprotege.stanford.edu/\u003c/span\u003e\u003cspan address=\"https://webprotege.stanford.edu/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://www.obofoundry.org/principles/fp-000-summary.html\u003c/span\u003e\u003cspan address=\"http://www.obofoundry.org/principles/fp-000-summary.html\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://protegewiki.stanford.edu/wiki/OWLDoc\u003c/span\u003e\u003cspan address=\"https://protegewiki.stanford.edu/wiki/OWLDoc\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://essepuntato.it/lode/\u003c/span\u003e\u003cspan address=\"https://essepuntato.it/lode/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.cochranelibrary.com/\u003c/span\u003e\u003cspan address=\"https://www.cochranelibrary.com/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.nice.org.uk/\u003c/span\u003e\u003cspan address=\"https://www.nice.org.uk/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.idsociety.org/\u003c/span\u003e\u003cspan address=\"https://www.idsociety.org/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.escmid.org/\u003c/span\u003e\u003cspan address=\"https://www.escmid.org/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.asid.net.au/\u003c/span\u003e\u003cspan address=\"https://www.asid.net.au/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://cts-sct.ca/\u003c/span\u003e\u003cspan address=\"https://cts-sct.ca/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://pulmccm.org/\u003c/span\u003e\u003cspan address=\"https://pulmccm.org/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.thoracic.org/\u003c/span\u003e\u003cspan address=\"https://www.thoracic.org/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.brit-thoracic.org.uk/\u003c/span\u003e\u003cspan address=\"https://www.brit-thoracic.org.uk/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://www.obofoundry.org/ontology/bfo.html\u003c/span\u003e\u003cspan address=\"http://www.obofoundry.org/ontology/bfo.html\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://bioportal.bioontology.org/ontologies/GFO\u003c/span\u003e\u003cspan address=\"https://bioportal.bioontology.org/ontologies/GFO\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://bioportal.bioontology.org/ontologies/OGMS\u003c/span\u003e\u003cspan address=\"https://bioportal.bioontology.org/ontologies/OGMS\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://bioportal.bioontology.org/ontologies/DIAGONT\u003c/span\u003e\u003cspan address=\"https://bioportal.bioontology.org/ontologies/DIAGONT\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://www.obofoundry.org/ontology/symp.html\u003c/span\u003e\u003cspan address=\"http://www.obofoundry.org/ontology/symp.html\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://bioportal.bioontology.org/ontologies/CSSO\u003c/span\u003e\u003cspan address=\"https://bioportal.bioontology.org/ontologies/CSSO\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://browser.ihtsdotools.org/?\u003c/span\u003e\u003cspan address=\"https://browser.ihtsdotools.org/?\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://www.obofoundry.org/ontology/ncbitaxon.html\u003c/span\u003e\u003cspan address=\"http://www.obofoundry.org/ontology/ncbitaxon.html\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://bioportal.bioontology.org/ontologies/GAMUTS\u003c/span\u003e\u003cspan address=\"https://bioportal.bioontology.org/ontologies/GAMUTS\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://radlex.org/\u003c/span\u003e\u003cspan address=\"http://radlex.org/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://loinc.org/\u003c/span\u003e\u003cspan address=\"https://loinc.org/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://bioportal.bioontology.org/ontologies/LABO\u003c/span\u003e\u003cspan address=\"https://bioportal.bioontology.org/ontologies/LABO\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://bioportal.bioontology.org/ontologies/ICD10\u003c/span\u003e\u003cspan address=\"https://bioportal.bioontology.org/ontologies/ICD10\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://www.obofoundry.org/ontology/doid.html\u003c/span\u003e\u003cspan address=\"http://www.obofoundry.org/ontology/doid.html\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://www.obofoundry.org/ontology/ro.html\u003c/span\u003e\u003cspan address=\"http://www.obofoundry.org/ontology/ro.html\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://bioportal.bioontology.org/ontologies/CPRO\u003c/span\u003e\u003cspan address=\"https://bioportal.bioontology.org/ontologies/CPRO\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://www.obofoundry.org/ontology/hp.html\u003c/span\u003e\u003cspan address=\"http://www.obofoundry.org/ontology/hp.html\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://www.obofoundry.org/ontology/ido.html\u003c/span\u003e\u003cspan address=\"http://www.obofoundry.org/ontology/ido.html\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://physionet.org/\u003c/span\u003e\u003cspan address=\"http://physionet.org/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://santemonteregie.qc.ca/installations/hopital-charles-le-moyne\u003c/span\u003e\u003cspan address=\"https://santemonteregie.qc.ca/installations/hopital-charles-le-moyne\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://cisss-outaouais.gouv.qc.ca/\u003c/span\u003e\u003cspan address=\"https://cisss-outaouais.gouv.qc.ca/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://ssuqo.ca/\u003c/span\u003e\u003cspan address=\"https://ssuqo.ca/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.cebm.net/\u003c/span\u003e\u003cspan address=\"https://www.cebm.net/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, last access 08/08/2024.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Medical ontology engineering, Ontology reuse, Clinical practice guidelines, OBO Foundry, Competency questions, Pneumonia diagnosis.","lastPublishedDoi":"10.21203/rs.3.rs-5305559/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5305559/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eOntology development is a multidisciplinary work involving domain experts and knowledge engineers. Bringing together such a team to develop an ontology of quality is not easy. Therefore, ontologies are often created with limited expertise either in the medical domain or in ontology engineering. Unfortunately, the existing methodologies do not provide much guidance on how the different steps of ontology development should be performed, particularly in the case of reduced involvement of domain experts. This challenge is getting more difficult when there is a multitude of medical knowledge sources and ontologies covering parts of the domain, and often each having a different representation of the same concept.\u003c/p\u003e \u003cp\u003eThis research presents a methodology for creating a medical ontology of quality with limited involvement of the domain experts. The latter are only consulted in the domain definition and evaluation phases. We combine building an ontology from codified knowledge and ontology reuse to enhance reusability and interoperability. The methodology is inspired by METHONTOLOGY for which we make several improvements, especially in the ontology reuse phase.\u003c/p\u003e \u003cp\u003eWe provide a proof of concept of the proposed methodology with a case study involving the development of the pneumonia diagnosis ontology (PNADO).\u003c/p\u003e","manuscriptTitle":"A methodology for building a medical ontology a with a limited domain experts’ involvement","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-11-11 06:52:44","doi":"10.21203/rs.3.rs-5305559/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"edc55896-8a51-49fd-96fc-57e1464f6b92","owner":[],"postedDate":"November 11th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-02-04T14:08:49+00:00","versionOfRecord":[],"versionCreatedAt":"2024-11-11 06:52:44","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5305559","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5305559","identity":"rs-5305559","version":["v1"]},"buildId":"WrCJVZZCHTDjtuVLN7oU0","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}