Restructuring knowledge graphs with conceptual models: implications for machine learning predictions in drug repurposing

Restructuring knowledge graphs with conceptual models: implications for machine learning predictions in drug repurposing | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Restructuring knowledge graphs with conceptual models: implications for machine learning predictions in drug repurposing César Bernabé, Rosa Zwart, Pablo Perdomo-Quinteiro, Annika Jacobsen, and 6 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5622649/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract This paper investigates the impact of restructuring knowl- edge graphs (KGs) with well-founded conceptual models to improve ma- chine learning (ML) predictions, particularly in drug repurposing appli- cations. These conceptual models were developed using OntoUML, which is grounded in the Unified Foundational Ontology, and were constructed following an established workflow for data FAIRification–a process aimed at making data more Findable, Accessible, Interoperable, and Reusable. We compared the performance of a Graph Neural Network model trained on original public KGs with models trained on the same restructured KGs. Our results indicate that while the ML model classification perfor- mance (measured in terms of accuracy and error metrics) remains similar for both, the models trained on restructured KGs produce more consis- tent predictions, reducing variability across multiple runs. These findings suggest that restructuring KGs using well-founded conceptual models can enhance the reliability of ML predictions without compromising model performance. We conclude by proposing future research directions to fur- ther explore the potential of conceptual models and FAIR principles in improving ML. Bioinformatics Artificial Intelligence and Machine Learning Conceptual Model Machine Learning FAIR Principles Knowledge Graphs Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 Figure 8 Figure 9 1 Introduction and Background Machine learning (ML) models are often trained using large knowledge bases [3]. However, constructing such voluminous datasets is both resource-intensive and time-consuming, as existing data is typically not prepared for reuse [12]. To facilitate data reusability, the FAIR principles were introduced to guide the pro- cess of making data and other resources Findable, Accessible, Interoperable, and Reusable [26]. Since their publication in 2016, the principles have gained significant traction across various fields [13]. Similarly, research and applications involving ML models have expanded rapidly in recent years [3]. However, al- though these areas complement each other, there has been little research on the specific impact of FAIR data on ML methods. Jacobsen et al. proposed a stepwise process of making existing data FAIR (referred to as FAIRification), which has been widely adopted. The generic FAIR- ification workflow is organised in steps, starting with the identification of FAIRi- fication objectives, followed by the analysis of (meta)data, the design of semantic models for (meta)data, and (meta)data linkage, hosting, and assessment. In the semantic modelling phase, a conceptual model [9] of the data elements (e.g., pa- tient, disease) and relationships (e.g. drug treats disease) is constructed. Since FAIR aims to support reuse by both humans and machines, the conceptual model is designed to be as accurate a reflection of the data domain as possible [13]. In the data linkage step, the data to be made FAIR is reorganised to align with the structure of the conceptual model, making it not only more understandable for humans but also more easily integrated with other FAIR data. In this work, we aim to assess the impact of this conceptual model-based data re-structuration on ML models. To conduct this experiment, we build on parts of the pipeline developed by Perdomo-Quinteiro et al. [19], which involves reusing data from public sources to create a knowledge graph (KG) for training a Graph Neural Network (GNN) to predict drugs that can be repurposed to treat symptoms of rare diseases. We replicate the data fetching process of Perdomo- Quinteiro et al.’s pipeline (named rd-explainer ) to generate an initial KG. Then, we restructure the KG previously generated based on a conceptual model and compare the performance and outputs of the GNN model when trained on both cases. Our results highlight promising directions for future research, despite being limited to a single domain and ML algorithm. Experimentation shows that mod- els trained on the conceptual-model-based KGs (CM-based KGs) produced more consistent predictions (i.e. less random), with variability in predictions across different runs of the GNN being 29.91% lower compared to those on the orig- inal KGs. Additionally, the predictive performance of the GNN model trained on the CM-based KGs did not show a significant difference from the original KGs in terms of accuracy and error metrics. These findings suggest that further exploration of CM-based KGs could yield valuable insights. For the sake of clarity of this text, it is important to note that the term “model” carries multiple definitions depending on the research field. In the con- text of machine learning, a “model” is a mathematical representation or algo- rithm used to make predictions or decisions based on data [18]. Conversely, in conceptual modelling research, a “model” serves as a structured framework that represents the concepts and relationships within a specific domain, thus pro- viding a formalised approach to organising and interpreting information [9]. To maintain clarity, we differentiate between these definitions using the wordings “ML model” and “conceptual model” to define the different interpretations in each case, respectively. The remainder of this paper is organised as follows: Section 2 provides a brief overview of rd-explainer . Section 3 describes the method used in our ex- plorations. Section 4 presents our results, followed by a discussion in Section 5. Sections 6 and 7 address the limitations of our study and related works, respec- tively. Finally, Section 8 concludes the paper. 2 The rd-explainer use case The experimentation described in this work builds upon the method presented by Perdomo-Quinteiro et al. , which developed rd-explainer , an innovative inter- pretable ML method for drug repurposing. Drug repurposing identifies new uses for existing drugs, a cost-effective strategy particularly valuable for rare diseases with limited pharmaceutical interest [20]. rd-explainer relies on aggregated data sourced from three key public knowledge bases: the Monarch Initiative [22], Drug Central [24], and the Therapeutic Target Database [4]. During the pipeline execution, data is initially retrieved from Monarch using a disease code from a community disease ontology as the starting point for constructing the initial KG. The fetching script uses the disease code to identify the corresponding disease node in Monarch, and subsequently fetches all nodes directly related to it. The KG is then enriched with data from Drug Central and the Therapeutic Target Database, incorporating information about drugs and their associated treated symptoms. The data from these two additional sources is adjusted to conform to the original graph structure defined by Monarch. Subsequent to generating the disease-specific KG, a GNN model (Graph- SAGE [10]) is trained on it. The output of this process is a ranked list of pre- dictions, with each entry representing the probability of a link existing between a drug and the target symptom. The higher the score, the greater the likelihood of an actual edge existing between the two nodes, indicating a stronger poten- tial relationship between the drug and the symptom. For more information on rd-explainer , the reader can refer to Perdomo-Quinteiro et al. [19]. 3 Method To restructure the KG produced by the rd-explainer , we followed relevant FAIR- ification steps: identification of FAIRification objectives, data analysis, concep- tual modelling, and data restructuring (data linkage). Subsequently, we com- pared the output and performance of the GNN model when trained on the original KG and on the CM-based one. Identification of FAIRification Objectives The objectives identified in this step focus on making data reusable for drug repurposing applications. These include ‘identifying existing drugs that can be repurposed to treat the symptoms of rare diseases’, as well as the sub-objectives ‘identifying drugs that target genes associated (in)directly with a rare disease’ and ‘identifying drugs known to treat phenotypes associated (in)directly with a rare disease.’ Data Analysis In this step, the original KG constructed during the execution of rd-explainer was analysed, and its structure was extracted as it serves as a starting point for conceptual modelling. This structural map (also referred to as metamodel) is illustrated in Fig. 1, and an excerpt of it is depicted in Fig. 2 for better visualisation. Conceptual Modelling The conceptual model for the drug repurposing domain was developed iteratively. This involved designing the domain model using Onto- UML [8], a modelling language based on the Unified Foundational Ontology [7], which supports the creation of ontologically well-founded conceptual models , ensuring semantic clarity in representing real-world phenomena. The resulting model was validated with experts through three rounds of validation. An illustration of the resulting conceptual model is shown in Fig. 3. It represents key biological entities and their relationships within the domain of drug repurposing and (rare) diseases. At the core, the model involves entities such as Gene, Variant, Genotype, and Phenotype, which are fundamental to un- derstanding genetic and phenotypic expressions of diseases. A Gene is a collective biological entity that may have interactions (e.g., it interacts with or co-localises with other genes). An Ortholog, shown as a subtype of Gene, refers to genes in different species that evolved from a common ancestral gene by speciation and typically retain the same function, making them crucial for studying dis- ease mechanisms across species. Variants are specific alterations in a gene, and they can be part of a Genotype, which represents the complete set of an organ- ism’s genetic information. The model shows how Variants and Genotypes express Genes, impacting Biological Processes like Molecular Functions and Cellular Com- ponents. Additionally, Drugs are connected to Diseases and Phenotypes through their treatment relationships ( is a substance that treats ), thus targeting Gene Products, which are produced by genes and influence disease-related functions. Data Linkage Following the conceptual modelling step, the initial KG was re- organised according to the elements and relationships defined in the conceptual model from Fig. 3. To achieve this, the data-fetching script from rd-explainer was modified to generate the CM-based KG. The mapping from the original to the CM-based KG was manually reviewed by one of the authors and an external bioinformatician. All data used in this study were retrieved from the Septem- ber 2021 version of Monarch on 1 May 2024. Data from Drug Central and the Therapeutic Target Database were also collected on 1 May 2024. GNN prediction and performance assessment After having both the original and CM-based KGs constructed, we proceeded to train separate GNN models using each KG type. This process was repeated ten times for each KG type to collect performance metrics that were averaged to ensure a balanced comparison (i.e. AUC-ROC [14], which evaluates a model’s capacity to distinguish between classes, where a higher score implies better classification; F1 score [11], which is the harmonic mean of precision and recall; and Cross Entropy Loss [17], which measures the difference between predicted probabilities and true labels, with lower values indicating a more accurate model performance). At the end of this process, 20 prediction lists are generated: 10 lists from the original KG and 10 from the CM-based KG. Next, to assess the reliability of the ML models, we evaluated the consis- tency of their predictions. This evaluation was motivated by Perdomo et al. ’s observation regarding the challenges of ensuring reproducibility in ML methods. For instance, given the stochastic nature of some components of rd-explainer (e.g. edge2vec [6]), different runs of the pipeline can output different lists of pre- dictions. Thus, a considerably reliable model would produce robust predictions that are not significantly affected by random variation. To measure this, we per- formed pairwise comparisons of the prediction lists separately for each of the two groups of 10 iterations from each KG type. For each pair of prediction lists, we calculated the percentage of overlapping predictions, with a lower overlap indicating less consistency across iterations and greater variability in the ML model’s outcomes. Targeting a rare disease As previously mentioned, the process described above requires specifying a target disease (i.e. a disease code) when constructing the KGs, meaning the GNN models are trained on disease-specific KGs. To gather more comprehensive insights from our experiments, we applied our method to three different rare diseases: Duchenne Muscular Dystrophy (DMD) [23], Hunt- ington’s Disease (HD) [25], and Osteogenesis Imperfecta (OI) [21]. This resulted in the construction of six distinct KGs—two for each disease: one original KG and one CM-based KG—and enabled disease-specific comparisons. A workflow illustration of the method described in this section, along with the detailed FAIRification objectives, the original KG metamodel, the conceptual model, the data-fetching and training scripts, the performance measurements and resulting predictions are available in the supplementary material.6 4 Results A readily applicable outcome of this work is the conceptual model developed dur- ing our method (Fig. 3 ). Moreover, our assessments indicate that the GNN models trained on both types of KGs achieved comparable performance in met- rics such as AUC and F1 scores. More significantly, the comparison in terms of output reliability reveals that models trained on CM-based KGs produced more consistent predictions (i.e. similar prediction results). Zoomed-in versions of the figures presented in the next subsections are available in the supplementary ma- terial. 4.1 The OntoUML-based conceptual model is reusable The conceptual model designed in this work can be reused in other ML systems, FAIRification processes and extended for various applications in related domains. When comparing the original and restructured KGs (Figs. 1 and 3 ), it can be observed that the number of nodes increased from the original KG to the CM-based KG. In contrast, the number of relationships decreased significantly, as some concepts previously defined as relationships in the original KG were transformed into concepts in the restructured version. For instance, the has phenotype relationship in the original KG, which linked Gene and Disease, was transformed into a Phenotype concept in the restructured KG. 4.2 Predictive performance is similar The (averaged) training curves of the GNN models are illustrated in Figs. 4, 5, and 6, for DMD, HD, and OI, respectively. High-resolution versions including the loss values are available in the supplementary material. Figures 4a, 5a, and 6a display the training metrics of the GNN models trained on the original KGs, while Figs. 4b, 5b, and 6b show the metrics from the training on the CM- based KGs. Each figure presents the AUC-ROC scores and the Cross Entropy 6 https://edu.nl/m8xg7 (scripts) and https://edu.nl/k4648 (figures and outputs) Loss of the training processes. When comparing the (a) and (b) versions of each figure, it is important to note that the training curves differ in the total number of epochs as distinct hyperparameter optimisation processes (random search) [ 2 ] were performed for each of the six KGs–this is motivated by our aim to evaluate the impact of CM-based restructuration on the entire process. Overall, for all models trained on both the original and CM-based KGs, the training process starts with a remarkably high AUC-ROC score for both the training and test sets. However, the improvement from the start to the end of training is minimal. Table 1 shows a summary of the average AUC-ROC and F1 scores for each case and target rare disease. For DMD, both the AUC-ROC and F1 scores are slightly higher when training the GNN model on the original KG, although the difference is minimal. For HD, training on the original KG resulted in a higher average AUC-ROC, while training on the CM-based KG resulted in a higher F1 score with a significant difference. For OI, the average AUC-ROC and F1 scores were higher when the ML model was trained on the CM-based KG. Table 1 AUC-ROC and F1 scores for training on original and CM-based KG, for each target rare disease. Disease DMD HD OI KG Type Original CM-based Original CM-based Original CM-based AUC-ROC F1 0.977 0.933 0.976 0.906 0.978 0.896 0.967 0.934 0.9602 0.812 0.974 0.915 4.3 Predictive consistency is higher for CM-based KGs Figures 7 , 8 , and 9, illustrate the degree of overlap (expressed as a percentage) between predicted drug-phenotype pairs across all ten runs for the original and CM-based KG for DMD, HD and OI, respectively. Figures 7 a, 8 a and 9a are derived from the predictions of the GNN model trained on the original KG, whereas those in Fig. 7 b, 8 b and 9b are derived from the predictions of the GNN model trained on the CM-based KG. The means and median of the overlaps described in Figs. 7 , 8 , and 9 are summarised in Table 2 . Given the mean and median values of the overlap of predicted drug-phenotype pairs, it can be observed a higher mean and median for when rd-explainer is applied to the CM-based KG in the experiments conducted for all target rare diseases. The complete lists of predictions are available in the supplementary material. 5 Discussion We extend our results to propose specific research questions (RQs) to guide fu- ture studies. Within the context of FAIR and FAIRification, exploring these RQs will enhance understanding of the benefits of FAIR principles for ML applica- tions. Additionally, addressing these RQs will support gathering more data to make the conclusions of our work more generalisable. Table 2 Summary of the consistency of predictions of all three experiments. The percentages in parenthesis represent the increase of the CM-based values when compared to original KG ones (e.g. increase in mean from original to CM-based). Disease DMD HD OI KG Type Original CM-based Original CM-based Original CM-based Consistency mean Consistency median 38.97 39.29 54.1 (+ 38.82%) 53.57 (+ 36.35%) 24.27 25.29 48.43 (+ 99.55%) 52.87 (+ 109.05%) 11.53 10.61 39.61 (+ 243.54%) 37.88 (+ 257.02%) Training data When examining the Cross Entropy Loss curve for OI (Fig. 6), it becomes evident that the curve of the ML model trained on the CM-based KG is steeper than that of the ML model trained on the original KG. This may suggest that the ML model trained on the CM-based KG “learned more” when compared to the one trained on the original KG. Consequently, future research should explore whether restructuring training data according to well-founded conceptual models can enhance learning in ML models (e.g. GNN models) that initially do not perform well when trained on current data. Thus, a related RQ could be: “RQ1 - Do conceptual model based data improve the predictive performance of ML algorithms that underperform on current data?” An example of an experiment to address RQ1 could involve identifying cases where data scientists do not manage to further improve the performance of ML models on specific datasets. In such cases, well-founded conceptual models of the datasets’ subjects would be designed and used to restructure those datasets. The performance of the ML algorithms would then be tested on these newly restructured datasets. Reproducibility When comparing the consistency of predictions from ML models trained on the original and CM-based KGs (Figs. 7 , 8 , and 9), it is observed that ML models trained on the latter produce more stable predictions than those trained on the former, as indicated by the average overlap among the 10 lists of predictions generated for each case (Table 2 ). This suggests that training ML models on data structured according to well-founded conceptual models enhances the consistency of predictions, thereby reducing the randomness of the results. Thus, a research question related to this aspect could be: “RQ2 - Do conceptual model based data contribute to the consistency of predictions of ML models?” To address RQ2, it would be necessary to replicate the experiments conducted in our study in a systematic manner, involving various algorithms and datasets. This approach will ensure that the results are statistically significant and that the conclusions can be generalised and reproduced across different scenarios. Design of AI systems During the application of our method, it was observed that conceptual models improved communication, understanding, and task exe- cution among different stakeholders. A pertinent research question arising from this observation is whether data scientists can enhance their performance in fea- ture engineering, ML model selection, and parameter tuning due to a better understanding of the domain data (provided by conceptual modelling tasks). This leads to the final research question: “RQ3 - Do conceptual models support stakeholders in the design of AI systems?” To test RQ3, a controlled experiment could be conducted in which one group of data scientists and developers is tasked with directly designing and implementing an ML pipeline, while another group is required to first create an OntoUML model before proceeding with the design and implementation of the ML pipeline. The results of these two groups would then be compared to evaluate the impact of conceptual modelling on the design and execution of AI systems. Such an experiment could also test whether the conceptual modelling task facilitates communication between data scientists and domain experts. It should be emphasised that the initial findings of this work are based on the assumption that future experiments will employ conceptual models that are (i) constructed using a well-founded modelling language or ontology as a foun- dation, and (ii) thoroughly validated by domain experts. Finally, it is important to highlight that the RQs described above are formulated from our initial ex- plorations. While some are based on subtle differences in the results (e.g., AUC curves), they remain valuable for further investigation, as they may reveal more significant and impactful differences in other ML applications, particularly those involving large KGs. 6 Limitations and future work Reproducibility and generalisability are the primary limitations of our results. Reproducibility, a well-recognised challenge in ML [1], remains difficult in this context. While our findings demonstrate that CM-based KGs lead to more con- sistent GNN models, achieving identical results to those presented in Section 4 is challenging due, for example, to the inherent randomness in certain compo- nents of the rd-explainer pipeline. To mitigate this issue, we ran the training and output generation ten times for each case and target disease, averaging the scores to reduce the impact of variability. Although we synthesised our findings into additional RQs to guide future research, we have not yet tested our findings sufficiently to draw generalisable conclusions. This limitation arises from the fact that our study focused on a single type of ML method, using the same set of data sources within a specific domain. Therefore, it is crucial to investigate the impact of conceptual models in different domains and with other types of ML methods, as well as with different types of data. For example, while our results may be relevant to graph data and AI methods designed for such data, the application of our method to other data types may not lead to meaningful differences in results. 7 Related works The use and impact of conceptual models and ontologies in AI has been a topic of discussion in the literature. Various studies examine how these artefacts can be applied in the field, such as to enhance outcomes and the design of AI systems. At this stage, we view related works as complementary efforts toward a shared ultimate goal: enhancing the understanding and application of ontologies in AI, and vice versa. Confalonieri & Guizzardi [ 5 ] outlined the different roles that ontologies and ontology-based conceptual models can play for neuro-symbolic AI systems from three key perspectives: reference modelling, common sense reasoning, and knowl- edge refinement and complexity management. Similarly, Maaß & Storey [ 16 ] ex- plored the benefits and synergies of integrating conceptual modelling with ML and proposed a framework that uses conceptual modelling to support the design and development of ML solutions. Lukyanenko et al. [ 15 ] explored how concep- tual modelling can address challenges in extracting insights from large datasets with machine learning. They showed that conceptual modelling aids various ML project phases: defining goals in the business understanding phase, modelling data and identifying quality issues in the data understanding phase, supporting attribute selection and transformation in data preparation, enhancing ML al- gorithm effectiveness with domain knowledge, improving result interpretability, and documenting process changes during deployment. 8 Final Remarks This work presented an exploratory study to investigate the impact of restruc- turing knowledge graphs using conceptual models–a step of FAIRification–on the performance of a GNN algorithm. We tested the GNN model’s behaviour when applied to both original and CM-based KGs across three different rare diseases targeted for drug repurposing. The initial results provided valuable insights and led to the formulation of additional refined research questions for future investi- gation, focusing on areas such as supporting the ML design process, improving predictive performance, and enhancing the consistency of predictions. The most prominent results of this work relate to the consistency of the pre- dictions. We evaluated the prediction lists generated across ten runs of the GNN method on both the original and CM-based KGs for each target disease. This evaluation involved pairwise comparisons measuring the overlap of predictions in each list. In all cases, the prediction lists generated from training the GNN model on the CM-based KGs were more consistent than those generated from the training on the original KGs. For FAIR and FAIRification, the initial findings herein presented serve as a proof-of-concept of the benefits of applying the FAIR principles to ML appli- cations. Our results demonstrate that FAIR data, structured using well-defined conceptual models, have the potential to enhance the consistency of ML out- puts without negatively impacting performance. Additionally, other elements of FAIR further enhance ML design and deployment. For instance, FAIR metadata improve resource discovery (e.g., finding data for training) and simplifies reuse by clearly specifying the conditions under which the data can be reused. ML and FAIR research are rapidly evolving fields. Our work contributes to this progress by exploring synergies between conceptual models, FAIR principles, and ML. As the next step, we aim to expand our research by applying our method to new domains, diverse data sources, and a broader range of ML approaches for more robust and generalisable results. Declarations Acknowledgments This initiative has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreements N°825575 and N° 101159589, and the EU4Health Programme under grant agreement N° 101129863. References Albertoni, R., Colantonio, S., Skrzypczyński, P., Stefanowski, J.: Reproducibility of machine learning: Terminology, recommendations and open issues. arXiv preprint arXiv:2302.12691 (2023) Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. Journal of machine learning research 13 (2) (2012) Brnabic, A., Hess, L.M.: Systematic literature review of machine learning methods used in the analysis of real-world data for patient-provider decision making. BMC Medical Informatics and Decision Making 21 (1), 54 (2021) Chen, X., Ji, Z.L., Chen, Y.Z.: Ttd: therapeutic target database. Nucleic acids research 30 (1), 412–415 (2002) Confalonieri, R., Guizzardi, G.: On the multiple roles of ontologies in explanations for neuro-symbolic ai. Neurosymbolic Artificial Intelligence (Preprint), 1–15 Gao, Z., Fu, G., Ouyang, C., Tsutsui, S., Liu, X., Yang, J., Gessner, C., Foote, B., Wild, D., Ding, Y., et al.: edge2vec: Representation learning using edge semantics for biomedical knowledge discovery. BMC bioinformatics 20 , 1–15 (2019) Guizzardi, G., Botti Benevides, A., Fonseca, C.M., et al.: UFO: Unified Founda- tional Ontology. Applied Ontology 17 (1), 167–210 (2022) Guizzardi, G., Fonseca, C.M., Benevides, A.B., et al.: Endurant types in ontology- driven conceptual modeling: Towards OntoUML 2.0. In: Conceptual Modeling. ER 2018. vol. 11157, p. 136–150. Springer (2018) Guizzardi, G., Wagner, G., Almeida, J.P.A., Guizzardi, R.S.: Towards ontological foundations for conceptual modeling: The unified foundational ontology (ufo) story. Applied ontology (2015) Hamilton, W., Ying, Z., Leskovec, J.: Inductive representation learning on large graphs. Advances in neural information processing systems 30 (2017) Hand, D.J., Christen, P., Kirielle, N.: F*: an interpretable transformation of the f-measure. Machine Learning 110 (3), 451–456 (2021) Hanson, B., Stall, S., Cutcher-Gershenfeld, J., Vrouwenvelder, K., Wirz, C., Rao, Y., Peng, G.: Garbage in, garbage out: mitigating risks and maximizing benefits of ai in research. Nature 623 (7985), 28–31 (2023) Jacobsen, A., de Miranda Azevedo, R., Juty, N., Batista, D., Coles, S., Cornet, R., Courtot, M., Crosas, M., Dumontier, M., Evelo, C.T., et al.: Fair principles: interpretations and implementation considerations (2020) Janssens, A.C.J., Martens, F.K.: Reflection on modern methods: Revisiting the area under the roc curve. International journal of epidemiology 49 (4), 1397–1403 (2020) Lukyanenko, R., Castellanos, A., Parsons, J., Chiarini Tremblay, M., Storey, V.C.: Using conceptual modeling to support machine learning. In: Information Systems Engineering in Responsible Information Systems: CAiSE Forum 2019, Rome, Italy, June 3–7, 2019, Proceedings 31. pp. 170–181. Springer (2019) Maass, W., Storey, V.C.: Pairing conceptual modeling with machine learning. Data & Knowledge Engineering 134 , 101909 (2021) Mao, A., Mohri, M., Zhong, Y.: Cross-entropy loss functions: Theoretical analysis and applications. In: International conference on Machine learning. pp. 23803– 23828. PMLR (2023) Ortigossa, E.S., Gonçalves, T., Nonato, L.G.: Explainable artificial intelligence (xai)–from theory to methods and applications. IEEE Access (2024) Perdomo-Quinteiro, P., Wolstencroft, K., Roos, M., Queralt-Rosinach, N.: Knowledge graphs and explainable ai for drug repurposing on rare diseases. bioRxiv (2024). https://doi.org/10.1101/2024.10.17.618804, https://www.biorxiv.org/content/early/2024/10/17/2024.10.17.618804 Pushpakom, S., Iorio, F., Eyers, P.A., Escott, K.J., Hopper, S., Wells, A., Doig, A., Guilliams, T., Latimer, J., McNamee, C., et al.: Drug repurposing: progress, chal- lenges and recommendations. Nature reviews Drug discovery 18 (1), 41–58 (2019) Rauch, F., Glorieux, F.H.: Osteogenesis imperfecta. The Lancet 363 (9418), 1377– 1385 (2004) Shefchek, K.A., Harris, N.L., Gargano, M., Matentzoglu, N., Unni, D., Brush, M., Keith, D., Conlin, T., Vasilevsky, N., Zhang, X.A., et al.: The monarch initia- tive in 2019: an integrative data and analytic platform connecting phenotypes to genotypes across species. Nucleic acids research 48 (D1), D704–D715 (2020) Szigyarto, C.A.K., Spitali, P.: Biomarkers of duchenne muscular dystrophy: current findings. Degenerative neurological and neuromuscular disease pp. 1–13 (2018) Ursu, O., Holmes, J., Knockel, J., Bologa, C.G., Yang, J.J., Mathias, S.L., Nelson, S.J., Oprea, T.I.: Drugcentral: online drug compendium. Nucleic acids research p. gkw993 (2016) Walker, F.O.: Huntington’s disease. The Lancet 369 (9557), 218–228 (2007) Wilkinson, M.D., Dumontier, M., Aalbersberg, I.J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.W., da Silva Santos, L.B., Bourne, P.E., et al.: The fair guiding principles for scientific data management and stewardship. Scien- tific data 3 , 1–9 (2016) Additional Declarations The authors declare no competing interests. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5622649","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":388947332,"identity":"dc1c7229-aea5-4be6-987f-fcd83419ed6e","order_by":0,"name":"César Bernabé","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAAyElEQVRIiWNgGAWjYFACHgZmMC3BwPjgAZjF3EC0FmaDBDCLkXgtbBJEaeFv4D3AXFBxL49/dvOxisS2bfIM0o34tUgc4EtgnnGmuFjizrG0G4lttw0bZA7i12LAwGPAzNuWkNhwI8cMpCWBQSKRGC3/EhLn38j/VkCCloaExA03ctgYiNIicZjH4PCMYwmJG+8cM5ZIOHfbsI2QFv72HsPHBTUJifNuNz/88KHstjy/RPIBvFpAkYKqgg2/+lEwCkbBKBgFxAAAR21FlcJf5ooAAAAASUVORK5CYII=","orcid":"https://orcid.org/0000-0003-1795-5930","institution":"Leiden University Medical Centre","correspondingAuthor":true,"prefix":"","firstName":"César","middleName":"","lastName":"Bernabé","suffix":""},{"id":388947333,"identity":"bbc272f2-f4ac-407e-ab1e-56fe31507a9b","order_by":1,"name":"Rosa Zwart","email":"","orcid":"","institution":"Leiden University Medical Centre","correspondingAuthor":false,"prefix":"","firstName":"Rosa","middleName":"","lastName":"Zwart","suffix":""},{"id":388947334,"identity":"a379f908-dc84-4566-9c92-1769bed8845f","order_by":2,"name":"Pablo Perdomo-Quinteiro","email":"","orcid":"https://orcid.org/0000-0001-8784-0907","institution":"Universidad Politécnica de Madrid","correspondingAuthor":false,"prefix":"","firstName":"Pablo","middleName":"","lastName":"Perdomo-Quinteiro","suffix":""},{"id":388947335,"identity":"a03b9a7b-a67e-4b61-8017-eb45d805392f","order_by":3,"name":"Annika Jacobsen","email":"","orcid":"https://orcid.org/0000-0003-4818-2360","institution":"Leiden University Medical Centre","correspondingAuthor":false,"prefix":"","firstName":"Annika","middleName":"","lastName":"Jacobsen","suffix":""},{"id":388947336,"identity":"2fb011a9-2d01-4295-bf79-23d8f4355c37","order_by":4,"name":"Tiago Prince Sales","email":"","orcid":"https://orcid.org/0000-0002-5385-5761","institution":"University of Twente","correspondingAuthor":false,"prefix":"","firstName":"Tiago","middleName":"Prince","lastName":"Sales","suffix":""},{"id":388947337,"identity":"b155c491-c72f-418a-a872-f997501dacc5","order_by":5,"name":"Núria Queralt-Rosinach","email":"","orcid":"https://orcid.org/0000-0003-0169-8159","institution":"Leiden University Medical Centre","correspondingAuthor":false,"prefix":"","firstName":"Núria","middleName":"","lastName":"Queralt-Rosinach","suffix":""},{"id":388947338,"identity":"4c106328-7cc6-43ef-905d-682f6440718b","order_by":6,"name":"Katherine Wolstencroft","email":"","orcid":"https://orcid.org/0000-0002-1279-5133","institution":"Amsterdam University Medical Centre","correspondingAuthor":false,"prefix":"","firstName":"Katherine","middleName":"","lastName":"Wolstencroft","suffix":""},{"id":388947339,"identity":"93e4777f-b5c5-47ba-a24e-c0533e1dc8d0","order_by":7,"name":"Luiz Olavo Bonino da Silva Santos","email":"","orcid":"https://orcid.org/0000-0002-1164-1351","institution":"Leiden University Medical Centre and University of Twente","correspondingAuthor":false,"prefix":"","firstName":"Luiz","middleName":"Olavo Bonino da Silva","lastName":"Santos","suffix":""},{"id":388947340,"identity":"bd688487-bbe3-4cab-aef0-1bf8bb137c24","order_by":8,"name":"Barend Mons","email":"","orcid":"","institution":"Leiden University Medical Centre","correspondingAuthor":false,"prefix":"","firstName":"Barend","middleName":"","lastName":"Mons","suffix":""},{"id":388947341,"identity":"7b6d4f1c-9b6e-4032-bb45-69ec15c779bc","order_by":9,"name":"Marco Roos","email":"","orcid":"https://orcid.org/0000-0002-8691-772X","institution":"Leiden University Medical Centre","correspondingAuthor":false,"prefix":"","firstName":"Marco","middleName":"","lastName":"Roos","suffix":""}],"badges":[],"createdAt":"2024-12-11 09:03:02","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-5622649/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5622649/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":75702810,"identity":"fe10f316-39bb-4abd-a212-56640b4e9a64","added_by":"auto","created_at":"2025-02-07 09:41:21","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":127002,"visible":true,"origin":"","legend":"\u003cp\u003eOverview of the metamodel of the original KG. Rectangles represent node types, and lines represent relationships types between the node types in the KG.\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-5622649/v1/bb6f510860254fe191909fe9.png"},{"id":75702809,"identity":"293ffc78-df01-4fed-8607-1c8081ca6418","added_by":"auto","created_at":"2025-02-07 09:41:21","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":89044,"visible":true,"origin":"","legend":"\u003cp\u003eAn excerpt of the metamodel of the original KG representing the concepts of Drug, Disease, Gene and Variant. Rectangles represent node types, and lines represent relationships types between the node types in the KG.\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-5622649/v1/e1e8a4f8a367612099cf7b46.png"},{"id":75704816,"identity":"a80a82c1-6fb6-4451-9065-4d52fa24c01a","added_by":"auto","created_at":"2025-02-07 10:05:21","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":204010,"visible":true,"origin":"","legend":"\u003cp\u003eConceptual Model for the drug repurposing domain. The resulting CM- based KG mirrors the structure of elements (nodes) and relationships shown in the model.\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-5622649/v1/9cb99c552f3e6410e9dd3b76.png"},{"id":75702814,"identity":"ac646f16-704a-4bab-80bc-8006ceffcd9f","added_by":"auto","created_at":"2025-02-07 09:41:21","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":87244,"visible":true,"origin":"","legend":"\u003cp\u003eTraining curves of the GNN models using the original and CM-based KG as input, for \u003cstrong\u003eDMD\u003c/strong\u003e. The blue, green and orange lines depicted in the figures on the right are the variations of AUC values among the 10 runs for the train, validation and test sets, respectively. The red line describes the variation in Cross-Entropy Loss.\u003c/p\u003e","description":"","filename":"4.png","url":"https://assets-eu.researchsquare.com/files/rs-5622649/v1/b41e799bd6e9734d6b1b802e.png"},{"id":75704426,"identity":"08e42a6b-d9a3-42db-b455-17c1b9c0cedf","added_by":"auto","created_at":"2025-02-07 09:57:21","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":93327,"visible":true,"origin":"","legend":"\u003cp\u003eTraining curves of the GNN models, for \u003cstrong\u003eHD\u003c/strong\u003e.\u003c/p\u003e","description":"","filename":"5.png","url":"https://assets-eu.researchsquare.com/files/rs-5622649/v1/7c824383298657394b0cc913.png"},{"id":75702819,"identity":"e48ff3a8-0a75-4e5b-a33b-cc51ccf7de2a","added_by":"auto","created_at":"2025-02-07 09:41:21","extension":"png","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":83991,"visible":true,"origin":"","legend":"\u003cp\u003eTraining curves of the GNN models, for \u003cstrong\u003eOI\u003c/strong\u003e.\u003c/p\u003e","description":"","filename":"6.png","url":"https://assets-eu.researchsquare.com/files/rs-5622649/v1/6c1fc7e685bfd815bf58b7a2.png"},{"id":75704814,"identity":"33c35e51-8f4e-46a0-9a75-1e547b2646dd","added_by":"auto","created_at":"2025-02-07 10:05:21","extension":"png","order_by":7,"title":"Figure 7","display":"","copyAsset":false,"role":"figure","size":154331,"visible":true,"origin":"","legend":"\u003cp\u003eHeat map that shows pairwise overlap of predicted drug-symptom pairs of all ten runs in percentages for each case, for \u003cstrong\u003eDMD\u003c/strong\u003e.\u003c/p\u003e","description":"","filename":"7.png","url":"https://assets-eu.researchsquare.com/files/rs-5622649/v1/55a89fd83869961a4f0562d8.png"},{"id":75702823,"identity":"00ccfd13-da53-40c7-8b30-37c665c3102e","added_by":"auto","created_at":"2025-02-07 09:41:21","extension":"png","order_by":8,"title":"Figure 8","display":"","copyAsset":false,"role":"figure","size":164761,"visible":true,"origin":"","legend":"\u003cp\u003ePairwise overlap of predicted drug-symptom pairs of all ten runs in per- centages for each case, for \u003cstrong\u003eHD\u003c/strong\u003e.\u003c/p\u003e","description":"","filename":"8.png","url":"https://assets-eu.researchsquare.com/files/rs-5622649/v1/6b88645e223c49325120a3b3.png"},{"id":75703134,"identity":"2822f2ca-801f-4e15-a0b7-9639a0c00da7","added_by":"auto","created_at":"2025-02-07 09:49:21","extension":"png","order_by":9,"title":"Figure 9","display":"","copyAsset":false,"role":"figure","size":165300,"visible":true,"origin":"","legend":"\u003cp\u003ePairwise overlap of predicted drug-symptom pairs of all ten runs in per- centages for each case, for \u003cstrong\u003eOI\u003c/strong\u003e.\u003c/p\u003e","description":"","filename":"9.png","url":"https://assets-eu.researchsquare.com/files/rs-5622649/v1/7570568c257eee3d316ca9a5.png"},{"id":75925271,"identity":"33550099-04a7-4f5c-a042-bee3b6421a46","added_by":"auto","created_at":"2025-02-10 15:12:10","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1645740,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5622649/v1/28a64c72-2a6d-4ff7-8497-bc6354b3277b.pdf"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003cp\u003eRestructuring knowledge graphs with conceptual models: implications for machine learning predictions in drug repurposing\u003c/p\u003e","fulltext":[{"header":"1 Introduction and Background","content":"\u003cp\u003eMachine learning (ML) models are often trained using large knowledge bases [3]. However, constructing such voluminous datasets is both resource-intensive and time-consuming, as existing data is typically not prepared for reuse [12]. To facilitate data reusability, the FAIR principles were introduced to guide the pro- cess of making data and other resources Findable, Accessible, Interoperable, and Reusable [26]. Since their publication in 2016, the principles have gained significant traction across various fields [13]. Similarly, research and applications involving ML models have expanded rapidly in recent years [3]. However, al- though these areas complement each other, there has been little research on the specific impact of FAIR data on ML methods.\u003c/p\u003e\n\u003cp\u003eJacobsen \u003cem\u003eet al. \u003c/em\u003eproposed a stepwise process of making existing data FAIR (referred to as FAIRification), which has been widely adopted. The generic FAIR- ification workflow is organised in steps, starting with the identification of FAIRi- fication objectives, followed by the analysis of (meta)data, the design of semantic models for (meta)data, and (meta)data linkage, hosting, and assessment. In the semantic modelling phase, a conceptual model [9] of the data elements (e.g., pa- tient, disease) and relationships (e.g. drug treats disease) is constructed. Since FAIR aims to support reuse by both humans and machines, the conceptual model is designed to be as accurate a reflection of the data domain as possible [13]. In the data linkage step, the data to be made FAIR is reorganised to align with the structure of the conceptual model, making it not only more understandable for humans but also more easily integrated with other FAIR data.\u003c/p\u003e\n\u003cp\u003eIn this work, we aim to assess the impact of this conceptual model-based data re-structuration on ML models. To conduct this experiment, we build on parts of the pipeline developed by Perdomo-Quinteiro \u003cem\u003eet al. \u003c/em\u003e[19], which involves reusing data from public sources to create a knowledge graph (KG) for training a Graph Neural Network (GNN) to predict drugs that can be repurposed to treat symptoms of rare diseases. We replicate the data fetching process of Perdomo- Quinteiro et al.’s pipeline (named \u003cem\u003erd-explainer \u003c/em\u003e) to generate an initial KG. Then, we restructure the KG previously generated based on a conceptual model and compare the performance and outputs of the GNN model when trained on both cases.\u003c/p\u003e\n\u003cp\u003eOur results highlight promising directions for future research, despite being limited to a single domain and ML algorithm. Experimentation shows that mod- els trained on the conceptual-model-based KGs (CM-based KGs) produced more consistent predictions (i.e. less random), with variability in predictions across different runs of the GNN being 29.91% lower compared to those on the orig- inal KGs. Additionally, the predictive performance of the GNN model trained on the CM-based KGs did not show a significant difference from the original KGs in terms of accuracy and error metrics. These findings suggest that further exploration of CM-based KGs could yield valuable insights.\u003c/p\u003e\n\u003cp\u003eFor the sake of clarity of this text, it is important to note that the term “model” carries multiple definitions depending on the research field. In the con- text of machine learning, a “model” is a mathematical representation or algo-\u003c/p\u003e\n\u003cp\u003erithm used to make predictions or decisions based on data [18]. Conversely, in conceptual modelling research, a “model” serves as a structured framework that represents the concepts and relationships within a specific domain, thus pro- viding a formalised approach to organising and interpreting information [9]. To maintain clarity, we differentiate between these definitions using the wordings “ML model” and “conceptual model” to define the different interpretations in each case, respectively.\u003c/p\u003e\n\u003cp\u003eThe remainder of this paper is organised as follows: Section 2 provides a brief overview of \u003cem\u003erd-explainer\u003c/em\u003e. Section 3 describes the method used in our ex- plorations. Section 4 presents our results, followed by a discussion in Section 5. Sections 6 and 7 address the limitations of our study and related works, respec- tively. Finally, Section 8 concludes the paper.\u003c/p\u003e"},{"header":"2\tThe rd-explainer use case","content":"\u003cp\u003eThe experimentation described in this work builds upon the method presented by Perdomo-Quinteiro \u003cem\u003eet al.\u003c/em\u003e, which developed \u003cem\u003erd-explainer\u003c/em\u003e, an innovative inter- pretable ML method for drug repurposing. Drug repurposing identifies new uses for existing drugs, a cost-effective strategy particularly valuable for rare diseases with limited pharmaceutical interest [20]. \u003cem\u003erd-explainer \u003c/em\u003erelies on aggregated data sourced from three key public knowledge bases: the Monarch Initiative [22], Drug Central [24], and the Therapeutic Target Database [4].\u003c/p\u003e\n\u003cp\u003eDuring the pipeline execution, data is initially retrieved from Monarch using a disease code from a community disease ontology as the starting point for constructing the initial KG. The fetching script uses the disease code to identify the corresponding disease node in Monarch, and subsequently fetches all nodes directly related to it. The KG is then enriched with data from Drug Central and the Therapeutic Target Database, incorporating information about drugs and their associated treated symptoms. The data from these two additional sources is adjusted to conform to the original graph structure defined by Monarch.\u003c/p\u003e\n\u003cp\u003eSubsequent to generating the disease-specific KG, a GNN model (Graph- SAGE [10]) is trained on it. The output of this process is a ranked list of pre- dictions, with each entry representing the probability of a link existing between a drug and the target symptom. The higher the score, the greater the likelihood of an actual edge existing between the two nodes, indicating a stronger poten- tial relationship between the drug and the symptom. For more information on \u003cem\u003erd-explainer\u003c/em\u003e, the reader can refer to Perdomo-Quinteiro \u003cem\u003eet al. \u003c/em\u003e[19].\u003c/p\u003e"},{"header":"3 Method","content":"\u003cdiv\u003e\n \u003cp\u003eTo restructure the KG produced by the \u003cem\u003erd-explainer\u003c/em\u003e, we followed relevant FAIR- ification steps: identification of FAIRification objectives, data analysis, concep- tual modelling, and data restructuring (data linkage). Subsequently, we com- pared the output and performance of the GNN model when trained on the original KG and on the CM-based one.\u003c/p\u003e\n \u003cp\u003e \u003cem\u003eIdentification of FAIRification Objectives\u003c/em\u003e The objectives identified in this step focus on making data reusable for drug repurposing applications. These include ‘identifying existing drugs that can be repurposed to treat the symptoms of rare diseases’, as well as the sub-objectives ‘identifying drugs that target genes associated (in)directly with a rare disease’ and ‘identifying drugs known to treat phenotypes associated (in)directly with a rare disease.’\u003c/p\u003e\n \u003cp\u003e \u003cem\u003eData Analysis\u003c/em\u003e In this step, the original KG constructed during the execution of \u003cem\u003erd-explainer\u003c/em\u003e was analysed, and its structure was extracted as it serves as a starting point for conceptual modelling. This structural map (also referred to as metamodel) is illustrated in Fig.\u0026nbsp;1, and an excerpt of it is depicted in Fig.\u0026nbsp;2 for better visualisation.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv\u003e\n \u003cp\u003e \u003cem\u003eConceptual Modelling\u003c/em\u003e The conceptual model for the drug repurposing domain was developed iteratively. This involved designing the domain model using Onto- UML [8], a modelling language based on the Unified Foundational Ontology [7], which supports the creation of ontologically \u003cem\u003ewell-founded conceptual models\u003c/em\u003e, ensuring semantic clarity in representing real-world phenomena. The resulting model was validated with experts through three rounds of validation.\u003c/p\u003e\n \u003cp\u003eAn illustration of the resulting conceptual model is shown in Fig.\u0026nbsp;3. It represents key biological entities and their relationships within the domain of drug repurposing and (rare) diseases. At the core, the model involves entities such as Gene, Variant, Genotype, and Phenotype, which are fundamental to un- derstanding genetic and phenotypic expressions of diseases. A Gene is a collective biological entity that may have interactions (e.g., it interacts with or co-localises\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv\u003e\n \u003cp\u003ewith other genes). An Ortholog, shown as a subtype of Gene, refers to genes in different species that evolved from a common ancestral gene by speciation and typically retain the same function, making them crucial for studying dis- ease mechanisms across species. Variants are specific alterations in a gene, and they can be part of a Genotype, which represents the complete set of an organ- ism’s genetic information. The model shows how Variants and Genotypes \u003cem\u003eexpress\u003c/em\u003e Genes, impacting Biological Processes like Molecular Functions and Cellular Com- ponents. Additionally, Drugs are connected to Diseases and Phenotypes through their treatment relationships (\u003cem\u003eis a substance that treats\u003c/em\u003e), thus targeting Gene Products, which are produced by genes and influence disease-related functions.\u003c/p\u003e\n \u003cp\u003e \u003cem\u003eData Linkage\u003c/em\u003e Following the conceptual modelling step, the initial KG was re- organised according to the elements and relationships defined in the conceptual model from Fig.\u0026nbsp;3. To achieve this, the data-fetching script from \u003cem\u003erd-explainer\u003c/em\u003e was modified to generate the CM-based KG. The mapping from the original to the CM-based KG was manually reviewed by one of the authors and an external bioinformatician. All data used in this study were retrieved from the Septem- ber 2021 version of Monarch on 1 May 2024. Data from Drug Central and the Therapeutic Target Database were also collected on 1 May 2024.\u003c/p\u003e\n \u003cp\u003e \u003cem\u003eGNN prediction and performance assessment\u003c/em\u003e After having both the original and CM-based KGs constructed, we proceeded to train separate GNN models using each KG type. This process was repeated ten times for each KG type to\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv\u003e\n \u003cp\u003ecollect performance metrics that were averaged to ensure a balanced comparison (i.e. AUC-ROC [14], which evaluates a model’s capacity to distinguish between classes, where a higher score implies better classification; F1 score [11], which is the harmonic mean of precision and recall; and Cross Entropy Loss [17], which measures the difference between predicted probabilities and true labels, with lower values indicating a more accurate model performance). At the end of this process, 20 prediction lists are generated: 10 lists from the original KG and 10 from the CM-based KG.\u003c/p\u003e\n \u003cp\u003eNext, to assess the reliability of the ML models, we evaluated the consis- tency of their predictions. This evaluation was motivated by Perdomo \u003cem\u003eet al.\u003c/em\u003e’s observation regarding the challenges of ensuring reproducibility in ML methods. For instance, given the stochastic nature of some components of \u003cem\u003erd-explainer\u003c/em\u003e (e.g. edge2vec [6]), different runs of the pipeline can output different lists of pre- dictions. Thus, a considerably reliable model would produce robust predictions that are not significantly affected by random variation. To measure this, we per- formed pairwise comparisons of the prediction lists separately for each of the two groups of 10 iterations from each KG type. For each pair of prediction lists, we calculated the percentage of overlapping predictions, with a lower overlap indicating less consistency across iterations and greater variability in the ML model’s outcomes.\u003c/p\u003e\n \u003cp\u003e \u003cem\u003eTargeting a rare disease\u003c/em\u003e As previously mentioned, the process described above requires specifying a target disease (i.e. a disease code) when constructing the KGs, meaning the GNN models are trained on disease-specific KGs. To gather more comprehensive insights from our experiments, we applied our method to three different rare diseases: Duchenne Muscular Dystrophy (DMD) [23], Hunt- ington’s Disease (HD) [25], and Osteogenesis Imperfecta (OI) [21]. This resulted in the construction of six distinct KGs—two for each disease: one original KG and one CM-based KG—and enabled disease-specific comparisons.\u003c/p\u003e\n \u003cp\u003eA workflow illustration of the method described in this section, along with the detailed FAIRification objectives, the original KG metamodel, the conceptual model, the data-fetching and training scripts, the performance measurements and resulting predictions are available in the supplementary material.6\u003c/p\u003e\n\u003c/div\u003e"},{"header":"4 Results","content":"\u003cdiv class=\"BlockQuote\"\u003e\n \u003cp\u003eA readily applicable outcome of this work is the conceptual model developed dur- ing our method (Fig. \u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e). Moreover, our assessments indicate that the GNN models trained on both types of KGs achieved comparable performance in met- rics such as AUC and F1 scores. More significantly, the comparison in terms of output reliability reveals that models trained on CM-based KGs produced more consistent predictions (i.e. similar prediction results). Zoomed-in versions of the figures presented in the next subsections are available in the supplementary ma- terial.\u003c/p\u003e\n\u003c/div\u003e\n\u003ch3\u003e4.1 The OntoUML-based conceptual model is reusable\u003c/h3\u003e\n\u003cdiv class=\"BlockQuote\"\u003e\n \u003cp\u003eThe conceptual model designed in this work can be reused in other ML systems, FAIRification processes and extended for various applications in related domains. When comparing the original and restructured KGs (Figs. \u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e and \u003cspan class=\"InternalRef\"\u003e3\u003c/span\u003e), it can be observed that the number of nodes increased from the original KG to the CM-based KG. In contrast, the number of relationships decreased significantly, as some concepts previously defined as relationships in the original KG were transformed into concepts in the restructured version. For instance, the \u003cem\u003ehas phenotype\u003c/em\u003e relationship in the original KG, which linked Gene and Disease, was transformed into a Phenotype concept in the restructured KG.\u003c/p\u003e\n\u003c/div\u003e\n\u003ch3\u003e4.2 Predictive performance is similar\u003c/h3\u003e\n\u003cdiv class=\"BlockQuote\"\u003e\n \u003cp\u003eThe (averaged) training curves of the GNN models are illustrated in Figs. 4, 5, and 6, for DMD, HD, and OI, respectively. High-resolution versions including the loss values are available in the supplementary material. Figures 4a, 5a, and 6a display the training metrics of the GNN models trained on the original KGs, while Figs. 4b, 5b, and 6b show the metrics from the training on the CM- based KGs. Each figure presents the AUC-ROC scores and the Cross Entropy\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv class=\"BlockQuote\"\u003e\n \u003cp\u003e\u003csup\u003e6\u003c/sup\u003e \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://edu.nl/m8xg7\u003c/span\u003e\u003c/span\u003e (scripts) and \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://edu.nl/k4648\u003c/span\u003e\u003c/span\u003e (figures and outputs)\u003c/p\u003e\n \u003cp\u003eLoss of the training processes. When comparing the (a) and (b) versions of each figure, it is important to note that the training curves differ in the total number of epochs as distinct hyperparameter optimisation processes (random search) [\u003cspan class=\"CitationRef\"\u003e2\u003c/span\u003e] were performed for each of the six KGs\u0026ndash;this is motivated by our aim to evaluate the impact of CM-based restructuration on the entire process.\u003c/p\u003e\n \u003cp\u003eOverall, for all models trained on both the original and CM-based KGs, the training process starts with a remarkably high AUC-ROC score for both the training and test sets. However, the improvement from the start to the end of training is minimal. Table \u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e shows a summary of the average AUC-ROC and F1 scores for each case and target rare disease. For DMD, both the AUC-ROC and F1 scores are slightly higher when training the GNN model on the original KG, although the difference is minimal. For HD, training on the original KG resulted in a higher average AUC-ROC, while training on the CM-based KG resulted in a higher F1 score with a significant difference. For OI, the average AUC-ROC and F1 scores were higher when the ML model was trained on the CM-based KG.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv class=\"gridtable\"\u003e\u0026nbsp;\u003ctable id=\"Tab1\" border=\"1\"\u003e\n \u003ccaption language=\"En\"\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003eAUC-ROC and F1 scores for training on original and CM-based KG, for each target rare disease.\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003ccolgroup cols=\"7\"\u003e\u003c/colgroup\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eDisease\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colspan=\"2\"\u003e\n \u003cp\u003eDMD\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colspan=\"2\"\u003e\n \u003cp\u003eHD\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colspan=\"2\"\u003e\n \u003cp\u003eOI\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eKG Type\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eOriginal\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eCM-based\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eOriginal\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eCM-based\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eOriginal\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eCM-based\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eAUC-ROC\u003c/p\u003e\n \u003cp\u003eF1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.977\u003c/strong\u003e\u003c/p\u003e\n \u003cp\u003e\u003cstrong\u003e0.933\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.976\u003c/p\u003e\n \u003cp\u003e0.906\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.978\u003c/strong\u003e\u003c/p\u003e\n \u003cp\u003e0.896\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.967\u003c/p\u003e\n \u003cp\u003e\u003cstrong\u003e0.934\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.9602\u003c/p\u003e\n \u003cp\u003e0.812\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.974\u003c/strong\u003e\u003c/p\u003e\n \u003cp\u003e\u003cstrong\u003e0.915\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n\u003c/div\u003e\n\u003ch3\u003e4.3 Predictive consistency is higher for CM-based KGs\u003c/h3\u003e\n\u003cdiv class=\"BlockQuote\"\u003e\n \u003cp\u003eFigures \u003cspan class=\"InternalRef\"\u003e7\u003c/span\u003e, \u003cspan class=\"InternalRef\"\u003e8\u003c/span\u003e, and 9, illustrate the degree of overlap (expressed as a percentage) between predicted drug-phenotype pairs across all ten runs for the original and CM-based KG for DMD, HD and OI, respectively. Figures \u003cspan class=\"InternalRef\"\u003e7\u003c/span\u003ea, \u003cspan class=\"InternalRef\"\u003e8\u003c/span\u003ea and 9a are derived from the predictions of the GNN model trained on the original KG, whereas those in Fig. \u003cspan class=\"InternalRef\"\u003e7\u003c/span\u003eb, \u003cspan class=\"InternalRef\"\u003e8\u003c/span\u003eb and 9b are derived from the predictions of the GNN model trained on the CM-based KG. The means and median of the overlaps described in Figs. \u003cspan class=\"InternalRef\"\u003e7\u003c/span\u003e, \u003cspan class=\"InternalRef\"\u003e8\u003c/span\u003e, and 9 are summarised in Table \u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e. Given the mean and median values of the overlap of predicted drug-phenotype pairs, it can be observed a higher mean and median for when \u003cem\u003erd-explainer\u003c/em\u003e is applied to the CM-based KG in the experiments conducted for all target rare diseases. The complete lists of predictions are available in the supplementary material.\u003c/p\u003e\n\u003c/div\u003e"},{"header":"5 Discussion","content":"\u003cdiv class=\"BlockQuote\"\u003e\n \u003cp\u003eWe extend our results to propose specific research questions (RQs) to guide fu- ture studies. Within the context of FAIR and FAIRification, exploring these RQs will enhance understanding of the benefits of FAIR principles for ML applica- tions. Additionally, addressing these RQs will support gathering more data to make the conclusions of our work more generalisable.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv class=\"gridtable\"\u003e\u0026nbsp;\u003ctable id=\"Tab2\" border=\"1\"\u003e\n \u003ccaption language=\"En\"\u003e\n \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e\n \u003cdiv class=\"CaptionContent\"\u003e\n \u003cp\u003eSummary of the consistency of predictions of all three experiments. The percentages in parenthesis represent the increase of the CM-based values when compared to original KG ones (e.g. increase in mean from original to CM-based).\u003c/p\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003ccolgroup cols=\"7\"\u003e\u003c/colgroup\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eDisease\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colspan=\"2\"\u003e\n \u003cp\u003eDMD\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colspan=\"2\"\u003e\n \u003cp\u003eHD\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colspan=\"2\"\u003e\n \u003cp\u003eOI\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eKG Type\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eOriginal\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eCM-based\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eOriginal\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eCM-based\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eOriginal\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eCM-based\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003eConsistency\u003c/p\u003e\n \u003cp\u003emean Consistency median\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e38.97\u003c/p\u003e\n \u003cp\u003e39.29\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003e54.1\u003c/strong\u003e\u003c/p\u003e\n \u003cp\u003e(+\u0026thinsp;38.82%)\u003c/p\u003e\n \u003cp\u003e\u003cstrong\u003e53.57\u003c/strong\u003e\u003c/p\u003e\n \u003cp\u003e(+\u0026thinsp;36.35%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e24.27\u003c/p\u003e\n \u003cp\u003e25.29\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003e48.43\u003c/strong\u003e\u003c/p\u003e\n \u003cp\u003e(+\u0026thinsp;99.55%)\u003c/p\u003e\n \u003cp\u003e\u003cstrong\u003e52.87\u003c/strong\u003e\u003c/p\u003e\n \u003cp\u003e(+\u0026thinsp;109.05%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e11.53\u003c/p\u003e\n \u003cp\u003e10.61\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003e39.61\u003c/strong\u003e\u003c/p\u003e\n \u003cp\u003e(+\u0026thinsp;243.54%)\u003c/p\u003e\n \u003cp\u003e\u003cstrong\u003e37.88\u003c/strong\u003e\u003c/p\u003e\n \u003cp\u003e(+\u0026thinsp;257.02%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n\u003c/div\u003e\n\u003cdiv class=\"BlockQuote\"\u003e\n \u003cp\u003e\u003cem\u003eTraining data\u003c/em\u003e When examining the Cross Entropy Loss curve for OI (Fig. 6), it becomes evident that the curve of the ML model trained on the CM-based KG is steeper than that of the ML model trained on the original KG. This may suggest that the ML model trained on the CM-based KG \u0026ldquo;learned more\u0026rdquo; when compared to the one trained on the original KG. Consequently, future research should explore whether restructuring training data according to well-founded conceptual models can enhance learning in ML models (e.g. GNN models) that initially do not perform well when trained on current data. Thus, a related RQ could be: \u003cem\u003e\u0026ldquo;RQ1 - Do conceptual model based data improve the predictive performance of ML algorithms that underperform on current data?\u0026rdquo;\u003c/em\u003e\u003c/p\u003e\n \u003cp\u003eAn example of an experiment to address RQ1 could involve identifying cases where data scientists do not manage to further improve the performance of ML models on specific datasets. In such cases, well-founded conceptual models of the datasets\u0026rsquo; subjects would be designed and used to restructure those datasets. The performance of the ML algorithms would then be tested on these newly restructured datasets.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv class=\"BlockQuote\"\u003e\n \u003cp\u003e\u003cem\u003eReproducibility\u003c/em\u003e When comparing the consistency of predictions from ML models trained on the original and CM-based KGs (Figs. \u003cspan class=\"InternalRef\"\u003e7\u003c/span\u003e, \u003cspan class=\"InternalRef\"\u003e8\u003c/span\u003e, and 9), it is observed that ML models trained on the latter produce more stable predictions than those trained on the former, as indicated by the average overlap among the 10 lists of predictions generated for each case (Table \u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e). This suggests that training ML models on data structured according to well-founded conceptual models enhances the consistency of predictions, thereby reducing the randomness of the results. Thus, a research question related to this aspect could be: \u003cem\u003e\u0026ldquo;RQ2 - Do conceptual model based data contribute to the consistency of predictions of ML models?\u0026rdquo;\u003c/em\u003e\u003c/p\u003e\n \u003cp\u003eTo address RQ2, it would be necessary to replicate the experiments conducted in our study in a systematic manner, involving various algorithms and datasets. This approach will ensure that the results are statistically significant and that the conclusions can be generalised and reproduced across different scenarios.\u003c/p\u003e\n \u003cp\u003e\u003cem\u003eDesign of AI systems\u003c/em\u003e During the application of our method, it was observed that conceptual models improved communication, understanding, and task exe- cution among different stakeholders. A pertinent research question arising from this observation is whether data scientists can enhance their performance in fea- ture engineering, ML model selection, and parameter tuning due to a better understanding of the domain data (provided by conceptual modelling tasks). This leads to the final research question: \u003cem\u003e\u0026ldquo;RQ3 - Do conceptual models support stakeholders in the design of AI systems?\u0026rdquo;\u003c/em\u003e\u003c/p\u003e\n \u003cp\u003eTo test RQ3, a controlled experiment could be conducted in which one group of data scientists and developers is tasked with directly designing and implementing an ML pipeline, while another group is required to first create an OntoUML model before proceeding with the design and implementation of the ML pipeline. The results of these two groups would then be compared to evaluate the impact of conceptual modelling on the design and execution of AI systems. Such an experiment could also test whether the conceptual modelling task facilitates communication between data scientists and domain experts.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv class=\"BlockQuote\"\u003e\n \u003cp\u003eIt should be emphasised that the initial findings of this work are based on the assumption that future experiments will employ conceptual models that are\u003c/p\u003e\n \u003cp\u003e(i) constructed using a well-founded modelling language or ontology as a foun- dation, and (ii) thoroughly validated by domain experts. Finally, it is important to highlight that the RQs described above are formulated from our initial ex- plorations. While some are based on subtle differences in the results (e.g., AUC curves), they remain valuable for further investigation, as they may reveal more significant and impactful differences in other ML applications, particularly those involving large KGs.\u003c/p\u003e\n\u003c/div\u003e"},{"header":"6\tLimitations and future work","content":"\u003cp\u003eReproducibility and generalisability are the primary limitations of our results. Reproducibility, a well-recognised challenge in ML [1], remains difficult in this context. While our findings demonstrate that CM-based KGs lead to more con- sistent GNN models, achieving identical results to those presented in Section 4 is challenging due, for example, to the inherent randomness in certain compo- nents of the \u003cem\u003erd-explainer\u0026nbsp;\u003c/em\u003epipeline.\u0026nbsp;To\u0026nbsp;mitigate\u0026nbsp;this\u0026nbsp;issue,\u0026nbsp;we\u0026nbsp;ran\u0026nbsp;the\u0026nbsp;training and output generation ten times for each case and target disease, averaging the scores to reduce the impact of variability.\u003c/p\u003e\n\u003cp\u003eAlthough we synthesised our findings into additional RQs to guide future research, we have not yet tested our findings sufficiently to draw generalisable conclusions. This limitation arises from the fact that our study focused on a single type of ML method, using the same set of data sources within a specific domain. Therefore, it is crucial to investigate the impact of conceptual models in different domains and with other types of ML methods, as well as with different types of data. For example, while our results may be relevant to graph data and AI methods designed for such data, the application of our method to other data types may not lead to meaningful differences in results.\u003c/p\u003e"},{"header":"7 Related works","content":"\u003cp\u003e \u003cdiv class=\"BlockQuote\"\u003e \u003cp\u003eThe use and impact of conceptual models and ontologies in AI has been a topic of discussion in the literature. Various studies examine how these artefacts can be applied in the field, such as to enhance outcomes and the design of AI systems. At this stage, we view related works as complementary efforts toward a shared ultimate goal: enhancing the understanding and application of ontologies in AI, and vice versa.\u003c/p\u003e \u003cp\u003eConfalonieri \u0026amp; Guizzardi [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e] outlined the different roles that ontologies and ontology-based conceptual models can play for neuro-symbolic AI systems from three key perspectives: reference modelling, common sense reasoning, and knowl- edge refinement and complexity management. Similarly, Maa\u0026szlig; \u0026amp; Storey [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e] ex- plored the benefits and synergies of integrating conceptual modelling with ML and proposed a framework that uses conceptual modelling to support the design and development of ML solutions. Lukyanenko \u003cem\u003eet al.\u003c/em\u003e [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e] explored how concep- tual modelling can address challenges in extracting insights from large datasets with machine learning. They showed that conceptual modelling aids various ML\u003c/p\u003e \u003cp\u003eproject phases: defining goals in the business understanding phase, modelling data and identifying quality issues in the data understanding phase, supporting attribute selection and transformation in data preparation, enhancing ML al- gorithm effectiveness with domain knowledge, improving result interpretability, and documenting process changes during deployment.\u003c/p\u003e \u003c/div\u003e \u003c/p\u003e"},{"header":"8 Final Remarks","content":"\u003cp\u003e \u003cdiv class=\"BlockQuote\"\u003e \u003cp\u003eThis work presented an exploratory study to investigate the impact of restruc- turing knowledge graphs using conceptual models\u0026ndash;a step of FAIRification\u0026ndash;on the performance of a GNN algorithm. We tested the GNN model\u0026rsquo;s behaviour when applied to both original and CM-based KGs across three different rare diseases targeted for drug repurposing. The initial results provided valuable insights and led to the formulation of additional refined research questions for future investi- gation, focusing on areas such as supporting the ML design process, improving predictive performance, and enhancing the consistency of predictions.\u003c/p\u003e \u003cp\u003eThe most prominent results of this work relate to the consistency of the pre- dictions. We evaluated the prediction lists generated across ten runs of the GNN method on both the original and CM-based KGs for each target disease. This evaluation involved pairwise comparisons measuring the overlap of predictions in each list. In all cases, the prediction lists generated from training the GNN model on the CM-based KGs were more consistent than those generated from the training on the original KGs.\u003c/p\u003e \u003cp\u003eFor FAIR and FAIRification, the initial findings herein presented serve as a proof-of-concept of the benefits of applying the FAIR principles to ML appli- cations. Our results demonstrate that FAIR data, structured using well-defined conceptual models, have the potential to enhance the consistency of ML out- puts without negatively impacting performance. Additionally, other elements of FAIR further enhance ML design and deployment. For instance, FAIR metadata improve resource discovery (e.g., finding data for training) and simplifies reuse by clearly specifying the conditions under which the data can be reused.\u003c/p\u003e \u003cp\u003eML and FAIR research are rapidly evolving fields. Our work contributes to this progress by exploring synergies between conceptual models, FAIR principles, and ML. As the next step, we aim to expand our research by applying our method to new domains, diverse data sources, and a broader range of ML approaches for more robust and generalisable results.\u003c/p\u003e \u003c/div\u003e \u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eAcknowledgments\u003c/h2\u003e \u003cp\u003eThis initiative has received funding from the European Union\u0026rsquo;s Horizon 2020 research and innovation programme under grant agreements N\u0026deg;825575 and N\u0026deg; 101159589, and the EU4Health Programme under grant agreement N\u0026deg; 101129863.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eAlbertoni, R., Colantonio, S., Skrzypczyński, P., Stefanowski, J.: Reproducibility of machine learning: Terminology, recommendations and open issues. arXiv preprint arXiv:2302.12691 (2023)\u003c/li\u003e\n\u003cli\u003eBergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. Journal of machine learning research \u003cstrong\u003e13\u003c/strong\u003e(2) (2012)\u003c/li\u003e\n\u003cli\u003eBrnabic, A., Hess, L.M.: Systematic literature review of machine learning methods used in the analysis of real-world data for patient-provider decision making. BMC Medical Informatics and Decision Making \u003cstrong\u003e21\u003c/strong\u003e(1), 54 (2021)\u003c/li\u003e\n\u003cli\u003eChen, X., Ji, Z.L., Chen, Y.Z.: Ttd: therapeutic target database. Nucleic acids research \u003cstrong\u003e30\u003c/strong\u003e(1), 412\u0026ndash;415 (2002)\u003c/li\u003e\n\u003cli\u003eConfalonieri, R., Guizzardi, G.: On the multiple roles of ontologies in explanations for neuro-symbolic ai. Neurosymbolic Artificial Intelligence (Preprint), 1\u0026ndash;15\u003c/li\u003e\n\u003cli\u003eGao, Z., Fu, G., Ouyang, C., Tsutsui, S., Liu, X., Yang, J., Gessner, C., Foote, B., Wild, D., Ding, Y., et al.: edge2vec: Representation learning using edge semantics for biomedical knowledge discovery. BMC bioinformatics \u003cstrong\u003e20\u003c/strong\u003e, 1\u0026ndash;15 (2019)\u003c/li\u003e\n\u003cli\u003eGuizzardi, G., Botti Benevides, A., Fonseca, C.M., et al.: UFO: Unified Founda- tional Ontology. Applied Ontology \u003cstrong\u003e17\u003c/strong\u003e(1), 167\u0026ndash;210 (2022)\u003c/li\u003e\n\u003cli\u003eGuizzardi, G., Fonseca, C.M., Benevides, A.B., et al.: Endurant types in ontology- driven conceptual modeling: Towards OntoUML 2.0. In: Conceptual Modeling. ER 2018. vol. 11157, p. 136\u0026ndash;150. Springer (2018)\u003c/li\u003e\n\u003cli\u003eGuizzardi, G., Wagner, G., Almeida, J.P.A., Guizzardi, R.S.: Towards ontological foundations for conceptual modeling: The unified foundational ontology (ufo) story. Applied ontology (2015)\u003c/li\u003e\n\u003cli\u003eHamilton, W., Ying, Z., Leskovec, J.: Inductive representation learning on large graphs. Advances in neural information processing systems \u003cstrong\u003e30 \u003c/strong\u003e(2017)\u003c/li\u003e\n\u003cli\u003eHand, D.J., Christen, P., Kirielle, N.: F*: an interpretable transformation of the f-measure. Machine Learning \u003cstrong\u003e110\u003c/strong\u003e(3), 451\u0026ndash;456 (2021)\u003c/li\u003e\n\u003cli\u003eHanson, B., Stall, S., Cutcher-Gershenfeld, J., Vrouwenvelder, K., Wirz, C., Rao, Y., Peng, G.: Garbage in, garbage out: mitigating risks and maximizing benefits of ai in research. Nature \u003cstrong\u003e623\u003c/strong\u003e(7985), 28\u0026ndash;31 (2023)\u003c/li\u003e\n\u003cli\u003eJacobsen, A., de Miranda Azevedo, R., Juty, N., Batista, D., Coles, S., Cornet, R., Courtot, M., Crosas, M., Dumontier, M., Evelo, C.T., et al.: Fair principles: interpretations and implementation considerations (2020)\u003c/li\u003e\n\u003cli\u003eJanssens, A.C.J., Martens, F.K.: Reflection on modern methods: Revisiting the area under the roc curve. International journal of epidemiology \u003cstrong\u003e49\u003c/strong\u003e(4), 1397\u0026ndash;1403 (2020)\u003c/li\u003e\n\u003cli\u003eLukyanenko, R., Castellanos, A., Parsons, J., Chiarini Tremblay, M., Storey, V.C.: Using conceptual modeling to support machine learning. In: Information Systems Engineering in Responsible Information Systems: CAiSE Forum 2019, Rome, Italy, June 3\u0026ndash;7, 2019, Proceedings 31. pp. 170\u0026ndash;181. Springer (2019)\u003c/li\u003e\n\u003cli\u003eMaass, W., Storey, V.C.: Pairing conceptual modeling with machine learning. Data \u0026amp; Knowledge Engineering \u003cstrong\u003e134\u003c/strong\u003e, 101909 (2021)\u003c/li\u003e\n\u003cli\u003eMao, A., Mohri, M., Zhong, Y.: Cross-entropy loss functions: Theoretical analysis and applications. In: International conference on Machine learning. pp. 23803\u0026ndash; 23828. PMLR (2023)\u003c/li\u003e\n\u003cli\u003eOrtigossa, E.S., Gon\u0026ccedil;alves, T., Nonato, L.G.: Explainable artificial intelligence (xai)\u0026ndash;from theory to methods and applications. IEEE Access (2024)\u003c/li\u003e\n\u003cli\u003ePerdomo-Quinteiro, P., Wolstencroft, K., Roos, M., Queralt-Rosinach, N.: Knowledge graphs and explainable ai for drug repurposing on rare diseases. bioRxiv (2024). https://doi.org/10.1101/2024.10.17.618804, https://www.biorxiv.org/content/early/2024/10/17/2024.10.17.618804\u003c/li\u003e\n\u003cli\u003ePushpakom, S., Iorio, F., Eyers, P.A., Escott, K.J., Hopper, S., Wells, A., Doig, A., Guilliams, T., Latimer, J., McNamee, C., et al.: Drug repurposing: progress, chal- lenges and recommendations. Nature reviews Drug discovery \u003cstrong\u003e18\u003c/strong\u003e(1), 41\u0026ndash;58 (2019)\u003c/li\u003e\n\u003cli\u003eRauch, F., Glorieux, F.H.: Osteogenesis imperfecta. The Lancet \u003cstrong\u003e363\u003c/strong\u003e(9418), 1377\u0026ndash; 1385 (2004)\u003c/li\u003e\n\u003cli\u003eShefchek, K.A., Harris, N.L., Gargano, M., Matentzoglu, N., Unni, D., Brush, M., Keith, D., Conlin, T., Vasilevsky, N., Zhang, X.A., et al.: The monarch initia- tive in 2019: an integrative data and analytic platform connecting phenotypes to genotypes across species. Nucleic acids research \u003cstrong\u003e48\u003c/strong\u003e(D1), D704\u0026ndash;D715 (2020)\u003c/li\u003e\n\u003cli\u003eSzigyarto, C.A.K., Spitali, P.: Biomarkers of duchenne muscular dystrophy: current findings. Degenerative neurological and neuromuscular disease pp. 1\u0026ndash;13 (2018)\u003c/li\u003e\n\u003cli\u003eUrsu, O., Holmes, J., Knockel, J., Bologa, C.G., Yang, J.J., Mathias, S.L., Nelson, S.J., Oprea, T.I.: Drugcentral: online drug compendium. Nucleic acids research p. gkw993 (2016)\u003c/li\u003e\n\u003cli\u003eWalker, F.O.: Huntington\u0026rsquo;s disease. The Lancet \u003cstrong\u003e369\u003c/strong\u003e(9557), 218\u0026ndash;228 (2007)\u003c/li\u003e\n\u003cli\u003eWilkinson, M.D., Dumontier, M., Aalbersberg, I.J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.W., da Silva Santos, L.B., Bourne, P.E., et al.: The fair guiding principles for scientific data management and stewardship. Scien- tific data \u003cstrong\u003e3\u003c/strong\u003e, 1\u0026ndash;9 (2016)\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[{"identity":"5cfec493-5f6a-46f1-a273-d4281de5b213","identifier":"10.13039/100010661","name":"Horizon 2020 Framework Programme","awardNumber":"825575","order_by":0},{"identity":"b0226a74-a439-4c65-bde2-c4b4d4e08eed","identifier":"10.13039/100010661","name":"Horizon 2020 Framework Programme","awardNumber":"101159589","order_by":1}],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"Leiden University Medical Center","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Conceptual Model, Machine Learning, FAIR Principles, Knowledge Graphs","lastPublishedDoi":"10.21203/rs.3.rs-5622649/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5622649/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThis paper investigates the impact of restructuring knowl- edge graphs (KGs) with well-founded conceptual models to improve ma- chine learning (ML) predictions, particularly in drug repurposing appli- cations. These conceptual models were developed using OntoUML, which is grounded in the Unified Foundational Ontology, and were constructed following an established workflow for data FAIRification\u0026ndash;a process aimed at making data more Findable, Accessible, Interoperable, and Reusable. We compared the performance of a Graph Neural Network model trained on original public KGs with models trained on the same restructured KGs. Our results indicate that while the ML model classification perfor- mance (measured in terms of accuracy and error metrics) remains similar for both, the models trained on restructured KGs produce more consis- tent predictions, reducing variability across multiple runs. These findings suggest that restructuring KGs using well-founded conceptual models can enhance the reliability of ML predictions without compromising model performance. We conclude by proposing future research directions to fur- ther explore the potential of conceptual models and FAIR principles in improving ML.\u003c/p\u003e","manuscriptTitle":"Restructuring knowledge graphs with conceptual models: implications for machine learning predictions in drug repurposing","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-02-07 09:41:16","doi":"10.21203/rs.3.rs-5622649/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"4226f17f-60d1-4273-8bac-be15ef8770b6","owner":[],"postedDate":"February 7th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":41450896,"name":"Bioinformatics"},{"id":41450897,"name":"Artificial Intelligence and Machine Learning"}],"tags":[],"updatedAt":"2025-02-07T09:41:16+00:00","versionOfRecord":[],"versionCreatedAt":"2025-02-07 09:41:16","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5622649","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5622649","identity":"rs-5622649","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}