EXPLORE: learning interpretable rules for patient-level prediction

EXPLORE: learning interpretable rules for patient-level prediction | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article EXPLORE: learning interpretable rules for patient-level prediction Aniek F. Markus, Jan A. Kors, Egill A. Fridgeirsson, Katia M.C. Verhamme, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5804837/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Objective We investigate whether a trade-off occurs between predictive performance and model interpretability in real-world health care data and illustrate how to develop clinically optimal decision rules by learning under constraints with the Exhaustive Procedure for Logic-Rule Extraction (EXPLORE) algorithm. Methods We enhanced EXPLORE’s scalability to enable its use with real-world datasets and developed an R package that generates simple decision rules. We compared EXPLORE’s performance to 7 state-of-the-art model algorithms across 5 prediction tasks using data from the Dutch Integrated Primary Care Information (IPCI) database. Additionally, we characterized EXPLORE’s space of near-optimal models (i.e. Rashomon set) and conducted experiments on incorporating domain knowledge and improving existing models. Results The prediction models developed using LASSO, RandomForest, and XGBoost consistently performed best in terms of AUROC, followed by DecisionTree and EXPLORE. However, the decision rules generated by EXPLORE are much simpler (at most 5 predictors) than the aforementioned. GOSDT-G, IHT, and RIPPER performed worse. Moreover, we demonstrated that EXPLORE’s Rashomon set is very large (1,381 − 20,320 models) with a large variability in both the generalizability and model diversity. We then showed there is a potential to find more clinically optimal decision rules using EXPLORE by incorporating domain knowledge (age/sex and task-specific features) or improving existing models (the CHADS 2 score). Conclusions Our study shows that more complex models generally outperform simpler ones, confirming the expected interpretability-performance trade-off, although it varies in strength across prediction tasks. EXPLORE’s ability to learn under constraints is valuable for generating clinically optimal decision rules. Clinical prediction modelling Optimal decision rule Interpretability-performance trade-off Data-and knowledge-driven learning Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 1. BACKGROUND Rule-based methods have been among the most popular classification methods in machine learning since the 1960s ( 1 ). As predictions of these models are the result of a series of relatively simple computations, small rule-based models are often considered to be interpretable ( 2 ). Interpretable models allow users to understand, verify, and improve model output ( 3 ), which is often demanded when the cost of misclassification is high (e.g. in health care). In addition, these models may be easier to implement as they can be used without system integration and face less technical challenges during deployment due to their smaller size. Moreover, rule-based methods show close resemblance to existing clinical workflows such as knowledge-based expert systems and clinical pathways. Despite these advantages, rule-based methods are less frequently used to develop clinical prediction models compared to other methods like regression analysis ( 4 ). In this paper we aim to develop clinically optimal decision rules using electronic health record (EHR) data. The concept of optimality in clinical prediction modelling goes beyond the traditional mathematical definition of a feasible solution that minimizes (or maximizes) a given objective function. Model performance is important, but for a model to be successful in practice more is needed. First, a clinically optimal model is interpretable, i.e., as simple as possible in both model form and size as well as the features included. A model (partially) aligning with prior knowledge might be better generalizable and more readily adopted in practice. Second, a clinically optimal model might need to satisfy certain constraints (e.g. a minimum level of specificity). Third, a clinically optimal model is stable when updated. Similar features are included in the model and predictions for similar examples are similar, all while achieving performance that is (close to) the ‘best’ possible for the given prediction task. The Exhaustive Procedure for Logic-Rule Extraction (EXPLORE) algorithm ( 5 ) has the necessary features to develop such clinically optimal decision rules: it produces small decision rules, has optimality guarantees, and allows the user to specify global model constraints. The contributions of this paper are as follows: We improve EXPLORE ( 5 ) in terms of scalability, making it accessible and executable for real-world sized datasets. We demonstrate this by developing decision rules for five prediction tasks (~ 50 features, ~ 4,000-200,000 observations) using Dutch general practitioner (GP) data. We investigate whether a trade-off occurs between predictive performance and model interpretability in real-world health care data by comparing EXPLORE’s performance to seven other commonly used state-of-the-art modelling methods in a large-scale empirical evaluation. We illustrate how to develop clinically optimal decision rules by learning under constraints with EXPLORE. 2. RELATED WORK 2.1. Optimal rule-based model algorithms Rule-based methods are a popular choice to develop interpretable models as they are human understandable (consist of a collection of statements connected using “IF-THEN”, “OR”, and “AND”) and allow to model categorical data with non-linear relationships. Examples include decision trees, decisions lists (i.e. ordered sets of rules or one-sided decision trees), and decision sets (i.e. unordered sets of rules). Historically, greedy approaches have often been used to grow decision trees (e.g. CART, C4.5) and decision lists/sets (e.g. RIPPER, C4.5R), as decision tree optimization is known to be an NP-complete problem ( 6 ). These approaches split the optimization problem into steps and search for the optimal solution at every split. Although this makes their computation more tractable, the disadvantage is that there are no optimality guarantees for their final performance. In case of bad performance, it is therefore unclear whether this is due to model choice or poor optimization. In recent years several methods have been proposed to grow optimal decision trees . For example, DL8.5 ( 7 ) combines dynamic programming on the space of the decision trees and branch-and-bound optimization, BinOCT ( 8 ) uses a novel binary linear program formulation for optimization, and GOSDT ( 9 ) uses dynamic programming with custom bounds. However, DL8.5 and BinOCT loose optimality when using bucketization to handle continuous features. Although less work focuses on optimal decision rules , some methods with optimization guarantees have been proposed: EXPLORE ( 5 ) uses a branch-and-bound approach for efficient exhaustive search, IDS ( 10 ) relies on submodular optimization, CG ( 11 ) uses a column generation algorithm, and LIBRE ( 12 ) creates an ensemble of weak learners. The exhaustive search performed by EXPLORE offers several advantages over other methods as it allows users to specify global constraints on the model (e.g. model size, included features) or its performance (e.g. minimum sensitivity and specificity). Earlier work investigating the performance of EXPLORE on UCI datasets has shown promising results ( 5 ), but is limited as the studied prediction tasks do not match the complexity of real-world data. 2.2. Interpretability-performance trade-off Ideally, we want models to produce the best possible predictions, while being simple enough to be understood by users. The interpretability-performance trade-off refers to the general assumption that increased model flexibility contributes to a better model fit (increasing performance) at the cost of additional complexity (reducing interpretability). Rudin ( 13 ) raised important awareness that this trade-off might be a myth and stressed the importance of using interpretable models especially for high-stake decisions. She argued that especially for structured data with meaningful features increased interpretability does not necessarily lead to a loss of performance. There is a growing body of research questioning the extent to which this trade-off occurs ( 13 – 15 ). Various studies have shown examples where complex models do not outperform simpler models ( 16 – 20 ). Furthermore, experiments in various domains show there might be multiple equally-good performing models, leading some to suggest that interpretable models are likely to exist among them ( 21 ). The extent of the trade-off might ultimately depend on the data characteristics ( 18 ) and used metrics ( 14 ). Also for medical data, it is often assumed that this trade-off exists ( 22 – 24 ). 3. METHODS 3.1. EXPLORE R package We developed the EXPLORE R package ( https://github.com/mi-erasmusmc/Explore ), which implements the EXPLORE algorithm ( 5 ) that generates optimal decision rules of specified length in disjunctive normal form (DNF). A formula in DNF is a disjunction (OR, ∨) of terms, where each term is a conjunction (AND, ∧) of literals, i.e. feature-operator-value triples (e.g. age > 40). The rules are systematically generated using three nested loops that evaluate the next term tuple (i.e. ordered list of term sizes), feature-operator pair, and threshold value, respectively (see Fig. 1 ). For example, when creating a rule with maximum length 3 for a dataset with 10 binary predictors EXPLORE generates 5540 candidate rules. For details on the original algorithm we refer to the original publication ( 5 ). We made two main contributions to improve EXPLORE’s scalability. First, we further optimized the generation of decision rules for binary data. In particular, we show that it is possible to simplify rules including a term of size 1 to a rule of shorter length using the redundancy law of Boolean algebra (see Appendix A.1). This reduces the number of feature-operator pairs that need to be generated, leading to a reduction in the number of fully instantiated rules of 7.0% for rule length 3 to 20.5% for rule length 5. Second, we parallelized EXPLORE using the RcppParallel package. Using the improved version of EXPLORE we reduced average computation time by 92.2% for rule length 4 (see Appendix A.2). 3.2. Dataset We used real-world health data from the Dutch Integrated Primary Care Information (IPCI) database ( 25 ). This database contains longitudinal, routinely collected data from computer-based patient records of around 350 GP practices throughout the Netherlands. In total, the database currently (January 1, 2024) contains 2.87 million patient records with a median follow-up duration of 4.7 years. The number of active patients is 1.3 million, which comprises about 8% of the Dutch population (~ 17 million). Data include patient demographics, information about contacts with GPs, symptoms, diagnoses, laboratory and clinical measurements, prescriptions, and information on use of secondary care. The IPCI database has been mapped to the Observational Medical Outcomes Partnership Common Data Model (OMOP CDM), which enables standardized extraction and analysis of health care data. This study was approved by the IPCI Governance Board (number 3/2023). 3.3. Prediction tasks We selected five prediction tasks for evaluation, for their definition see Table 1 . These tasks represent a range of different target populations (generic and disease specific), time-at-risks (from 1 month up to 5 years), and outcome rates. Furthermore, these prediction tasks have a clear clinical use case and have been more commonly investigated in the literature: hospital readmission ( 26 , 27 ), end-of-life care ( 28 , 29 ), asthma exacerbations ( 30 , 31 ), mortality in chronic obstructive pulmonary disease (COPD) ( 32 ), and cardiovascular disease in type-2 diabetes mellitus (T2DM) ( 33 ). We restricted all target populations to adults (age > = 18) and include events in the last five years before database end minus the time-at-risk period (e.g. for hospital readmission, cohort entry date after December 1, 2018). Table 1 Specification of prediction tasks. Generic prediction question has the form ( 34 ): “among *target population*, which patients will develop *outcome of interest* during *time-at-risk period*?” Task Target population Outcome of interest Time-at-risk period Hospital readmission Adult patients discharged from hospital Hospital admission 1 month End-of-life care GP visit of older patients (60+) with a prior diagnosis of heart failure, COPD, or cancer End-of-life conversation 6 months Asthma exacerbations Adult patients with new asthma diagnosis receiving medication Exacerbation (as defined by 3–30 days of systemic corticosteroids) 2 years Mortality in COPD Patients with new COPD diagnosis (40+) All-cause mortality 2 years Cardiovascular disease in T2DM Adult patients with new T2DM diagnosis Heart failure or stroke 5 years 3.4. Model development We used the PatientLevelPrediction R package ( 34 ) to develop prediction models. This tool provides a streamlined pipeline to develop clinical prediction models in line with current best practices using OMOP CDM data. The tool extracts the necessary information from the database based on the specified prediction task, splits the data into a training (75%) and test set (25%), constructs and pre-processes the covariates (see Section 3.5), fits models using different modelling algorithms on the training set (see Section 3.6), and performs internal validation of the model on the test set (see Section 3.7). 3.5. Candidate predictors The predictors included for model development were patient demographics (i.e. age and sex) and binary indicators for the presence of 47 clinical events in the 365 days prior to the index date. The set of clinical events - including common medical conditions and drug exposures - is based on the constrained set of clinically meaningful predictors identified by Reps, Wong ( 35 ) to achieve minimal performance loss across a variety of prediction tasks (when compared to a large set of predictors). We removed predictors that occurred in less than 0.1 percent of the target population. 3.6. Model algorithms In addition to EXPLORE, we investigated 7 machine learning algorithms that are commonly used in clinical prediction modelling or rule-based learning: DecisionTree ( 36 ), GOSDT-G ( 37 ), Iterative Hard Thresholding (IHT) ( 38 ), LASSO ( 39 ), RandomForest ( 40 ), RIPPER ( 41 ), and XGBoost ( 42 ). We used 3-fold cross validation to optimize the hyperparameters, including a range of values for each model algorithm to allow for varying levels of model complexity. For details on used libraries and hyperparameters we refer to Appendix B. 3.7. Model evaluation Predictive performance The area under the receiver operating characteristic (AUROC) curve was used to evaluate the ability of the model to distinguish between the classes. Some of the included model algorithms predict classes instead of probabilities (DecisionTree, EXPLORE, GOSDT-G, RIPPER). These models do not allow to vary the risk threshold and hence have a fixed number of true/false positives. EXPLORE’s ability to learn under constraints opens possibilities to evaluate its performance in a similar way as models predicting probabilities (despite it predicting classes). Details on how to create the AUROC curve for EXPLORE by developing models for different constraints on sensitivity and specificity can be found in Appendix C. Interpretability Model size was used as a proxy for model interpretability ( 17 , 43 ). For EXPLORE and RIPPER model size is measured by the total length of rules. In the case of tree-based algorithms like DecisionTree, GOSDT-G, RandomForest and XGBoost, we consider the total number of nodes in the tree(s). For LASSO and IHT, we use the number of non-zero coefficients in the model. As various proxies can be chosen for each model algorithm, we performed a sensitivity analysis by counting the number of unique features used per model. While the first metric captures to the number of computations required for model application and aims to represent the complexity of the model’s decision-making process, the second metric represents the amount of information a clinician must collect (e.g. by asking questions) to use the model. 3.8. Experimental setup We developed a total of 40 models (8 model algorithms x 5 prediction tasks) to evaluate the performance of EXPLORE and more generally investigate the existence of the interpretability-performance trade-off. In addition, we conducted experiments to illustrate how EXPLORE’s learning under constraints can be used to develop clinically optimal decision rules. Quantifying the search space of good candidate models The phenomenon that different prediction models relying on different features can perform (almost) equally well is referred to as the Rashomon effect ( 44 ). The Rashomon set is defined as the collection of near-optimal models, and its size can be measured by comparing it to the total number of possible models, resulting in the Rashomon ratio ( 21 ). Although the Rashomon set and ratio are difficult to compute (exactly) for most model classes (due to their hypothesis space being infinite), these can easily be computed for EXPLORE using its ability to learn under constraints. We constructed the Rashomon set for each prediction task by setting a global model constraint at 99% of the maximum balanced accuracy (i.e. performance of the best model). We further examined the Rashomon set in terms of generalizability (i.e. whether training performance transfers to the test set) and model diversity (i.e. frequency of included features and variation using the model stability metric of Nogueira, Sechidis ( 45 ))). Incorporating domain knowledge and improving existing models Next, we investigated whether we can enforce constraints to include clinically relevant features for the prediction tasks without losing performance. Two cases were considered: a) including age/sex and b) two task-specific features selected based on clinical knowledge, the remaining predictors were in both cases selected by EXPLORE. Furthermore, we investigated if we can improve the performance of an existing model, the CHADS 2 score ( 46 ), which estimates the risk of stroke within 1 year for patients with atrial fibrillation. The original CHADS 2 score includes the following covariates: congestive heart failure history, hypertension history, age, diabetes mellitus history, stroke or transient ischemic attack (TIA) symptoms. We compared the prediction of the original model score with two updated versions: a) we retrained the model using EXPLORE with the same set of covariates, and b) allowed EXPLORE to add 4 additional covariates on top of the CHADS 2 score (included as numerical feature). The full code to run our experiments (including all cohort definitions) is available on GitHub: https://github.com/mi-erasmusmc/ExploreExperiments . Experiments can be repeated on (other) databases mapped to the OMOP CDM. 4. RESULTS 4.1. Baseline characteristics Table 2 presents a summary of the five prediction tasks. As shown the experiments include a variety of tasks with different target population sizes (4,002–230,952) and outcome rates (0.9% – 29.1%), demographics vary across tasks as expected. An overview of all clinical characteristics is included in Appendix D. Table 2 Characteristics of the prediction tasks. Hospital readmission End-of-life care Asthma exacerbations Mortality in COPD Cardiovascular disease in T2DM Target population size 75,375 230,952 4,002 4,094 6,634 Age, mean ± SD 63.6 ± 17.9 71.6 ± 8.3 48.3 ± 17.7 67.1 ± 11.1 60.2 ± 12.4 Birth sex, % female 51.9 52.8 61.2 47.5 44.5 Number of outcomes (% of population) 7,512 (10.0) 2,171 (0.9) 1,164 (29.1) 495 (12.1) 458 (1.6) 4.2. Performance comparison of model algorithms The maximum AUROC value across prediction tasks and model algorithms ranges from 0.65 (for Hospital Readmission) to 0.77 (for Mortality in COPD) on the test set. Figure 2 shows the difference in the performance of the different model algorithms based on discrimination. AUROC curves and values are included in Appendix D. Across the prediction tasks, LASSO, RandomForest, and XGBoost resulted in models with the best predictive performance. These models show superior performance to the other algorithms, but there is no clear winner among them. The predictive performance in terms of AUROC value for EXPLORE ranged between 0.61 and 0.71 across the prediction tasks. GOSDT-G, and IHT are sometimes close in performance to EXPLORE, but perform worse on average and show a larger variability between tasks. The performance of GOSDT-G may be suboptimal due to the range of selected hyperparameters, which were chosen to remain computationally feasible for our size dataset. DecisionTree performed on average slightly better in terms of AUROC than EXPLORE, whereas RIPPER showed consistently poor performance. 4.3. Model complexity and its relation to predictive performance We also investigated the relation with model interpretability. The most interpretable models (model size across tasks indicated between brackets) were found by EXPLORE ( 4 – 5 ), GOSDT-G ( 2 – 3 ), IHT ( 4 – 9 ), and RIPPER ( 5 – 21 ). DecisionTree ( 21 – 31 ), LASSO ( 24 – 39 ), XGBoost (259–3935), and RandomForest (628-35028) resulted in more complex models. For a complete overview of model sizes we refer to Appendix D. Figure 3 shows the relation between interpretability (defined here as the inverse of model size) and performance. A vertical (or horizontal) line indicates no interpretability-performance trade-off. We observe a strong variability in the performance of simple models, while the more complex models show a performance increase that is accompanied by a large reduction in interpretability. These findings confirm the expected trade-off between model performance and interpretability. However, this observed trade-off varies across prediction tasks. For example, the observed trade-off is stronger for Cardiovascular disease in T2DM than for Asthma exacerbations. In the latter case, a more interpretable model can be chosen with less reduction in performance. Overall, trade-off patterns were found to be similar in the training and test set. Similar results were observed for the sensitivity analysis counting the number of unique features used per model (Appendix D). 4.4. Quantifying the search space of good candidate models The number of models with less than 1% deviation from the best performance ranges from 153–675 for maximum rule length 4 and 1,381 − 20,320 for maximum rule length 5. Hence, there are many good candidate models for each prediction task. Figure 4 a shows the performance of all models in the Rashomon set with maximum rule length 5 compared to the best model (i.e. the model that would be selected during standard model development). For some tasks the best model in the training set is among the best performing on the test set (e.g. End-of-life care and Hospital Readmission), but for the other tasks there exist models in the Rashomon set with much better generalization performance. Figure 4 b shows there is a large number of features that are included in one of the models in the Rashomon set and only a handful that are frequently occurring (e.g. age is always included for End-of-life care, Mortality in COPD, and Cardiovascular disease in T2DM). Most features (84–90%) are occurring in less than 10% of the unique feature sets and only 4–6% of the features in more than 75%. According to the stability metric of Nogueira, Sechidis ( 45 ), there is most stability in the Rashomon set of Mortality in COPD (0.58) and least for End-of-life care (0.36). 4.5. Incorporating domain knowledge and improving existing models The previous results suggest there is space to search for models with higher face value and good overall performance, which here we demonstrate with two experiments based on clinical input. Figure 5 a shows we find rules with very similar performance for age/sex and the task-specific features: there is less than 0.020 and 0.005 deviation in balanced accuracy and AUROC points, respectively (for both training and test sets). The constraints further lead to significant reductions in computation time (depending on the task and features). Figure 5 b shows the performance improvements when using EXPLORE to update the existing CHADS 2 model. We identified 88,803 patients in IPCI with newly diagnosed atrial fibrillation of whom 1,422 experienced stroke in the next year (1.6%). The original CHADS 2 score has an AUROC value of 0.79 in the test set, which already demonstrates good discriminative model performance. However, when retraining using EXPLORE with CHADS 2 as feature, we obtain the best performing model with an AUROC of 0.83. 5. DISCUSSION 5.1. Interpretability-performance trade-off We have studied the predictive performance and interpretability of eight model algorithms across five prediction tasks in the Dutch GP database IPCI. LASSO, RandomForest, and XGBoost consistently performed best in terms of AUROC, followed by DecisionTree and EXPLORE. As the decision rules generated by EXPLORE are much simpler (at most 5 predictors), EXPLORE is an attractive candidate if the aim is to develop an interpretable prediction model. Moreover, the optimal decision rules obtained by EXPLORE perform similar or better than GOSDT-G for all prediction tasks. Our findings indicate a trade-off between model performance and interpretability, with its strength varying across prediction tasks. In some cases, a more interpretable model can be found with minor performance loss. Most previous work examining the trade-off has used open-source benchmark datasets (e.g. OpenML) ( 17 – 19 ). However, routinely collected health care data is typically more complex as it contains many observations/features and is very sparse. Earlier work that did study health care data only considered a limited number of prediction tasks (at most two) and did not investigate many of the model algorithms in this paper (e.g. EXPLORE, IHT, GOSDT-G, RIPPER, XGBoost) ( 16 , 20 ). We further contribute to the existing empirical evidence by evaluating the performance in terms of AUROC instead of accuracy, which is more commonly used ( 15 ). AUROC is more suitable to evaluate clinical prediction models as accuracy is very sensitive to imbalanced data (i.e. low outcome rate) and the chosen risk threshold. To our knowledge, this is the largest study examining the interpretability-performance trade-off in EHR data. 5.2. Clinically optimal decision rules Further, we showed that learning under constraints with EXPLORE can be used to find a set of near-optimal models. We found that the set of good candidate models is very large (1,381 − 20,320 for maximum rule length 5) for all prediction tasks. Moreover, we examined the Rashomon set of models and found there is a large variability in both the generalizability and features included. This shows there is a potential to find more clinically optimal decision rules. Earlier work concluded that a large Rashomon set means an equally well-performing, interpretable model is likely to exist and correlates with the existence of models with good generalizability ( 21 , 47 ). The ability to enhance the interpretability of the models was demonstrated by the experiments incorporating domain knowledge; both incorporating age/sex and task-specific features resulted in decision rules with similar performance on average and very small differences overall. These constraints can also lead to significant reductions in computation time by reducing the search space. Finally, we showed we can improve the original CHADS 2 score with EXPLORE by allowing EXPLORE to select additional covariates on top of the original score. This approach can also be applied to model updating when there is a desire for model stability over time (i.e. similar features in the model), as it keeps the original model intact. Learning under constraints with EXPLORE provides a novel solution to open challenges for (optimal) rule-based methods such as scalability and addressing user preferences ( 48 ). Furthermore, access to the Rashomon set of models is an asset of EXPLORE, shared by only a few other model algorithms (e.g. TreeFARMS ( 49 ), FasterRisk ( 50 )). 5.3. Limitations In this study we have used model size as a proxy measure for interpretability, which might not directly translate to human interpretability. Likewise, we have not been able to assess the human-machine task performance, which is ultimately what matters in real-world applications. This may both impact the true interpretability-performance trade-off. To assess this, user studies need to be performed ( 51 ). Additionally, while we studied a wide range of prediction tasks, the trade-off and best choice of model algorithm may differ based on the specific clinical application, which was not investigated in this study. 5.4. Future work First, we may continue to improve EXPLORE by further enhancements in speed thereby allowing longer decision rules (e.g. reducing the search space using reversed class labels) or supporting direct optimization over rank statistics (e.g. AUROC). Second, as external validation is a good indicator of model generalizability, this might expose more benefits of simple models and hence the interpretability-performance trade-off could be further investigated when transporting models to other databases. Finally, developing and testing search strategies to find the clinically optimal model from the Rashomon set of models is an interesting direction of research. 6. CONCLUSIONS Our study shows that more complex models generally outperform simpler ones, confirming the expected trade-off between model performance and interpretability. However, the observed trade-off varies across prediction tasks. While none of the complex models systematically outperforms the others, the simple models show significant variations in performance. EXPLORE’s ability to learn under constraints makes it an attractive candidate to find clinically optimal decision rules. Declarations Ethics approval and consent to participate The current study was approved by the IPCI Governance Board under number 3/2023. No human participants were involved in the study and informed consent was not necessary. Consent for publication Not applicable. Availability of data and materials The datasets generated and/or analysed during the current study are not publicly available due to patient privacy and data protection concerns. The code for EXPLORE and the experiments is available on GitHub. Competing interests The authors declare that they have no competing interests. Funding This project has received support from the European Health Data and Evidence Network (EHDEN) project. EHDEN received funding from the Innovative Medicines Initiative 2 Joint Undertaking (JU) under grant agreement No 806968. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and EFPIA. Author contributions A.M., J.K. and P.R. conceived of the presented idea. A.M. designed the study, carried out the implementation, and performed the experiments. J.K, E.F., K.V., and P.R. provided critical feedback and helped shape the research. A.M. wrote the article with input from all other authors. Acknowledgements We would like to thank Cesar Barboza Gutierrez for his contribution to testing the EXPLORE R package. References Costa VG, Pedreira CE. Recent advances in decision trees: An updated survey. Artificial Intelligence Review. 2023;56(5):4765-800. https://doi.org/10.1007/s10462-022-10275-5. Lipton ZC. The mythos of model interpretability. Queue. 2018:31–57. https://doi.org/10.1145/3236386.3241340. Markus AF, Kors JA, Rijnbeek PR. The role of explainability in creating trustworthy artificial intelligence for health care: A comprehensive survey of the terminology, design choices, and evaluation strategies. J Biomed Inform. 2021;113:103655. https://doi.org/10.1016/j.jbi.2020.103655. Yang C, Kors JA, Ioannou S, John LH, Markus AF, Rekkas A, et al. Trends in the conduct and reporting of clinical prediction model development and validation: A systematic review. J Am Med Inform Assoc. 2022;29(5):983-9. https://doi.org/10.1093/jamia/ocac002. Rijnbeek PR, Kors JA. Finding a short and accurate decision rule in disjunctive normal form by exhaustive search. Machine Learning. 2010;80(1):33-62. https://doi.org/10.1007/s10994-010-5168-9. Laurent H, Rivest RL. Constructing optimal binary decision trees is NP-complete. Information Processing Letters. 1976;5(1):15-7. https://doi.org/10.1016/0020-0190(76)90095-8. Aglin G, Nijssen S, Schaus P, editors. Learning optimal decision trees using caching branch-and-bound search. AAAI; 2020 04-08-2020. Verwer S, Zhang Y, editors. Learning optimal classification trees using a binary linear program formulation. Proceedings of the AAAI Conference on Artificial Intelligence; 2019 22-01-2020. Lin J, Zhong C, Hu D, Rudin C, Seltzer M. Generalized optimal sparse decision trees. arXiv preprint. 2020. https://doi.org/10.48550/arXiv.2006.08690. Lakkaraju H, Bach SH, Leskovec J, editors. Interpretable decision sets: A joint framework for description and prediction. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; 2016: ACM. Dash S, Gunluk O, Wei D, editors. Boolean decision rules via column generation. Adv Neural Inf Process Syst; 2018 16-01-2020. Mita G, Papotti P, Filippone M, Michiardi P, editors. Libre: Learning interpretable boolean rule ensembles. International Conference on Artificial Intelligence and Statistics; 2020: PMLR. Rudin C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell. 2019;1(5):206-15. https://doi.org/10.1038/s42256-019-0048-x. Bell A, Solano-Kamaiko I, Nov O, Stoyanovich J, editors. It’s just not that simple: An empirical study of the accuracy-explainability trade-off in machine learning for public policy. 2022 ACM Conference on Fairness, Accountability, and Transparency; 2022. Leblanc B, Germain P. On the relationship between interpretability and explainability in machine learning. arXiv preprint. 2024. https://doi.org/10.48550/arXiv.2311.11491. Caruana R, Lou Y, Gehrke J, Koch P, Sturm M, Elhadad N, editors. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; 2015 18-11-2019. Gosiewska A, Kozak A, Biecek P. Simpler is better: Lifting interpretability-performance trade-off via automated feature engineering. Decis Support Syst. 2021;150:113556. https://doi.org/10.1016/j.dss.2021.113556. McElfresh D, Khandagale S, Valverde J, Prasad C V, Ramakrishnan G, Goldblum M, et al. When do neural nets outperform boosted trees on tabular data? Adv Neural Inf Process Syst. 2024;36. https://doi.org/10.48550/arXiv.2305.02997. Liu J, Zhong C, Seltzer M, Rudin C. Fast sparse classification for generalized linear and additive models. Proceedings of Machine Learning Research. 2022;151:9304. https://doi.org/https://proceedings.mlr.press/v151/liu22f/liu22f.pdf. Letham B, Rudin C, McCormick TH, Madigan D. Interpretable classifiers using rules and bayesian analysis: Building a better stroke prediction model. Ann Appl Stat. 2015;9(3):1350-71. https://doi.org/10.1214/15-AOAS848. Semenova L, Rudin C, Parr R, editors. On the existence of simpler machine learning models. 2022 ACM Conference on Fairness, Accountability, and Transparency; 2022. Tonekaboni S, Joshi S, McCradden MD, Goldenberg A. What clinicians want: Contextualizing explainable machine learning for clinical end use. Proceedings of Machine Learning research. 2019:1-21. https://doi.org/http://proceedings.mlr.press/v106/tonekaboni19a/tonekaboni19a.pdf. Payrovnaziri SN, Chen Z, Rengifo-Moreno P, Miller T, Bian J, Chen JH, et al. Explainable artificial intelligence models using real-world electronic health record data: A systematic scoping review. J Am Med Inform Assoc. 2020. https://doi.org/10.1093/jamia/ocaa053. Jaotombo F, Adorni L, Ghattas B, Boyer L. Finding the best trade-off between performance and interpretability in predicting hospital length of stay using structured and unstructured data. PLoS One. 2023;18(11):e0289795. https://doi.org/10.1371/journal.pone.0289795. de Ridder MAJ, de Wilde M, de Ben C, Leyba AR, Mosseveld BMT, Verhamme KMC, et al. Data resource profile: The integrated primary care information (IPCI) database, the Netherlands. Int J Epidemiol. 2022;51(6):e314-e23. https://doi.org/10.1093/ije/dyac026. Hasan O, Meltzer DO, Shaykevich SA, Bell CM, Kaboli PJ, Auerbach AD, et al. Hospital readmission in general medicine patients: A prediction model. J Gen Intern Med. 2010;25:211-9. https://doi.org/10.1007/s11606-009-1196-1. Kansagara D, Englander H, Salanitro A, Kagen D, Theobald C, Freeman M, et al. Risk prediction models for hospital readmission: A systematic review. JAMA. 2011;306(15):1688-98. https://doi.org/10.1001/jama.2011.1515. Thoonsen B, Engels Y, van Rijswijk E, Verhagen S, van Weel C, Groot M, et al. Early identification of palliative care patients in general practice: Development of radboud indicators for palliative care needs (RADPAC). Br J Gen Pract. 2012;62(602):e625-e31. https://doi.org/10.3399/bjgp12x654597. ElMokhallalati Y, Bradley SH, Chapman E, Ziegler L, Murtagh FE, Johnson MJ, et al. Identification of patients with potential palliative care needs: A systematic review of screening tools in primary care. Palliat Med. 2020;34(8):989-1005. https://doi.org/10.1177/0269216320929552. Tibble H, Tsanas A, Horne E, Horne R, Mizani M, Simpson CR, et al. Predicting asthma attacks in primary care: Protocol for developing a machine learning-based prediction model. BMJ Open. 2019;9(7):e028375. https://doi.org/10.1136/bmjopen-2018-028375. Zein JG, Wu C-P, Attaway AH, Zhang P, Nazha A. Novel machine learning can predict acute asthma exacerbation. Chest. 2021;159(5):1747-57. https://doi.org/10.1016/j.chest.2020.12.051. Bloom C, Ricciardi F, Smeeth L, Stone P, Quint J. Predicting COPD 1-year mortality using prognostic predictors routinely measured in primary care. BMC Med. 2019;17:1-10. https://doi.org/10.1186/s12916-019-1310-0. Dziopa K, Asselbergs FW, Gratton J, Chaturvedi N, Schmidt AF. Cardiovascular risk prediction in type 2 diabetes: A comparison of 22 risk scores in primary care settings. Diabetologia. 2022;65(4):644-56. https://doi.org/10.1007/s00125-021-05640-y. Reps JM, Schuemie MJ, Suchard MA, Ryan PB, Rijnbeek PR. Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data. J Am Med Inform Assoc. 2018;25(8):969-75. https://doi.org/10.1093/jamia/ocy032. Reps J, Wong J, Fridgeirsson EA, Kim C, John LH, Williams R, et al. Finding a constrained number of predictor phenotypes for multiple outcome prediction. SSRN preprint. 2024. https://doi.org/https://dx.doi.org/10.2139/ssrn.4874415. Breiman L, Friedman J, Olshen R, Stone C. CART: Classification and regression trees1984. McTavish H, Zhong C, Achermann R, Karimalis I, Chen J, Rudin C, et al. Fast sparse decision tree optimization via reference ensembles. Proceedings of the AAAI Conference on Artificial Intelligence. 2022;36(9):9604-13. https://doi.org/10.1609/aaai.v36i9.21194. Blumensath T, Davies ME. Iterative hard thresholding for compressed sensing. Applied and Computational Harmonic Analysis. 2009;27(3):265-74. https://doi.org/10.1016/j.acha.2009.04.002. Suchard MA, Simpson SE, Zorych I, Ryan P, Madigan D. Massive parallelization of serial inference algorithms for a complex generalized linear model. ACM Transactions on Modeling and Computer Simulation (TOMACS). 2013;23(1):1-17. https://doi.org/10.1145/2414416.2414791. Breiman L. Random forests. Machine Learning. 2001;45(1):5-32. https://doi.org/10.1023/A:1010933404324. Cohen WW, editor Fast effective rule induction. Machine Learning Proceedings; 1995 18-12-2019: Elsevier. Chen T, Guestrin C, editors. Xgboost: A scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; 2016. Guidotti R, Monreale A, Ruggieri S, Turini F, Giannotti F, Pedreschi D. A survey of methods for explaining black box models. ACM Computing Surveys. 2018;51(5):93. https://doi.org/10.1145/3236009. Breiman L. Statistical modeling: The two cultures. Statistical Science. 2001;16(3):199-231, 33. https://doi.org/10.1214/ss/1009213726. Nogueira S, Sechidis K, Brown G. On the stability of feature selection algorithms. J Mach Learn Res. 2018;18(1):6345-98. https://doi.org/https://jmlr.org/papers/volume18/17-514/17-514.pdf. Gage BF, Waterman AD, Shannon W, Boechler M, Rich MW, Radford MJ. Validation of clinical classification schemes for predicting stroke results from the national registry of atrial fibrillation. JAMA. 2001;285(22):2864-70. https://doi.org/10.1001/jama.285.22.2864. Rudin C, Zhong C, Semenova L, Seltzer M, Parr R, Liu J, et al. Amazing things come from having many good models. arXiv preprint. 2024. https://doi.org/10.48550/arXiv.2407.04846. Rudin C, Chen C, Chen Z, Huang H, Semenova L, Zhong C. Interpretable machine learning: Fundamental principles and 10 grand challenges. Statistics Surveys. 2022;16:1-85. https://doi.org/10.1214/21-SS133. Xin R, Zhong C, Chen Z, Takagi T, Seltzer M, Rudin C. Exploring the whole rashomon set of sparse decision trees. Adv Neural Inf Process Syst. 2022;35:14071-84. https://doi.org/https://proceedings.neurips.cc/paper_files/paper/2022/file/5afaa8b4dd18eb1eed055d2d821b58ae-Paper-Conference.pdf. Liu J, Zhong C, Li B, Seltzer M, Rudin C. FasterRisk: Fast and accurate interpretable risk scores. Adv Neural Inf Process Syst. 2022;35:17760-73. https://doi.org/https://proceedings.neurips.cc/paper_files/paper/2022/file/7103444259031cc58051f8c9a4868533-Paper-Conference.pdf. Poursabzi-Sangdeh F, Goldstein DG, Hofman JM, Wortman Vaughan JW, Wallach H, editors. Manipulating and measuring model interpretability. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems; 2021. Additional Declarations No competing interests reported. Supplementary Files EXPLORESupplementaryMaterial.docx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5804837","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":403507449,"identity":"701d774c-49f3-4ae5-bf41-61c7c37b2911","order_by":0,"name":"Aniek F. Markus","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA2UlEQVRIie3PMQrCMBSA4YRCHCR0TRD0BEKlUJAOuUqz2MUDOGbSyd3NKwjCmwOBdskB2k0XZ8Upm+2mCLGjQ/7p8eAjLwiFQn+bRij+WFyGEK7eF8UQkuihJFZj+dxYI9J6f324tRHznSYXH2GanrltjARbpxMK/VCMEu9hmp64upsia1ZogqEfEGE+MutecR0R6fEWOQdG/CSJpsBVY/CJEcK6wzD8IgtDYalsKQ92RXIKZfcXufWSab0/t6rKRbyrotZBLrLaVOzuISj6XmHlA6FQKBQa0AtyY1LD1px7fwAAAABJRU5ErkJggg==","orcid":"","institution":"Erasmus University Medical Center","correspondingAuthor":true,"prefix":"","firstName":"Aniek","middleName":"F.","lastName":"Markus","suffix":""},{"id":403507450,"identity":"72ab7317-4bdc-46e0-9b72-90ec483c99ec","order_by":1,"name":"Jan A. Kors","email":"","orcid":"","institution":"Erasmus University Medical Center","correspondingAuthor":false,"prefix":"","firstName":"Jan","middleName":"A.","lastName":"Kors","suffix":""},{"id":403507451,"identity":"84d2166c-94a8-420e-b654-f3208f7baebf","order_by":2,"name":"Egill A. Fridgeirsson","email":"","orcid":"","institution":"Erasmus University Medical Center","correspondingAuthor":false,"prefix":"","firstName":"Egill","middleName":"A.","lastName":"Fridgeirsson","suffix":""},{"id":403507452,"identity":"bae0b72e-95d9-4861-aa6e-d0c5b3556229","order_by":3,"name":"Katia M.C. Verhamme","email":"","orcid":"","institution":"Erasmus University Medical Center","correspondingAuthor":false,"prefix":"","firstName":"Katia","middleName":"M.C.","lastName":"Verhamme","suffix":""},{"id":403507453,"identity":"49d3cf62-2c2a-4340-b467-3a32b17e49e6","order_by":4,"name":"Peter R. Rijnbeek","email":"","orcid":"","institution":"Erasmus University Medical Center","correspondingAuthor":false,"prefix":"","firstName":"Peter","middleName":"R.","lastName":"Rijnbeek","suffix":""}],"badges":[],"createdAt":"2025-01-10 15:23:18","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5804837/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5804837/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":74246986,"identity":"484dbaf4-3003-444d-a721-32d865695ecf","added_by":"auto","created_at":"2025-01-20 09:58:35","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":64225,"visible":true,"origin":"","legend":"\u003cp\u003eOverview of the improved EXPLORE algorithm.\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-5804837/v1/32e0b160ce4b85782b79f031.png"},{"id":74244338,"identity":"0e48d6f2-61b1-4cf6-9a87-7660accfd3a0","added_by":"auto","created_at":"2025-01-20 09:50:35","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":120131,"visible":true,"origin":"","legend":"\u003cp\u003eMedian-centered AUROC on the test set averaged across five prediction tasks for different model algorithms.\u003c/p\u003e","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-5804837/v1/4354298277c644044f25d07f.png"},{"id":74244344,"identity":"476e41c3-5627-471f-9a19-d7a60ac2050d","added_by":"auto","created_at":"2025-01-20 09:50:35","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":176851,"visible":true,"origin":"","legend":"\u003cp\u003eModel interpretability versus performance for prediction tasks on test set. Each dot corresponds to a prediction model; colors represent different model algorithms and shapes represent different prediction tasks. The lines (fitted using a linear model) indicate the observed relation between interpretability and performance across the different models for each prediction task.\u003c/p\u003e","description":"","filename":"floatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-5804837/v1/36a1aea52af00f1f696754b2.png"},{"id":74246987,"identity":"5a58cd4c-a347-45a7-b5d3-2da048a67acc","added_by":"auto","created_at":"2025-01-20 09:58:35","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":241547,"visible":true,"origin":"","legend":"\u003cp\u003eCharacterizing EXPLORE’s Rashomon set of models with less than 1% deviation from the balanced accuracy of the best model. The vertical lines indicate the performance of the best model in the training and test set.\u003c/p\u003e","description":"","filename":"floatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-5804837/v1/0fe55199e632b3017b6c39fe.png"},{"id":74244341,"identity":"4bcbd84d-b32e-4e52-9dc0-f09c70b343cf","added_by":"auto","created_at":"2025-01-20 09:50:35","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":116542,"visible":true,"origin":"","legend":"\u003cp\u003eImprovement in AUROC and balanced accuracy values for mandatory features (a) and improved CHADS\u003csub\u003e2\u003c/sub\u003e score (b) with EXPLORE on training and test set. AUROC and balanced accuracy values can be found in Appendix D.\u003c/p\u003e","description":"","filename":"floatimage5.png","url":"https://assets-eu.researchsquare.com/files/rs-5804837/v1/bb46f499966089a962f83126.png"},{"id":74288604,"identity":"581fdf39-a5d7-4f1c-9fc1-2ca3792f2ccf","added_by":"auto","created_at":"2025-01-20 16:23:56","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1501530,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5804837/v1/f7abb1b2-9876-457f-b2e3-d442934a3e00.pdf"},{"id":74244357,"identity":"cfaeaa42-e5c1-408c-ac17-d12f2fc130d2","added_by":"auto","created_at":"2025-01-20 09:50:36","extension":"docx","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":626430,"visible":true,"origin":"","legend":"","description":"","filename":"EXPLORESupplementaryMaterial.docx","url":"https://assets-eu.researchsquare.com/files/rs-5804837/v1/88b5815c515b2dc377ef25c2.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"EXPLORE: learning interpretable rules for patient-level prediction","fulltext":[{"header":"1. BACKGROUND","content":"\u003cp\u003eRule-based methods have been among the most popular classification methods in machine learning since the 1960s (\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e). As predictions of these models are the result of a series of relatively simple computations, small rule-based models are often considered to be interpretable (\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e). Interpretable models allow users to understand, verify, and improve model output (\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e), which is often demanded when the cost of misclassification is high (e.g. in health care). In addition, these models may be easier to implement as they can be used without system integration and face less technical challenges during deployment due to their smaller size. Moreover, rule-based methods show close resemblance to existing clinical workflows such as knowledge-based expert systems and clinical pathways. Despite these advantages, rule-based methods are less frequently used to develop clinical prediction models compared to other methods like regression analysis (\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eIn this paper we aim to develop \u003cem\u003eclinically optimal\u003c/em\u003e decision rules using electronic health record (EHR) data. The concept of \u003cem\u003eoptimality\u003c/em\u003e in clinical prediction modelling goes beyond the traditional mathematical definition of a feasible solution that minimizes (or maximizes) a given objective function. Model performance is important, but for a model to be successful in practice more is needed. First, a clinically optimal model is interpretable, i.e., as simple as possible in both model form and size as well as the features included. A model (partially) aligning with prior knowledge might be better generalizable and more readily adopted in practice. Second, a clinically optimal model might need to satisfy certain constraints (e.g. a minimum level of specificity). Third, a clinically optimal model is stable when updated. Similar features are included in the model and predictions for similar examples are similar, all while achieving performance that is (close to) the \u0026lsquo;best\u0026rsquo; possible for the given prediction task.\u003c/p\u003e \u003cp\u003eThe Exhaustive Procedure for Logic-Rule Extraction (EXPLORE) algorithm (\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e) has the necessary features to develop such clinically optimal decision rules: it produces small decision rules, has optimality guarantees, and allows the user to specify global model constraints. The contributions of this paper are as follows:\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003eWe improve EXPLORE (\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e) in terms of scalability, making it accessible and executable for real-world sized datasets. We demonstrate this by developing decision rules for five prediction tasks (~\u0026thinsp;50 features, ~\u0026thinsp;4,000-200,000 observations) using Dutch general practitioner (GP) data.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eWe investigate whether a trade-off occurs between predictive performance and model interpretability in real-world health care data by comparing EXPLORE\u0026rsquo;s performance to seven other commonly used state-of-the-art modelling methods in a large-scale empirical evaluation.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eWe illustrate how to develop clinically optimal decision rules by learning under constraints with EXPLORE.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e"},{"header":"2. RELATED WORK","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003e2.1. Optimal rule-based model algorithms\u003c/h2\u003e \u003cp\u003eRule-based methods are a popular choice to develop interpretable models as they are human understandable (consist of a collection of statements connected using \u0026ldquo;IF-THEN\u0026rdquo;, \u0026ldquo;OR\u0026rdquo;, and \u0026ldquo;AND\u0026rdquo;) and allow to model categorical data with non-linear relationships. Examples include decision trees, decisions lists (i.e. ordered sets of rules or one-sided decision trees), and decision sets (i.e. unordered sets of rules). Historically, greedy approaches have often been used to grow decision trees (e.g. CART, C4.5) and decision lists/sets (e.g. RIPPER, C4.5R), as decision tree optimization is known to be an NP-complete problem (\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e). These approaches split the optimization problem into steps and search for the optimal solution at every split. Although this makes their computation more tractable, the disadvantage is that there are no optimality guarantees for their final performance. In case of bad performance, it is therefore unclear whether this is due to model choice or poor optimization.\u003c/p\u003e \u003cp\u003eIn recent years several methods have been proposed to grow \u003cem\u003eoptimal decision trees\u003c/em\u003e. For example, DL8.5 (\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e) combines dynamic programming on the space of the decision trees and branch-and-bound optimization, BinOCT (\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e) uses a novel binary linear program formulation for optimization, and GOSDT (\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e) uses dynamic programming with custom bounds. However, DL8.5 and BinOCT loose optimality when using bucketization to handle continuous features. Although less work focuses on \u003cem\u003eoptimal decision rules\u003c/em\u003e, some methods with optimization guarantees have been proposed: EXPLORE (\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e) uses a branch-and-bound approach for efficient exhaustive search, IDS (\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e) relies on submodular optimization, CG (\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e) uses a column generation algorithm, and LIBRE (\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e) creates an ensemble of weak learners.\u003c/p\u003e \u003cp\u003eThe exhaustive search performed by EXPLORE offers several advantages over other methods as it allows users to specify global constraints on the model (e.g. model size, included features) or its performance (e.g. minimum sensitivity and specificity). Earlier work investigating the performance of EXPLORE on UCI datasets has shown promising results (\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e), but is limited as the studied prediction tasks do not match the complexity of real-world data.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e2.2. Interpretability-performance trade-off\u003c/h2\u003e \u003cp\u003eIdeally, we want models to produce the best possible predictions, while being simple enough to be understood by users. The interpretability-performance trade-off refers to the general assumption that increased model flexibility contributes to a better model fit (increasing performance) at the cost of additional complexity (reducing interpretability). Rudin (\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e) raised important awareness that this trade-off might be a myth and stressed the importance of using interpretable models especially for high-stake decisions. She argued that especially for structured data with meaningful features increased interpretability does not necessarily lead to a loss of performance.\u003c/p\u003e \u003cp\u003eThere is a growing body of research questioning the extent to which this trade-off occurs (\u003cspan additionalcitationids=\"CR14\" citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e). Various studies have shown examples where complex models do not outperform simpler models (\u003cspan additionalcitationids=\"CR17 CR18 CR19\" citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e). Furthermore, experiments in various domains show there might be multiple equally-good performing models, leading some to suggest that interpretable models are likely to exist among them (\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e). The extent of the trade-off might ultimately depend on the data characteristics (\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e) and used metrics (\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e). Also for medical data, it is often assumed that this trade-off exists (\u003cspan additionalcitationids=\"CR23\" citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e).\u003c/p\u003e \u003c/div\u003e"},{"header":"3. METHODS","content":"\u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003e3.1. EXPLORE R package\u003c/h2\u003e \u003cp\u003eWe developed the EXPLORE R package (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://github.com/mi-erasmusmc/Explore\u003c/span\u003e\u003cspan address=\"https://github.com/mi-erasmusmc/Explore\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e), which implements the EXPLORE algorithm (\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e) that generates optimal decision rules of specified length in disjunctive normal form (DNF). A formula in DNF is a disjunction (OR, \u0026or;) of terms, where each term is a conjunction (AND, \u0026and;) of literals, i.e. feature-operator-value triples (e.g. age\u0026thinsp;\u0026gt;\u0026thinsp;40). The rules are systematically generated using three nested loops that evaluate the next term tuple (i.e. ordered list of term sizes), feature-operator pair, and threshold value, respectively (see Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). For example, when creating a rule with maximum length 3 for a dataset with 10 binary predictors EXPLORE generates 5540 candidate rules. For details on the original algorithm we refer to the original publication (\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eWe made two main contributions to improve EXPLORE\u0026rsquo;s scalability. First, we further optimized the generation of decision rules for binary data. In particular, we show that it is possible to simplify rules including a term of size 1 to a rule of shorter length using the redundancy law of Boolean algebra (see Appendix A.1). This reduces the number of feature-operator pairs that need to be generated, leading to a reduction in the number of fully instantiated rules of 7.0% for rule length 3 to 20.5% for rule length 5. Second, we parallelized EXPLORE using the RcppParallel package. Using the improved version of EXPLORE we reduced average computation time by 92.2% for rule length 4 (see Appendix A.2).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003e3.2. Dataset\u003c/h2\u003e \u003cp\u003eWe used real-world health data from the Dutch Integrated Primary Care Information (IPCI) database (\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e). This database contains longitudinal, routinely collected data from computer-based patient records of around 350 GP practices throughout the Netherlands. In total, the database currently (January 1, 2024) contains 2.87\u0026nbsp;million patient records with a median follow-up duration of 4.7 years. The number of active patients is 1.3\u0026nbsp;million, which comprises about 8% of the Dutch population (~\u0026thinsp;17\u0026nbsp;million). Data include patient demographics, information about contacts with GPs, symptoms, diagnoses, laboratory and clinical measurements, prescriptions, and information on use of secondary care. The IPCI database has been mapped to the Observational Medical Outcomes Partnership Common Data Model (OMOP CDM), which enables standardized extraction and analysis of health care data. This study was approved by the IPCI Governance Board (number 3/2023).\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003e3.3. Prediction tasks\u003c/h2\u003e \u003cp\u003eWe selected five prediction tasks for evaluation, for their definition see Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e. These tasks represent a range of different target populations (generic and disease specific), time-at-risks (from 1 month up to 5 years), and outcome rates. Furthermore, these prediction tasks have a clear clinical use case and have been more commonly investigated in the literature: hospital readmission (\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e, \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e), end-of-life care (\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e, \u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e), asthma exacerbations (\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e, \u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e), mortality in chronic obstructive pulmonary disease (COPD) (\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e), and cardiovascular disease in type-2 diabetes mellitus (T2DM) (\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e). We restricted all target populations to adults (age\u0026thinsp;\u0026gt;\u0026thinsp;=\u0026thinsp;18) and include events in the last five years before database end minus the time-at-risk period (e.g. for hospital readmission, cohort entry date after December 1, 2018).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eSpecification of prediction tasks. Generic prediction question has the form (\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e): \u0026ldquo;among *target population*, which patients will develop *outcome of interest* during *time-at-risk period*?\u0026rdquo;\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTask\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTarget population\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eOutcome of interest\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eTime-at-risk period\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHospital readmission\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAdult patients discharged from hospital\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eHospital admission\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e1 month\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eEnd-of-life care\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eGP visit of older patients (60+) with a prior diagnosis of heart failure, COPD, or cancer\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eEnd-of-life conversation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e6 months\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAsthma exacerbations\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAdult patients with new asthma diagnosis receiving medication\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eExacerbation (as defined by 3\u0026ndash;30 days of systemic corticosteroids)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e2 years\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMortality in COPD\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003ePatients with new COPD diagnosis (40+)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAll-cause mortality\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e2 years\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCardiovascular disease in T2DM\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAdult patients with new T2DM diagnosis\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eHeart failure or stroke\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e5 years\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec9\" class=\"Section2\"\u003e \u003ch2\u003e3.4. Model development\u003c/h2\u003e \u003cp\u003eWe used the PatientLevelPrediction R package (\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e) to develop prediction models. This tool provides a streamlined pipeline to develop clinical prediction models in line with current best practices using OMOP CDM data. The tool extracts the necessary information from the database based on the specified prediction task, splits the data into a training (75%) and test set (25%), constructs and pre-processes the covariates (see Section 3.5), fits models using different modelling algorithms on the training set (see Section 3.6), and performs internal validation of the model on the test set (see Section 3.7).\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec10\" class=\"Section2\"\u003e \u003ch2\u003e3.5. Candidate predictors\u003c/h2\u003e \u003cp\u003eThe predictors included for model development were patient demographics (i.e. age and sex) and binary indicators for the presence of 47 clinical events in the 365 days prior to the index date. The set of clinical events - including common medical conditions and drug exposures - is based on the constrained set of clinically meaningful predictors identified by Reps, Wong (\u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e) to achieve minimal performance loss across a variety of prediction tasks (when compared to a large set of predictors). We removed predictors that occurred in less than 0.1 percent of the target population.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003e3.6. Model algorithms\u003c/h2\u003e \u003cp\u003eIn addition to EXPLORE, we investigated 7 machine learning algorithms that are commonly used in clinical prediction modelling or rule-based learning: DecisionTree (\u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e), GOSDT-G (\u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e), Iterative Hard Thresholding (IHT) (\u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e38\u003c/span\u003e), LASSO (\u003cspan citationid=\"CR39\" class=\"CitationRef\"\u003e39\u003c/span\u003e), RandomForest (\u003cspan citationid=\"CR40\" class=\"CitationRef\"\u003e40\u003c/span\u003e), RIPPER (\u003cspan citationid=\"CR41\" class=\"CitationRef\"\u003e41\u003c/span\u003e), and XGBoost (\u003cspan citationid=\"CR42\" class=\"CitationRef\"\u003e42\u003c/span\u003e). We used 3-fold cross validation to optimize the hyperparameters, including a range of values for each model algorithm to allow for varying levels of model complexity. For details on used libraries and hyperparameters we refer to Appendix B.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003e3.7. Model evaluation\u003c/h2\u003e \u003cp\u003e \u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003ePredictive performance\u003c/span\u003e \u003c/p\u003e \u003cp\u003eThe area under the receiver operating characteristic (AUROC) curve was used to evaluate the ability of the model to distinguish between the classes. Some of the included model algorithms predict classes instead of probabilities (DecisionTree, EXPLORE, GOSDT-G, RIPPER). These models do not allow to vary the risk threshold and hence have a fixed number of true/false positives. EXPLORE\u0026rsquo;s ability to learn under constraints opens possibilities to evaluate its performance in a similar way as models predicting probabilities (despite it predicting classes). Details on how to create the AUROC curve for EXPLORE by developing models for different constraints on sensitivity and specificity can be found in Appendix C.\u003c/p\u003e \u003cp\u003e \u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003eInterpretability\u003c/span\u003e \u003c/p\u003e \u003cp\u003eModel size was used as a proxy for model interpretability (\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e, \u003cspan citationid=\"CR43\" class=\"CitationRef\"\u003e43\u003c/span\u003e). For EXPLORE and RIPPER model size is measured by the total length of rules. In the case of tree-based algorithms like DecisionTree, GOSDT-G, RandomForest and XGBoost, we consider the total number of nodes in the tree(s). For LASSO and IHT, we use the number of non-zero coefficients in the model. As various proxies can be chosen for each model algorithm, we performed a sensitivity analysis by counting the number of unique features used per model. While the first metric captures to the number of computations required for model application and aims to represent the complexity of the model\u0026rsquo;s decision-making process, the second metric represents the amount of information a clinician must collect (e.g. by asking questions) to use the model.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003e3.8. Experimental setup\u003c/h2\u003e \u003cp\u003eWe developed a total of 40 models (8 model algorithms x 5 prediction tasks) to evaluate the performance of EXPLORE and more generally investigate the existence of the interpretability-performance trade-off. In addition, we conducted experiments to illustrate how EXPLORE\u0026rsquo;s learning under constraints can be used to develop clinically optimal decision rules.\u003c/p\u003e \u003cp\u003e \u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003eQuantifying the search space of good candidate models\u003c/span\u003e \u003c/p\u003e \u003cp\u003eThe phenomenon that different prediction models relying on different features can perform (almost) equally well is referred to as the Rashomon effect (\u003cspan citationid=\"CR44\" class=\"CitationRef\"\u003e44\u003c/span\u003e). The Rashomon set is defined as the collection of near-optimal models, and its size can be measured by comparing it to the total number of possible models, resulting in the Rashomon ratio (\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e). Although the Rashomon set and ratio are difficult to compute (exactly) for most model classes (due to their hypothesis space being infinite), these can easily be computed for EXPLORE using its ability to learn under constraints. We constructed the Rashomon set for each prediction task by setting a global model constraint at 99% of the maximum balanced accuracy (i.e. performance of the best model). We further examined the Rashomon set in terms of generalizability (i.e. whether training performance transfers to the test set) and model diversity (i.e. frequency of included features and variation using the model stability metric of Nogueira, Sechidis (\u003cspan citationid=\"CR45\" class=\"CitationRef\"\u003e45\u003c/span\u003e))).\u003c/p\u003e \u003cp\u003e \u003cspan type=\"Underline\" class=\"Underline\" name=\"Emphasis\"\u003eIncorporating domain knowledge and improving existing models\u003c/span\u003e \u003c/p\u003e \u003cp\u003eNext, we investigated whether we can enforce constraints to include clinically relevant features for the prediction tasks without losing performance. Two cases were considered: a) including age/sex and b) two task-specific features selected based on clinical knowledge, the remaining predictors were in both cases selected by EXPLORE. Furthermore, we investigated if we can improve the performance of an existing model, the CHADS\u003csub\u003e2\u003c/sub\u003e score (\u003cspan citationid=\"CR46\" class=\"CitationRef\"\u003e46\u003c/span\u003e), which estimates the risk of stroke within 1 year for patients with atrial fibrillation. The original CHADS\u003csub\u003e2\u003c/sub\u003e score includes the following covariates: congestive heart failure history, hypertension history, age, diabetes mellitus history, stroke or transient ischemic attack (TIA) symptoms. We compared the prediction of the original model score with two updated versions: a) we retrained the model using EXPLORE with the same set of covariates, and b) allowed EXPLORE to add 4 additional covariates on top of the CHADS\u003csub\u003e2\u003c/sub\u003e score (included as numerical feature).\u003c/p\u003e \u003cp\u003eThe full code to run our experiments (including all cohort definitions) is available on GitHub: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://github.com/mi-erasmusmc/ExploreExperiments\u003c/span\u003e\u003cspan address=\"https://github.com/mi-erasmusmc/ExploreExperiments\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. Experiments can be repeated on (other) databases mapped to the OMOP CDM.\u003c/p\u003e \u003c/div\u003e"},{"header":"4. RESULTS","content":"\u003cdiv id=\"Sec15\" class=\"Section2\"\u003e \u003ch2\u003e4.1. Baseline characteristics\u003c/h2\u003e \u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e presents a summary of the five prediction tasks. As shown the experiments include a variety of tasks with different target population sizes (4,002\u0026ndash;230,952) and outcome rates (0.9% \u0026ndash; 29.1%), demographics vary across tasks as expected. An overview of all clinical characteristics is included in Appendix D.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eCharacteristics of the prediction tasks.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"6\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHospital readmission\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eEnd-of-life care\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eAsthma exacerbations\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eMortality\u003c/p\u003e \u003cp\u003ein COPD\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eCardiovascular disease in T2DM\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTarget population size\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e75,375\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e230,952\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e4,002\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e4,094\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e6,634\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAge, mean\u0026thinsp;\u0026plusmn;\u0026thinsp;SD\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e63.6\u0026thinsp;\u0026plusmn;\u0026thinsp;17.9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e71.6\u0026thinsp;\u0026plusmn;\u0026thinsp;8.3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e48.3\u0026thinsp;\u0026plusmn;\u0026thinsp;17.7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e67.1\u0026thinsp;\u0026plusmn;\u0026thinsp;11.1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e60.2\u0026thinsp;\u0026plusmn;\u0026thinsp;12.4\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBirth sex, % female\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e51.9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e52.8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e61.2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e47.5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e44.5\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eNumber of outcomes (% of population)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e7,512\u003c/p\u003e \u003cp\u003e(10.0)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e2,171 (0.9)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e1,164\u003c/p\u003e \u003cp\u003e(29.1)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e495\u003c/p\u003e \u003cp\u003e(12.1)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e458\u003c/p\u003e \u003cp\u003e(1.6)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec16\" class=\"Section2\"\u003e \u003ch2\u003e4.2. Performance comparison of model algorithms\u003c/h2\u003e \u003cp\u003eThe maximum AUROC value across prediction tasks and model algorithms ranges from 0.65 (for Hospital Readmission) to 0.77 (for Mortality in COPD) on the test set. Figure\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e shows the difference in the performance of the different model algorithms based on discrimination. AUROC curves and values are included in Appendix D. Across the prediction tasks, LASSO, RandomForest, and XGBoost resulted in models with the best predictive performance. These models show superior performance to the other algorithms, but there is no clear winner among them. The predictive performance in terms of AUROC value for EXPLORE ranged between 0.61 and 0.71 across the prediction tasks. GOSDT-G, and IHT are sometimes close in performance to EXPLORE, but perform worse on average and show a larger variability between tasks. The performance of GOSDT-G may be suboptimal due to the range of selected hyperparameters, which were chosen to remain computationally feasible for our size dataset. DecisionTree performed on average slightly better in terms of AUROC than EXPLORE, whereas RIPPER showed consistently poor performance.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec17\" class=\"Section2\"\u003e \u003ch2\u003e4.3. Model complexity and its relation to predictive performance\u003c/h2\u003e \u003cp\u003eWe also investigated the relation with model interpretability. The most interpretable models (model size across tasks indicated between brackets) were found by EXPLORE (\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e), GOSDT-G (\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e), IHT (\u003cspan additionalcitationids=\"CR5 CR6 CR7 CR8\" citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e), and RIPPER (\u003cspan additionalcitationids=\"CR6 CR7 CR8 CR9 CR10 CR11 CR12 CR13 CR14 CR15 CR16 CR17 CR18 CR19 CR20\" citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e). DecisionTree (\u003cspan additionalcitationids=\"CR22 CR23 CR24 CR25 CR26 CR27 CR28 CR29 CR30\" citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e), LASSO (\u003cspan additionalcitationids=\"CR25 CR26 CR27 CR28 CR29 CR30 CR31 CR32 CR33 CR34 CR35 CR36 CR37 CR38\" citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR39\" class=\"CitationRef\"\u003e39\u003c/span\u003e), XGBoost (259\u0026ndash;3935), and RandomForest (628-35028) resulted in more complex models. For a complete overview of model sizes we refer to Appendix D.\u003c/p\u003e \u003cp\u003eFigure \u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e shows the relation between interpretability (defined here as the inverse of model size) and performance. A vertical (or horizontal) line indicates no interpretability-performance trade-off. We observe a strong variability in the performance of simple models, while the more complex models show a performance increase that is accompanied by a large reduction in interpretability. These findings confirm the expected trade-off between model performance and interpretability. However, this observed trade-off varies across prediction tasks. For example, the observed trade-off is stronger for Cardiovascular disease in T2DM than for Asthma exacerbations. In the latter case, a more interpretable model can be chosen with less reduction in performance. Overall, trade-off patterns were found to be similar in the training and test set. Similar results were observed for the sensitivity analysis counting the number of unique features used per model (Appendix D).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec18\" class=\"Section2\"\u003e \u003ch2\u003e4.4. Quantifying the search space of good candidate models\u003c/h2\u003e \u003cp\u003eThe number of models with less than 1% deviation from the best performance ranges from 153\u0026ndash;675 for maximum rule length 4 and 1,381\u0026thinsp;\u0026minus;\u0026thinsp;20,320 for maximum rule length 5. Hence, there are many good candidate models for each prediction task. Figure\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003ea shows the performance of all models in the Rashomon set with maximum rule length 5 compared to the best model (i.e. the model that would be selected during standard model development). For some tasks the best model in the training set is among the best performing on the test set (e.g. End-of-life care and Hospital Readmission), but for the other tasks there exist models in the Rashomon set with much better generalization performance. Figure\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003eb shows there is a large number of features that are included in one of the models in the Rashomon set and only a handful that are frequently occurring (e.g. age is always included for End-of-life care, Mortality in COPD, and Cardiovascular disease in T2DM). Most features (84\u0026ndash;90%) are occurring in less than 10% of the unique feature sets and only 4\u0026ndash;6% of the features in more than 75%. According to the stability metric of Nogueira, Sechidis (\u003cspan citationid=\"CR45\" class=\"CitationRef\"\u003e45\u003c/span\u003e), there is most stability in the Rashomon set of Mortality in COPD (0.58) and least for End-of-life care (0.36).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec19\" class=\"Section2\"\u003e \u003ch2\u003e4.5. Incorporating domain knowledge and improving existing models\u003c/h2\u003e \u003cp\u003eThe previous results suggest there is space to search for models with higher face value and good overall performance, which here we demonstrate with two experiments based on clinical input. Figure\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003ea shows we find rules with very similar performance for age/sex and the task-specific features: there is less than 0.020 and 0.005 deviation in balanced accuracy and AUROC points, respectively (for both training and test sets). The constraints further lead to significant reductions in computation time (depending on the task and features). Figure\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003eb shows the performance improvements when using EXPLORE to update the existing CHADS\u003csub\u003e2\u003c/sub\u003e model. We identified 88,803 patients in IPCI with newly diagnosed atrial fibrillation of whom 1,422 experienced stroke in the next year (1.6%). The original CHADS\u003csub\u003e2\u003c/sub\u003e score has an AUROC value of 0.79 in the test set, which already demonstrates good discriminative model performance. However, when retraining using EXPLORE with CHADS\u003csub\u003e2\u003c/sub\u003e as feature, we obtain the best performing model with an AUROC of 0.83.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"5. DISCUSSION","content":"\u003cdiv id=\"Sec21\" class=\"Section2\"\u003e \u003ch2\u003e5.1. Interpretability-performance trade-off\u003c/h2\u003e \u003cp\u003eWe have studied the predictive performance and interpretability of eight model algorithms across five prediction tasks in the Dutch GP database IPCI. LASSO, RandomForest, and XGBoost consistently performed best in terms of AUROC, followed by DecisionTree and EXPLORE. As the decision rules generated by EXPLORE are much simpler (at most 5 predictors), EXPLORE is an attractive candidate if the aim is to develop an interpretable prediction model. Moreover, the optimal decision rules obtained by EXPLORE perform similar or better than GOSDT-G for all prediction tasks. Our findings indicate a trade-off between model performance and interpretability, with its strength varying across prediction tasks. In some cases, a more interpretable model can be found with minor performance loss. Most previous work examining the trade-off has used open-source benchmark datasets (e.g. OpenML) (\u003cspan additionalcitationids=\"CR18\" citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e). However, routinely collected health care data is typically more complex as it contains many observations/features and is very sparse. Earlier work that did study health care data only considered a limited number of prediction tasks (at most two) and did not investigate many of the model algorithms in this paper (e.g. EXPLORE, IHT, GOSDT-G, RIPPER, XGBoost) (\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e, \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e). We further contribute to the existing empirical evidence by evaluating the performance in terms of AUROC instead of accuracy, which is more commonly used (\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e). AUROC is more suitable to evaluate clinical prediction models as accuracy is very sensitive to imbalanced data (i.e. low outcome rate) and the chosen risk threshold. To our knowledge, this is the largest study examining the interpretability-performance trade-off in EHR data.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec22\" class=\"Section2\"\u003e \u003ch2\u003e5.2. Clinically optimal decision rules\u003c/h2\u003e \u003cp\u003eFurther, we showed that learning under constraints with EXPLORE can be used to find a set of near-optimal models. We found that the set of good candidate models is very large (1,381\u0026thinsp;\u0026minus;\u0026thinsp;20,320 for maximum rule length 5) for all prediction tasks. Moreover, we examined the Rashomon set of models and found there is a large variability in both the generalizability and features included. This shows there is a potential to find more clinically optimal decision rules. Earlier work concluded that a large Rashomon set means an equally well-performing, interpretable model is likely to exist and correlates with the existence of models with good generalizability (\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e, \u003cspan citationid=\"CR47\" class=\"CitationRef\"\u003e47\u003c/span\u003e). The ability to enhance the interpretability of the models was demonstrated by the experiments incorporating domain knowledge; both incorporating age/sex and task-specific features resulted in decision rules with similar performance on average and very small differences overall. These constraints can also lead to significant reductions in computation time by reducing the search space. Finally, we showed we can improve the original CHADS\u003csub\u003e2\u003c/sub\u003e score with EXPLORE by allowing EXPLORE to select additional covariates on top of the original score. This approach can also be applied to model updating when there is a desire for model stability over time (i.e. similar features in the model), as it keeps the original model intact. Learning under constraints with EXPLORE provides a novel solution to open challenges for (optimal) rule-based methods such as scalability and addressing user preferences (\u003cspan citationid=\"CR48\" class=\"CitationRef\"\u003e48\u003c/span\u003e). Furthermore, access to the Rashomon set of models is an asset of EXPLORE, shared by only a few other model algorithms (e.g. TreeFARMS (\u003cspan citationid=\"CR49\" class=\"CitationRef\"\u003e49\u003c/span\u003e), FasterRisk (\u003cspan citationid=\"CR50\" class=\"CitationRef\"\u003e50\u003c/span\u003e)).\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec23\" class=\"Section2\"\u003e \u003ch2\u003e5.3. Limitations\u003c/h2\u003e \u003cp\u003eIn this study we have used model size as a proxy measure for interpretability, which might not directly translate to human interpretability. Likewise, we have not been able to assess the human-machine task performance, which is ultimately what matters in real-world applications. This may both impact the true interpretability-performance trade-off. To assess this, user studies need to be performed (\u003cspan citationid=\"CR51\" class=\"CitationRef\"\u003e51\u003c/span\u003e). Additionally, while we studied a wide range of prediction tasks, the trade-off and best choice of model algorithm may differ based on the specific clinical application, which was not investigated in this study.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec24\" class=\"Section2\"\u003e \u003ch2\u003e5.4. Future work\u003c/h2\u003e \u003cp\u003eFirst, we may continue to improve EXPLORE by further enhancements in speed thereby allowing longer decision rules (e.g. reducing the search space using reversed class labels) or supporting direct optimization over rank statistics (e.g. AUROC). Second, as external validation is a good indicator of model generalizability, this might expose more benefits of simple models and hence the interpretability-performance trade-off could be further investigated when transporting models to other databases. Finally, developing and testing search strategies to find the clinically optimal model from the Rashomon set of models is an interesting direction of research.\u003c/p\u003e \u003c/div\u003e"},{"header":"6. CONCLUSIONS","content":"\u003cp\u003eOur study shows that more complex models generally outperform simpler ones, confirming the expected trade-off between model performance and interpretability. However, the observed trade-off varies across prediction tasks. While none of the complex models systematically outperforms the others, the simple models show significant variations in performance. EXPLORE\u0026rsquo;s ability to learn under constraints makes it an attractive candidate to find clinically optimal decision rules.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eEthics approval and consent to participate\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe current study was approved by the IPCI Governance Board under number 3/2023. No human participants were involved in the study and informed consent was not necessary.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConsent for publication\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNot applicable.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAvailability of data and materials\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe datasets generated and/or analysed during the current study are not publicly available due to patient privacy and data protection concerns. The code for EXPLORE and the experiments is available on GitHub.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting interests\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe authors declare that they have no competing interests.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis project has received support from the European Health Data and Evidence Network (EHDEN) project. EHDEN received funding from the Innovative Medicines Initiative 2 Joint Undertaking (JU) under grant agreement No 806968. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and EFPIA.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthor contributions\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eA.M., J.K. and P.R. conceived of the presented idea. A.M. designed the study, carried out the implementation, and performed the experiments. J.K, E.F., K.V., and P.R. provided critical feedback and helped shape the research. A.M. wrote the article with input from all other authors.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAcknowledgements\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWe would like to thank Cesar Barboza Gutierrez for his contribution to testing the EXPLORE R package.\u0026nbsp;\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eCosta VG, Pedreira CE. Recent advances in decision trees: An updated survey. Artificial Intelligence Review. 2023;56(5):4765-800. https://doi.org/10.1007/s10462-022-10275-5.\u003c/li\u003e\n\u003cli\u003eLipton ZC. The mythos of model interpretability. Queue. 2018:31\u0026ndash;57. https://doi.org/10.1145/3236386.3241340.\u003c/li\u003e\n\u003cli\u003eMarkus AF, Kors JA, Rijnbeek PR. The role of explainability in creating trustworthy artificial intelligence for health care: A comprehensive survey of the terminology, design choices, and evaluation strategies. J Biomed Inform. 2021;113:103655. https://doi.org/10.1016/j.jbi.2020.103655.\u003c/li\u003e\n\u003cli\u003eYang C, Kors JA, Ioannou S, John LH, Markus AF, Rekkas A, et al. Trends in the conduct and reporting of clinical prediction model development and validation: A systematic review. J Am Med Inform Assoc. 2022;29(5):983-9. https://doi.org/10.1093/jamia/ocac002.\u003c/li\u003e\n\u003cli\u003eRijnbeek PR, Kors JA. Finding a short and accurate decision rule in disjunctive normal form by exhaustive search. Machine Learning. 2010;80(1):33-62. https://doi.org/10.1007/s10994-010-5168-9.\u003c/li\u003e\n\u003cli\u003eLaurent H, Rivest RL. Constructing optimal binary decision trees is NP-complete. Information Processing Letters. 1976;5(1):15-7. https://doi.org/10.1016/0020-0190(76)90095-8.\u003c/li\u003e\n\u003cli\u003eAglin G, Nijssen S, Schaus P, editors. Learning optimal decision trees using caching branch-and-bound search. AAAI; 2020 04-08-2020.\u003c/li\u003e\n\u003cli\u003eVerwer S, Zhang Y, editors. Learning optimal classification trees using a binary linear program formulation. Proceedings of the AAAI Conference on Artificial Intelligence; 2019 22-01-2020.\u003c/li\u003e\n\u003cli\u003eLin J, Zhong C, Hu D, Rudin C, Seltzer M. Generalized optimal sparse decision trees. arXiv preprint. 2020. https://doi.org/10.48550/arXiv.2006.08690.\u003c/li\u003e\n\u003cli\u003eLakkaraju H, Bach SH, Leskovec J, editors. Interpretable decision sets: A joint framework for description and prediction. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; 2016: ACM.\u003c/li\u003e\n\u003cli\u003eDash S, Gunluk O, Wei D, editors. Boolean decision rules via column generation. Adv Neural Inf Process Syst; 2018 16-01-2020.\u003c/li\u003e\n\u003cli\u003eMita G, Papotti P, Filippone M, Michiardi P, editors. Libre: Learning interpretable boolean rule ensembles. International Conference on Artificial Intelligence and Statistics; 2020: PMLR.\u003c/li\u003e\n\u003cli\u003eRudin C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell. 2019;1(5):206-15. https://doi.org/10.1038/s42256-019-0048-x.\u003c/li\u003e\n\u003cli\u003eBell A, Solano-Kamaiko I, Nov O, Stoyanovich J, editors. It\u0026rsquo;s just not that simple: An empirical study of the accuracy-explainability trade-off in machine learning for public policy. 2022 ACM Conference on Fairness, Accountability, and Transparency; 2022.\u003c/li\u003e\n\u003cli\u003eLeblanc B, Germain P. On the relationship between interpretability and explainability in machine learning. arXiv preprint. 2024. https://doi.org/10.48550/arXiv.2311.11491.\u003c/li\u003e\n\u003cli\u003eCaruana R, Lou Y, Gehrke J, Koch P, Sturm M, Elhadad N, editors. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; 2015 18-11-2019.\u003c/li\u003e\n\u003cli\u003eGosiewska A, Kozak A, Biecek P. Simpler is better: Lifting interpretability-performance trade-off via automated feature engineering. Decis Support Syst. 2021;150:113556. https://doi.org/10.1016/j.dss.2021.113556.\u003c/li\u003e\n\u003cli\u003eMcElfresh D, Khandagale S, Valverde J, Prasad C V, Ramakrishnan G, Goldblum M, et al. When do neural nets outperform boosted trees on tabular data? Adv Neural Inf Process Syst. 2024;36. https://doi.org/10.48550/arXiv.2305.02997.\u003c/li\u003e\n\u003cli\u003eLiu J, Zhong C, Seltzer M, Rudin C. Fast sparse classification for generalized linear and additive models. Proceedings of Machine Learning Research. 2022;151:9304. https://doi.org/https://proceedings.mlr.press/v151/liu22f/liu22f.pdf.\u003c/li\u003e\n\u003cli\u003eLetham B, Rudin C, McCormick TH, Madigan D. Interpretable classifiers using rules and bayesian analysis: Building a better stroke prediction model. Ann Appl Stat. 2015;9(3):1350-71. https://doi.org/10.1214/15-AOAS848.\u003c/li\u003e\n\u003cli\u003eSemenova L, Rudin C, Parr R, editors. On the existence of simpler machine learning models. 2022 ACM Conference on Fairness, Accountability, and Transparency; 2022.\u003c/li\u003e\n\u003cli\u003eTonekaboni S, Joshi S, McCradden MD, Goldenberg A. What clinicians want: Contextualizing explainable machine learning for clinical end use. Proceedings of Machine Learning research. 2019:1-21. https://doi.org/http://proceedings.mlr.press/v106/tonekaboni19a/tonekaboni19a.pdf.\u003c/li\u003e\n\u003cli\u003ePayrovnaziri SN, Chen Z, Rengifo-Moreno P, Miller T, Bian J, Chen JH, et al. Explainable artificial intelligence models using real-world electronic health record data: A systematic scoping review. J Am Med Inform Assoc. 2020. https://doi.org/10.1093/jamia/ocaa053.\u003c/li\u003e\n\u003cli\u003eJaotombo F, Adorni L, Ghattas B, Boyer L. Finding the best trade-off between performance and interpretability in predicting hospital length of stay using structured and unstructured data. PLoS One. 2023;18(11):e0289795. https://doi.org/10.1371/journal.pone.0289795.\u003c/li\u003e\n\u003cli\u003ede Ridder MAJ, de Wilde M, de Ben C, Leyba AR, Mosseveld BMT, Verhamme KMC, et al. Data resource profile: The integrated primary care information (IPCI) database, the Netherlands. Int J Epidemiol. 2022;51(6):e314-e23. https://doi.org/10.1093/ije/dyac026.\u003c/li\u003e\n\u003cli\u003eHasan O, Meltzer DO, Shaykevich SA, Bell CM, Kaboli PJ, Auerbach AD, et al. Hospital readmission in general medicine patients: A prediction model. J Gen Intern Med. 2010;25:211-9. https://doi.org/10.1007/s11606-009-1196-1.\u003c/li\u003e\n\u003cli\u003eKansagara D, Englander H, Salanitro A, Kagen D, Theobald C, Freeman M, et al. Risk prediction models for hospital readmission: A systematic review. JAMA. 2011;306(15):1688-98. https://doi.org/10.1001/jama.2011.1515.\u003c/li\u003e\n\u003cli\u003eThoonsen B, Engels Y, van Rijswijk E, Verhagen S, van Weel C, Groot M, et al. Early identification of palliative care patients in general practice: Development of radboud indicators for palliative care needs (RADPAC). Br J Gen Pract. 2012;62(602):e625-e31. https://doi.org/10.3399/bjgp12x654597.\u003c/li\u003e\n\u003cli\u003eElMokhallalati Y, Bradley SH, Chapman E, Ziegler L, Murtagh FE, Johnson MJ, et al. Identification of patients with potential palliative care needs: A systematic review of screening tools in primary care. Palliat Med. 2020;34(8):989-1005. https://doi.org/10.1177/0269216320929552.\u003c/li\u003e\n\u003cli\u003eTibble H, Tsanas A, Horne E, Horne R, Mizani M, Simpson CR, et al. Predicting asthma attacks in primary care: Protocol for developing a machine learning-based prediction model. BMJ Open. 2019;9(7):e028375. https://doi.org/10.1136/bmjopen-2018-028375.\u003c/li\u003e\n\u003cli\u003eZein JG, Wu C-P, Attaway AH, Zhang P, Nazha A. Novel machine learning can predict acute asthma exacerbation. Chest. 2021;159(5):1747-57. https://doi.org/10.1016/j.chest.2020.12.051.\u003c/li\u003e\n\u003cli\u003eBloom C, Ricciardi F, Smeeth L, Stone P, Quint J. Predicting COPD 1-year mortality using prognostic predictors routinely measured in primary care. BMC Med. 2019;17:1-10. https://doi.org/10.1186/s12916-019-1310-0.\u003c/li\u003e\n\u003cli\u003eDziopa K, Asselbergs FW, Gratton J, Chaturvedi N, Schmidt AF. Cardiovascular risk prediction in type 2 diabetes: A comparison of 22 risk scores in primary care settings. Diabetologia. 2022;65(4):644-56. https://doi.org/10.1007/s00125-021-05640-y.\u003c/li\u003e\n\u003cli\u003eReps JM, Schuemie MJ, Suchard MA, Ryan PB, Rijnbeek PR. Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data. J Am Med Inform Assoc. 2018;25(8):969-75. https://doi.org/10.1093/jamia/ocy032.\u003c/li\u003e\n\u003cli\u003eReps J, Wong J, Fridgeirsson EA, Kim C, John LH, Williams R, et al. Finding a constrained number of predictor phenotypes for multiple outcome prediction. SSRN preprint. 2024. https://doi.org/https://dx.doi.org/10.2139/ssrn.4874415.\u003c/li\u003e\n\u003cli\u003eBreiman L, Friedman J, Olshen R, Stone C. CART: Classification and regression trees1984.\u003c/li\u003e\n\u003cli\u003eMcTavish H, Zhong C, Achermann R, Karimalis I, Chen J, Rudin C, et al. Fast sparse decision tree optimization via reference ensembles. Proceedings of the AAAI Conference on Artificial Intelligence. 2022;36(9):9604-13. https://doi.org/10.1609/aaai.v36i9.21194.\u003c/li\u003e\n\u003cli\u003eBlumensath T, Davies ME. Iterative hard thresholding for compressed sensing. Applied and Computational Harmonic Analysis. 2009;27(3):265-74. https://doi.org/10.1016/j.acha.2009.04.002.\u003c/li\u003e\n\u003cli\u003eSuchard MA, Simpson SE, Zorych I, Ryan P, Madigan D. Massive parallelization of serial inference algorithms for a complex generalized linear model. ACM Transactions on Modeling and Computer Simulation (TOMACS). 2013;23(1):1-17. https://doi.org/10.1145/2414416.2414791.\u003c/li\u003e\n\u003cli\u003eBreiman L. Random forests. Machine Learning. 2001;45(1):5-32. https://doi.org/10.1023/A:1010933404324.\u003c/li\u003e\n\u003cli\u003eCohen WW, editor Fast effective rule induction. Machine Learning Proceedings; 1995 18-12-2019: Elsevier.\u003c/li\u003e\n\u003cli\u003eChen T, Guestrin C, editors. Xgboost: A scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; 2016.\u003c/li\u003e\n\u003cli\u003eGuidotti R, Monreale A, Ruggieri S, Turini F, Giannotti F, Pedreschi D. A survey of methods for explaining black box models. ACM Computing Surveys. 2018;51(5):93. https://doi.org/10.1145/3236009.\u003c/li\u003e\n\u003cli\u003eBreiman L. Statistical modeling: The two cultures. Statistical Science. 2001;16(3):199-231, 33. https://doi.org/10.1214/ss/1009213726.\u003c/li\u003e\n\u003cli\u003eNogueira S, Sechidis K, Brown G. On the stability of feature selection algorithms. J Mach Learn Res. 2018;18(1):6345-98. https://doi.org/https://jmlr.org/papers/volume18/17-514/17-514.pdf.\u003c/li\u003e\n\u003cli\u003eGage BF, Waterman AD, Shannon W, Boechler M, Rich MW, Radford MJ. Validation of clinical classification schemes for predicting stroke results from the national registry of atrial fibrillation. JAMA. 2001;285(22):2864-70. https://doi.org/10.1001/jama.285.22.2864.\u003c/li\u003e\n\u003cli\u003eRudin C, Zhong C, Semenova L, Seltzer M, Parr R, Liu J, et al. Amazing things come from having many good models. arXiv preprint. 2024. https://doi.org/10.48550/arXiv.2407.04846.\u003c/li\u003e\n\u003cli\u003eRudin C, Chen C, Chen Z, Huang H, Semenova L, Zhong C. Interpretable machine learning: Fundamental principles and 10 grand challenges. Statistics Surveys. 2022;16:1-85. https://doi.org/10.1214/21-SS133.\u003c/li\u003e\n\u003cli\u003eXin R, Zhong C, Chen Z, Takagi T, Seltzer M, Rudin C. Exploring the whole rashomon set of sparse decision trees. Adv Neural Inf Process Syst. 2022;35:14071-84. https://doi.org/https://proceedings.neurips.cc/paper_files/paper/2022/file/5afaa8b4dd18eb1eed055d2d821b58ae-Paper-Conference.pdf.\u003c/li\u003e\n\u003cli\u003eLiu J, Zhong C, Li B, Seltzer M, Rudin C. FasterRisk: Fast and accurate interpretable risk scores. Adv Neural Inf Process Syst. 2022;35:17760-73. https://doi.org/https://proceedings.neurips.cc/paper_files/paper/2022/file/7103444259031cc58051f8c9a4868533-Paper-Conference.pdf.\u003c/li\u003e\n\u003cli\u003ePoursabzi-Sangdeh F, Goldstein DG, Hofman JM, Wortman Vaughan JW, Wallach H, editors. Manipulating and measuring model interpretability. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems; 2021.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Clinical prediction modelling, Optimal decision rule, Interpretability-performance trade-off, Data-and knowledge-driven learning","lastPublishedDoi":"10.21203/rs.3.rs-5804837/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5804837/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eObjective\u003c/h2\u003e \u003cp\u003eWe investigate whether a trade-off occurs between predictive performance and model interpretability in real-world health care data and illustrate how to develop \u003cem\u003eclinically optimal\u003c/em\u003e decision rules by learning under constraints with the Exhaustive Procedure for Logic-Rule Extraction (EXPLORE) algorithm.\u003c/p\u003e\u003ch2\u003eMethods\u003c/h2\u003e \u003cp\u003eWe enhanced EXPLORE\u0026rsquo;s scalability to enable its use with real-world datasets and developed an R package that generates simple decision rules. We compared EXPLORE\u0026rsquo;s performance to 7 state-of-the-art model algorithms across 5 prediction tasks using data from the Dutch Integrated Primary Care Information (IPCI) database. Additionally, we characterized EXPLORE\u0026rsquo;s space of near-optimal models (i.e. Rashomon set) and conducted experiments on incorporating domain knowledge and improving existing models.\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e \u003cp\u003eThe prediction models developed using LASSO, RandomForest, and XGBoost consistently performed best in terms of AUROC, followed by DecisionTree and EXPLORE. However, the decision rules generated by EXPLORE are much simpler (at most 5 predictors) than the aforementioned. GOSDT-G, IHT, and RIPPER performed worse. Moreover, we demonstrated that EXPLORE\u0026rsquo;s Rashomon set is very large (1,381\u0026thinsp;\u0026minus;\u0026thinsp;20,320 models) with a large variability in both the generalizability and model diversity. We then showed there is a potential to find more clinically optimal decision rules using EXPLORE by incorporating domain knowledge (age/sex and task-specific features) or improving existing models (the CHADS\u003csub\u003e2\u003c/sub\u003e score).\u003c/p\u003e\u003ch2\u003eConclusions\u003c/h2\u003e \u003cp\u003eOur study shows that more complex models generally outperform simpler ones, confirming the expected interpretability-performance trade-off, although it varies in strength across prediction tasks. EXPLORE\u0026rsquo;s ability to learn under constraints is valuable for generating clinically optimal decision rules.\u003c/p\u003e","manuscriptTitle":"EXPLORE: learning interpretable rules for patient-level prediction","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-01-20 09:50:30","doi":"10.21203/rs.3.rs-5804837/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"83ce5fa0-12e1-4077-acf1-9e67fcb3b214","owner":[],"postedDate":"January 20th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-01-20T16:23:44+00:00","versionOfRecord":[],"versionCreatedAt":"2025-01-20 09:50:30","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5804837","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5804837","identity":"rs-5804837","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}