{"paper_id":"2f648d1f-8a4e-45bf-a26b-ee4f9d56310a","body_text":"Can ensemble methods improve predictive performance of existing models estimating chronic kidney disease among patients with diabetes? | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Can ensemble methods improve predictive performance of existing models estimating chronic kidney disease among patients with diabetes? Jason E. Black, David J.T. Campbell, Paul E. Ronksley, Kerry A. McBrien, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6297944/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Clinical prediction models often suffer from poor model transportability and/or subgroup performance resulting from using a single data source. We aimed to determine whether ensemble methods can combine multiple existing models to improve predictive performance when compared to component models. As a case study, we used electronic medical records from the Canadian Primary Care Sentinel Surveillance Network (CPCSSN) to test ensemble methods for models estimating the risk of developing chronic kidney disease (CKD) among people with diabetes in a cohort of 37,604 individuals. We considered 13 models identified from prior systematic reviews and combined their unique risk estimates using many strategies (e.g., averaging or mixture-of-experts). We assessed discrimination, precision, recall, calibration, net reclassification index, and integrated discrimination improvement. Ensemble methods performed well, but no better than the best performing component model. This study suggests ensemble methods may not improve predictive performance, though further research should confirm these findings. Figures Figure 1 Figure 2 Figure 3 Figure 4 Introduction Clinical prediction models are typically used to estimate disease risk in clinical settings and are often developed using a single data source, such as a health administrative database. However, research has consistently shown that models rarely perform as strongly in different settings and/or populations 1 . Poor model transportability (i.e., model performance in a new or different setting) can arise when the data source used to develop the model is not representative of the population where the model is to be used in practice 2 . Similarly, exclusion of important groups, such as females, older people, and those with multiple comorbidities, during model development may result in poor model performance when estimating disease risk among these groups. Commonly, multiple clinical prediction models for the same outcome have been developed independently by different researchers from various countries and/or health systems. Thus, the transportability and performance of clinical prediction models might theoretically be improved using ensemble methods—a type of machine learning that can combine models. Like meta-analyses that synthesize information from multiple studies, ensemble methods have the potential to leverage predictive relationships from many existing models to improve predictive accuracy and reduce selection bias 3 . Ensemble methods (also referred to as ensembles, ensemble models, or ensemble learning algorithms) operate under the assumption that many weak models operating in concert will outperform any single best-performing model 4 ; this assumption has been confirmed in several studies 5 – 7 . When developing an ensemble model traditionally, multiple models are fit to the same dataset while varying the type of model (e.g., logistic regression, decision tree, naïve Bayes), predictors considered by the model, and/or observations used to fit the model. When predicting the outcome for a new person whose outcome is unknown, each component model is evaluated and predictions are combined by some method, such as majority voting where the most frequently predicted outcome is assigned. Ensemble methods have been successfully applied to develop novel clinical prediction models using a single data source [8]. However, it remains unknown whether ensemble methods can improve predictive performance by combining multiple existing clinical prediction models previously developed across a variety of data sources. Case study: Clinical prediction models estimating the risk of developing chronic kidney disease among people with diabetes Diabetes is a common chronic condition associated with considerable morbidity worldwide 8 . As diabetes continues to increase in prevalence, the burden of complications resulting from this condition are also on the rise 9 . In particular, cardiovascular disease, as well as microvascular complications (including nephropathy leading to chronic kidney disease [CKD]) are known to result from all forms of diabetes. However, diabetes-self management resulting in optimized blood glucose levels, blood pressure, and cholesterol levels can reduce the risk of vascular complications. As most people with diabetes are managed by primary care practitioners, opportunities to optimize care and prevent diabetes complications exist in primary care 10 . Indeed, many clinical prediction models have been developed to estimate risk of developing CKD among people with diabetes to support preventative care 11 , 12 . Objectives We aimed to determine whether ensemble methods can improve predictive performance by combining existing clinical prediction models compared to the individual component models, using the risk of CKD development in people with diabetes as a case study. We assessed for improved performance overall and among subgroups defined by sex/gender, age group, number of comorbidities, and level of social and material deprivation. Methods Study setting and data source We used electronic medical record (EMR) data from the Canadian Primary Care Sentinel Surveillance Network (CPCSSN), which was established in 2008 as a Canada-wide repository of primary care records for more than 2 million patients (excluding Saskatchewan) 13 . Patient information (e.g., diagnoses, procedures, laboratory tests, medication prescriptions, and referrals) as recorded by the primary care practitioner is de-identified and contributed to 14 regional networks that comprise CPCSSN. Prior work has found that patients in CPCSSN tend to be more likely to be female and older than the general population in Canada 14 , though this is typical of primary care patients 15 . CPCSSN facilitates research use by cleaning and standardising patient records (e.g., assigning diagnostic codes to free-text diagnoses) 16 . For this study, we used CPCSSN data from January 2014 to December 2019. Models Two recent systematic reviews identified 47 clinical prediction models for incident CKD among people with diabetes 11 , 12 . We previously evaluated the performance of 13 models for incident CKD with predictors available among Canadian primary care patients and found mixed performance when predicting incident CKD in this setting 17 . We considered these clinical prediction models to develop an ensemble model for incident CKD. Participants We identified adult (18+) patients with any form of diabetes (excluding gestational) in CPCSSN using validated case definitions 18 , 19 . We identified baseline visits for each patient between January to December 2014, then included patients without an existing diagnosis of CKD (see §Measures ) prior to baseline, with at least 3 primary care visits in the 2 years prior to baseline, and with at least one visit within 5 years after baseline. For patients who were identified as having type 1 diabetes 19 , we excluded those who were diagnosed within 5 years before baseline as these patients were not yet recommended for CKD screening per clinical practice guidelines 20 , 21 . We excluded patients from Quebec and Manitoba due to data quality issues in the recording of kidney function [estimated glomerular fitration rate (eGFR)] laboratory tests. We noted those with incident CKD diagnoses within 5 years from baseline and censored patients at their last visit within 5 years from baseline. Per clinical guidelines, we identified incident CKD as two or more serum creatinine laboratory values which correspond to an eGFR less than 60 mL/min/1.73 m 2 that were separated by at least 90 days but no more than 1 year apart 22 . Measures We identified predictors of CKD in CPCSSN using (in the following order): directly reported values (e.g., laboratory test results for hemoglobin A1c [HbA1c]); case definitions previously validated in CPCSSN (e.g., hypertension); case definitions previously applied in CPCSSN but not validated (e.g., macrovascular complications); or case definitions developed ad-hoc in consultation with clinical colleagues. See Appendix A for details on how we identified each predictor. We considered subgroups defined by sex/gender (female/woman and male/man), age group (18 to 39, 40 to 64, and 65 and older), number of comorbidities (none, one, two, or three or more), and social and material deprivation quintile. We identified sex/gender as recorded in the EMR by the practitioner—though we cannot be certain which was recorded. We calculated age at baseline visit based on patient birth dates. We identified the following comorbidities using validated case definitions: cardiovascular disease; liver cirrhosis; chronic obstructive pulmonary disease; dementia; depression; epilepsy; hypertension; osteoarthritis; Parkinson’s disease; obesity; and dyslipidemia 18 , 23 – 26 . We estimated patient deprivation quintiles using the neighbourhood-level Pampalon social and material deprivation indices mapped to patient postal codes 27 . Sample size We estimated the required sample size for a reasonable range of anticipated areas under the receiver operating characteristic curve (AUROC) and incidences of CKD based on those of existing CKD clinical prediction models. We required from 689 to 21,925 patients to estimate 95% confidence intervals for the AUROC with a width of roughly 0.04 and the calibration intercept and slope both with widths of roughly 0.2 28 . Missing data We accounted for missing data using single imputation. This approach enabled comparisons between point estimates while ensuring computation feasibility. However, precision of the confidence intervals for these estimates may be artificially inflated. Statistical analysis We described the baseline predictors in our cohort using means (standard deviations), medians (first and third quartiles), and frequencies (percentages), as appropriate. We determined the incidence proportion (%) of CKD diagnosis over 5 years of follow-up. We combined risk estimates from models that were previously developed using the procedure presented in Fig. 1 . We restricted the ensemble methods we considered to those that did not estimate new model parameters (e.g., stacking) based on a dataset (i.e., CPCSSN diabetes patients). Instead, we relied on data-independent approaches (e.g., bagging) to combine models then validated the resulting ensemble model in CPCSSN patients with diabetes. First, we considered the availability of predictors within CPCSSN. We excluded models without information on ≥2 predictors in CPCSSN to include many models while minimizing the loss of performance associated with missing predictors 29 . Next, we updated the models by re-estimating model intercepts and scaling model coefficients (i.e., reducing or increasing the magnitude of all coefficients by the same factor) using the CPCSSN cohort 30 . Model updating helps correct for differences in CKD incidence and prediction horizons (e.g., estimating 3- vs. 5-year CKD risks) between CPCSSN and the development cohort. For example, we assume that 3- and 5-year CKD risks should rank patients similarly, despite assigning different absolute values of risk. Updating corrects for differences in the mean and distribution of risks while retaining their ranking. For each patient, we estimated several 5-year CKD risks by using all updated models. We combined each patient’s risk estimates through an averaging and/or selection process, with or without weighting according to various metrics (Table 1 ). We considered two types of means when averaging risk estimates: the arithmetic mean and the geometric mean (i.e., the mean computed on the logarithmic scale converted back to the original scale). A mixture-of-experts process 31 selected the most appropriate models and corresponding risk estimates for each patient according to similarity or age-similarity and computed the arithmetic mean to determine their estimated risk. Similarly, the median-of-experts process selected the median estimated risk among the most appropriate models for each patient according to similarity or age-similarity to determine their estimated risk. We tested the mixture- and median-of-experts approaches using the 1, 3 and 5 most similar models. We weighted estimates or selected a model according to factors that may be associated with improved model performance (i.e., development cohort size; development AUROCs; similarity to the development cohort according to age, sex/gender, recorded HbA1c, and BMI; and similarity to the development cohort in age). We measured similarity based on the Euclidean distance after standardizing age, HbA1c, and BMI. Table 1 Averaging or selection processes and corresponding weighting techniques, where applicable None Development cohort size Development AUROC Similarity Age-similarity Arithmetic mean X X X X X Geometric mean X X X X X Mixture-of-experts X X Median-of-experts X X We evaluated the performance of the resulting ensemble models in terms of their discrimination (AUROC), precision and recall (area under the precision-recall curve), and calibration (calibration plots). We compared the ensemble results to the best performing component model (in terms of AUROC) by examining differences in discrimination, precision, recall, and calibration and estimating the category-free net reclassification index (NRI > 0 ) and integrated discrimination improvement (IDI) 32 , 33 . Comparing a new model to a reference model, the NRI > 0 measures whether individual estimated risks based on the new model increase for those who experience the outcome or decrease for those who do not experience the outcome. Similarly, IDI considers whether the average estimated risk based on the new model increases among those who experience the outcome and decreases among those who do not experience the outcome. From these comparisons, we determined 1) which averaging or selection process resulted in the best performing ensemble model and 2) whether ensemble models outperform their component models. Subgroup analysis We evaluated the performance of each ensemble model among groups defined by sex/gender, age (18 to 39, 40 to 64, and 65 and older), number of comorbidities (none, one, two, or three or more), and material and social deprivation quintile based on the Pampalon index 27 . Ethics approval Our study was approved by the University of Calgary Conjoint Health Research Ethics Board under study ID REB21-1741. Results We identified 37,604 patients with diabetes in CPCSSN to assess the performance of ensemble methods using several existing clinical prediction models for incident CKD. Patient characteristics are summarized in Table 2 and Appendix B. CPCSSN patients with diabetes were on average 63.4 (SD: 12.2) years old, were more frequently male/men (53.0%) than female/women (47.0%), frequently had HbA1c values below 7.0% (median: 6.7%; first—third quartiles: 6.3—7.4), and had a mean body mass index (BMI) of 32.3 (SD: 7.4) kg/m 2 . Table 2 Characteristics of CPCSSN diabetes cohort for validation of ensemble models N = 37,604 Age group, n (%) 18 to 39 1,073 (2.9) 40 to 64 18,370 (48.9) 65 and older 18,161 (48.3) Sex/gender, n (%) Male/men 23,174 (53.0) Female/women 20,516 (47.0) Social and material deprivation, n (%) 1st quintile (least deprived) 6,564 (17.5) 2nd quintile 7,521 (20.0) 3rd quintile 7,032 (18.7) 4th quintile 7,123 (18.9) 5th quintile (most deprived) 7,039 (18.7) Missing 2,325 (6.2) Urban residence, n (%) 29,088 (77.4) HbA1c (%), median (first–third quartiles) 6.7 (6.3—7.4) Diabetes duration (years), median (first–third quartiles) 2.8 (1.3—5.1) Non-insulin antihyperglycemic drugs, n (%) 21,460 (57.1) Insulin, n (%) 4,128 (11.0) BMI (kg/m 2 ), mean ± SD 32.3 ± 7.4 Baseline eGFR (mL/min/1.73 m 2 ), mean ± SD 84 ± 18 Urine Albumin-to-Creatinine Ratio (mg/mmol), median (first–third quartiles/IQR) 1.3 (0.6—2.9) Comorbidities, n (%) None 3,692 (9.8) One 9,062 (24.1) Two 10,826 (28.8) Three or more 14,024 (37.3) RAS-antagonist: renin–angiotensin system antagonist. We identified incident CKD among 14.6% of CPCSSN patients with diabetes over an average follow-up of 4.8 years, though we identified incident CKD more frequently among older patients and those with multiple comorbidities (Fig. 2 ). We previously found that the performance of existing clinical prediction models for CKD among CPCSSN patients with diabetes was mixed, with AUROCs ranging from 0.492 to 0.826 17 . The best performing model according to AUROC and AUPRC was developed by Nelson et al. 34 (AUROC: 0.826 [95% CI: 0.820 to 0.832]; AUPRC: 0.467 [95% CI: 0.462 to 0.472]). Ensemble methods displayed strong performance predicting CKD incidence in CPCSSN according to AUROC and AUPRC (Figs. 3 a and 3 b); however, no ensemble method had better performance than the best performing component model. Among ensemble methods, the averaging or selection process with the best performance weighted the predictions from all component models by their development cohort size (AUROC: 0.827 [95% CI: 0.821 to 0.833]). Based on the NRI > 0 , estimated risks based on the ensemble methods were often worse (i.e., for those that developed CKD, risk estimates were further from 1, whereas for those that did not develop CKD, risk estimates were further from 0) than the best performing component model (Fig. 3 c), except for the geometric mean for all component models without weighting (NRI > 0 : 0.062 [95% CI: 0.052 to 0.071]). Similarly, the IDI for all ensemble methods showed that risk estimates were less accurate compared to the best component model: the average estimated risk for all patients that developed CKD was further from 1 and the average estimated risk for all patients that did not develop CKD was further from 0 (Fig. 3 d). All ensemble models demonstrated some miscalibration: the calibration plot for the ensemble model with the best AUROC which weighted the predictions from all component models by their development cohort size showed overestimation of lower risks (Fig. 4 ). All calibration plots are available in Appendix C. We found that mixture- and median-of-experts ensemble methods resulted in poorer performance when compared to weighting techniques that considered all component models, though performance improved as more component models were included. In most cases, the ensemble methods that relied on the geometric mean performed slightly better than the corresponding techniques that relied on the arithmetic mean. Performance was not improved across subgroups when using ensemble methods. Instead, performance between subgroups followed the same patterns as when using the best performing component model (Appendices D to G). Discussion We assessed the performance of several models using ensemble methods to combine 13 previously developed clinical prediction models estimating the risk of incident CKD among patients with diabetes. While the performance of some ensemble methods equalled that of the best performing component model, no ensemble method was able to surpass this level of performance. The best ensemble method performance according to AUROC and AUPRC was achieved by computing the geometric mean of all component model predictions weighted based on development cohort size (AUROC: 0.827 [95% CI: 0.821 to 0.833]; AUPRC: 0.471 [95% CI: 0.466 to 0.476]). However, this ensemble method had similar performance to the best performing component model based on AUROC (i.e., the Nelson et al. model; AUROC: 0.826 [95% CI: 0.82 to 0.826]; AUPRC: 0.467 [95% CI: 0.462 to 0.472]). Overlapping confidence intervals suggested these performance measures were not statistically different. Indeed, due to the considerable size of the development cohort used by Nelson et al. compared to all others (Nelson et al. combined several datasets for a development cohort totalling 781,627 people, compared to 43,362 people among all other development cohorts combined), the ensemble predictions based on development cohort size weighting were almost the same as the Nelson et al. model predictions. According to NRI > 0 , improved predictions compared to the best performing component model were only achieved by the geometric mean without weighting; however, this improvement was small (0.062 [95% CI: 0.052 to 0.071]). No ensemble methods displayed improvement compared to the best performing component model considering the IDI. Our results suggest that this application of ensemble methods may not provide improved predictive performance compared to its component models. Several reasons may explain why ensemble methods failed to improve upon existing clinical prediction models for CKD in our analysis. Foremost, ensemble methods were designed to combine many models with poor performance (a.k.a., weak learners) to create a model that outperforms all its component models 4 . Applying existing clinical prediction models for CKD in CPCSSN diabetes patients, we observed mixed performances—many models performed poorly but some performed quite well (AUROC > 0.80; a.k.a., strong learners). As such, the addition of weak learners to strong learners may not have resulted in improvements upon the performance of the strong learners. Further, while 13 models were included in our ensemble methods, some ensemble methods employ more than 500 to 1,000 component models 3 , 4 . It is possible that more models are required to observe increased performance using ensemble methods; however, this limits the feasibility of this technique as each model must be operationalized within the validation dataset. Ensemble methods have not frequently been utilized to combine existing clinical prediction models into a single ensemble model—instead, research has focused on developing novel clinical prediction models using ensemble methods, despite calls to externally validate existing models 35 . However, one study that used ensemble methods with existing models was performed by Wu et al. 36 , where they combined 7 previously developed clinical prediction models to identify individuals at high-risk of experiencing poor outcomes related to coronavirus disease (COVID-19). Wu et al. used similar ensemble methods, such as weighted averaging and mixture of experts, to combine predictions from each component model and assess their performance across 4 validation datasets. They found that ensemble methods could perform well; however, overlapping confidence intervals suggested they did not outperform the best performing component model in each validation dataset. These results are consistent with our findings evaluating ensemble methods to combine CKD clinical prediction models—though Wu et al. found consistent improvements in predictions from an ensemble method based on the NRI, whereas we only observed such an improvement for one ensemble method. Strengths and limitations Our study used a large cohort of more than 30,000 patients to test the performance of existing prediction models for CKD and determined whether ensemble methods might improve upon these performances. We leveraged real patient data to measure model performances, thus we can be confident that the performances we observed may approximate those expected in Canadian primary care. Further, we considered numerous models for CKD previously identified. We restricted to those that could be operationalized in CPCSSN based on predictor availability. In doing so, we ensured the models we included could be readily implemented in Canadian primary care, though this reduced the total amount of predictive information available for ensemble methods. Our study had some limitations. Due to differences in the data and its structure, we could not measure some predictors in the same way as their development cohorts for the included models. However, model updating through intercept re-estimation and coefficient scaling may have accounted for some differences in measurement. Further, while we restricted to include only CKD models based on eGFR measurements, sometimes different eGFR thresholds were used. We used single imputation to account for missing data, as this was sufficient to compare point estimates of performance and did not require prohibitively intensive computational processes. As a result, our estimates may be artificially more precise; however, we did not observe a difference between the ensemble method performance and that of the best component model despite being more likely to do so based on the inflated precision of our estimates. Conclusion Clinical prediction models are only useful when they accurately estimate the risk of disease in their intended setting. We sought to understand how ensemble methods might increase the predictive performance of clinical prediction models, using CKD clinical prediction models as a case study. However, we found no improvement in performance using the ensemble methods we considered, though we observed strong performance among some component models that likely accounted for our null findings. When externally validating existing clinical prediction models, if one model performs better than all others, we encourage use of that best performing component model rather than combining models using ensemble methods. In situations where data are unavailable for external validation, ensemble methods may present a solution to promote strong performance without the ability to determine which component model performs best. However, these situations are likely uncommon and prone to issues with calibration, as model updating is not possible without data for external validation. Future work to combine prediction models may explore Bayesian approaches, where existing clinical prediction models may inform the priors in the development of a Bayesian prediction model 37 . Alternatively, simulation studies may explore the conditions under which model performance may be improved using ensemble methods. For example, perhaps when all models perform similarly (i.e., all models are weak learners) ensemble methods may result in improved performance compared to these weak learners. Nonetheless, this approach should be replicated in other settings and disease contexts to confirm whether ensemble methods can improve the performance of existing clinical prediction models. Declarations Acknowledgements This research partly comprises Jason E. Black’s doctoral work, which is supported by the Achievers in Medical Sciences, Alberta Innovates, and Artificial Intelligence for Public Health scholarships. References Moons KGM, Kengne AP, Grobbee DE, Royston P, Vergouwe Y, Altman DG, et al. Risk prediction models: II. External validation, model updating, and impact assessment. Heart Br Card Soc [Internet]. 2012 May;98(9):691–8. Pajouheshnia R, Smeden M van, Peelen LM, Groenwold RHH. How variation in predictor measurement affects the discriminative ability and transportability of a prediction model. J Clin Epidemiol. 2019 Jan;105:136–41. Zhi-Hua Z. Ensemble Methods: Foundations and Algorithms. Vol. 13, IEEE Intelligent Informatics Bulletin. Chapman and Hall; 2012. Polikar R. Ensemble Learning. In: Zhang C, Ma Y, editors. Ensemble Machine Learning: Methods and Applications [Internet]. Boston, MA: Springer US; 2012. p. 1–34. Hu X, Madden LV, Edwards S, Xu X. Combining Models is More Likely to Give Better Predictions than Single Models. Httpdxdoiorg101094PHYTO-11-14-0315-R [Internet]. 2015 Aug;105(9):1174–82. Bates JM, Granger CWJ. The Combination of Forecasts. OR [Internet]. 1969;20(4):451–68. Krogh A, Sollich P. Statistical mechanics of ensemble learning. Phys Rev E [Internet]. 1997 Jan 1;55(1):811–25. Khan MAB, Hashim MJ, King JK, Govender RD, Mustafa H, Al Kaabi J. Epidemiology of Type 2 Diabetes – Global Burden of Disease and Forecasted Trends. J Epidemiol Glob Health [Internet]. 2020 Mar;10(1):107–11. Harding JL, Pavkov ME, Magliano DJ, Shaw JE, Gregg EW. Global trends in diabetes complications: a review of current evidence. Diabetologia [Internet]. 2019 Jan 1;62(1):3–16. American Diabetes Association Professional Practice Committee. 4. Comprehensive Medical Evaluation and Assessment of Comorbidities: Standards of Medical Care in Diabetes—2022. Diabetes Care [Internet]. 2021 Dec 16;45(Supplement_1):S46–59. Ndjaboue R, Ngueta G, Rochefort-Brihay C, Delorme S, Guay D, Ivers N, et al. Prediction models of diabetes complications: a scoping review. J Epidemiol Community Health [Internet]. 2022 Jun 30; Slieker RC, van der Heijden AAWA, Siddiqui MK, Langendoen-Gort M, Nijpels G, Herings R, et al. Performance of prediction models for nephropathy in people with type 2 diabetes: systematic review and external validation study. BMJ. 2021 Sep 28;374:n2134. Garies S, Birtwhistle R, Drummond N, Queenan J, Williamson T. Data Resource Profile: National electronic medical record data from the Canadian Primary Care Sentinel Surveillance Network (CPCSSN). Int J Epidemiol [Internet]. 2017 Aug 1;46(4):1091–1092f. Queenan JA, Williamson T, Khan S, Drummond N, Garies S, Morkem R, et al. Representativeness of patients and providers in the Canadian Primary Care Sentinel Surveillance Network: a cross-sectional study. CMAJ Open [Internet]. 2016;4(1):E28-32. Nie JX, Wang L, Tracy CS, Moineddin R, Upshur RE. Health care service utilization among the elderly: findings from the Study to Understand the Chronic Condition Experience of the Elderly and the Disabled (SUCCEED project). J Eval Clin Pract [Internet]. 2008 Dec;14(6):1044–9. Morkem R, Salman A, Herman C, Shah R, Wong S, Barber D. CPCSSN Data Quality: An Opportunity for Enhancing Canadian Primary Care Data. 2023 Apr. Black JE, Campbell DJ, Ronksley PE, McBrien KA, Williamson TS. Performance of clinical prediction models for chronic kidney disease among people with diabetes: External validation using the Canadian Primary Care Sentinel Surveillance Network (CPCSSN) [Internet]. Research Square; 2025. Williamson T, Green ME, Birtwhistle R, Khan S, Garies S, Wong ST, et al. Validating the 8 CPCSSN Case Definitions for Chronic Disease Surveillance in a Primary Care Database of Electronic Health Records. Ann Fam Med [Internet]. 2014 Jul;12(4):367–72. Lethebe BC, Williamson T, Garies S, McBrien K, Leduc C, Butalia S, et al. Developing a case definition for type 1 diabetes mellitus in a primary care electronic medical record database: an exploratory study. Can Med Assoc Open Access J [Internet]. 2019 Apr;7(2):E246–51. McFarlane P, Cherney D, Gilbert R, Senior P. Diabetes Canada 2018 Clinical Practice Guidelines for the Prevention and Management of Diabetes in Canada: Chronic Kidney Disease in Diabetes. Can J Diabetes. 2018; Boer IH de, Caramori ML, Chan JCN, Heerspink HJL, Hurst C, Khunti K, et al. KDIGO 2020 Clinical Practice Guideline for Diabetes Management in Chronic Kidney Disease. Kidney Int [Internet]. 2020 Oct 1;98(4):S1–115. Committee; CDACPGE, Cheng AYY. Canadian Diabetes Association 2013 clinical practice guidelines for the prevention and management of diabetes in Canada. Can J Diabetes [Internet]. 2013 Apr;37:S1–3. Thomas RD, Kosowan L, Rabey M, Bell A, Connelly KA, Hawkins NM, et al. Validation of a case definition to identify patients diagnosed with cardiovascular disease in Canadian primary care practices. CJC Open [Internet]. 2023 Apr 22; Faisal N, Kosowan L, Zafari H, Zulkernine F, Lix L, Mahar A, et al. Development and validation of a case definition to estimate the prevalence and incidence of cirrhosis in pan-Canadian primary care databases. Can Liver J [Internet]. 2023 Oct 19;e20230002. Rigobon AV, Birtwhistle R, Khan S, Barber D, Biro S, Morkem R, et al. Adult obesity prevalence in primary care users: An exploration using Canadian Primary Care Sentinel Surveillance Network (CPCSSN) data. Can J Public Health Rev Can Sante Publique. 2015 Apr 30;106(5):e283-289. Spohn O, Morkem R, Singer AG, Barber D. Prevalence and management of dyslipidemia in primary care practices in Canada. Can Fam Physician [Internet]. 2024 Mar 1;70(3):187–96. Pampalon R, Hamel D, Gamache P, Raymond G. A deprivation index for health planning in Canada. Chronic Dis Can. 2009;29(4):178–91. Pavlou M, Qu C, Omar RZ, Seaman SR, Steyerberg EW, White IR, et al. Estimation of required sample size for external validation of risk models for binary outcomes. Stat Methods Med Res. 2021 Oct;30(10):2187–206. Janssen KJM, Vergouwe Y, Donders ART, Harrell FE Jr, Chen Q, Grobbee DE, et al. Dealing with Missing Predictor Values When Applying Clinical Prediction Models. Clin Chem [Internet]. 2009 May 1;55(5):994–1001. Vergouwe Y, Nieboer D, Oostenbrink R, Debray TPA, Murray GD, Kattan MW, et al. A closed testing procedure to select an appropriate method for updating prediction models. Stat Med. 2017 Dec;36(28):4529–39. Jacobs RA, Jordan MI, Nowlan SJ, Hinton GE. Adaptive mixtures of local experts. Neural Comput. 1991;3(1):79–87. Pencina MJ, D’ Agostino Sr RB, D’ Agostino Jr RB, Vasan RS. Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond. Stat Med [Internet]. 2008;27(2):157–72. Pencina MJ, D’Agostino RB, Steyerberg EW. Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Stat Med [Internet]. 2011 Jan;30(1):11–21. Nelson RG, Grams ME, Ballew SH, Sang Y, Azizi F, Chadban SJ, et al. Development of Risk Prediction Equations for Incident Chronic Kidney Disease. JAMA [Internet]. 2019 Dec 3;322(21):2104–14. Collins GS, Groot JA de, Dutton S, Omar O, Shanyinde M, Tajar A, et al. External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC Med Res Methodol [Internet]. 2014;14:40. Wu H, Zhang H, Karwath A, Ibrahim Z, Shi T, Zhang X, et al. Ensemble learning for poor prognosis predictions: A case study on SARS-CoV-2. J Am Med Inform Assoc JAMIA. 2021 Mar 18;28(4):791–800. Arora P, Boyne D, Slater JJ, Gupta A, Brenner DR, Druzdzel MJ. Bayesian Networks for Risk Prediction Using Real-World Data: A Tool for Precision Medicine. Value Health [Internet]. 2019 Apr 1;22(4):439–45. Additional Declarations The authors declare no competing interests. Supplementary Files Appendix.docx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {\"props\":{\"pageProps\":{\"initialData\":{\"identity\":\"rs-6297944\",\"acceptedTermsAndConditions\":true,\"allowDirectSubmit\":true,\"archivedVersions\":[],\"articleType\":\"Research Article\",\"associatedPublications\":[],\"authors\":[{\"id\":433371536,\"identity\":\"81c9bad9-1155-40fe-b0cc-8f11620ce0c0\",\"order_by\":0,\"name\":\"Jason E. Black\",\"email\":\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABAUlEQVRIie3PMWrDMBSA4WdU1OWZrioJ9hUkDKFT1l7DYFCWDIUuBRuiKVkKXjP0MIGAvKS7IRkSAuqSwUeooqlQZDwGon+QhOBDegCh0A0WKbdtxgA0OtrTeDBBSwi3Jxz6mCOUDSJk1ZzOb3DAp5WWJVZTTF+/DYNq6v/Y5zzL1mCQ7aTeoy5Q7GYTBrromWVORwhbhPZxuY8VQaEkZZEiflL/GEdSS95jtUBRmytZ+Mk6nzjCW6pJrKxl7pVtD7lkGXJjR5DF85dukDNDXnLdeImoZ6czfhySpNGiu1RlktYyaruq9BN1XfmfG76BB5Z7AUD6/0YB6XpEKBQK3V+/sMFLzG/+oFsAAAAASUVORK5CYII=\",\"orcid\":\"\",\"institution\":\"University of Calgary\",\"correspondingAuthor\":true,\"prefix\":\"\",\"firstName\":\"Jason\",\"middleName\":\"E.\",\"lastName\":\"Black\",\"suffix\":\"\"},{\"id\":433371690,\"identity\":\"e95a4800-f90e-415f-97ea-a4a6b6db69b8\",\"order_by\":1,\"name\":\"David J.T. Campbell\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"University of Calgary\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"David\",\"middleName\":\"J.T.\",\"lastName\":\"Campbell\",\"suffix\":\"\"},{\"id\":433371691,\"identity\":\"50b6e5a5-08b3-4424-b652-06b730b76027\",\"order_by\":2,\"name\":\"Paul E. Ronksley\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"University of Calgary\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Paul\",\"middleName\":\"E.\",\"lastName\":\"Ronksley\",\"suffix\":\"\"},{\"id\":433371692,\"identity\":\"15df7121-6a4a-425c-808c-364e0f01e38e\",\"order_by\":3,\"name\":\"Kerry A. McBrien\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"University of Calgary\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Kerry\",\"middleName\":\"A.\",\"lastName\":\"McBrien\",\"suffix\":\"\"},{\"id\":433371693,\"identity\":\"b1f8b172-f2ed-4490-b2d4-1a47859d98d7\",\"order_by\":4,\"name\":\"Tyler S. Williamson\",\"email\":\"\",\"orcid\":\"\",\"institution\":\"University of Calgary\",\"correspondingAuthor\":false,\"prefix\":\"\",\"firstName\":\"Tyler\",\"middleName\":\"S.\",\"lastName\":\"Williamson\",\"suffix\":\"\"}],\"badges\":[],\"createdAt\":\"2025-03-24 19:12:36\",\"currentVersionCode\":1,\"declarations\":{\"humanSubjects\":true,\"vertebrateSubjects\":false,\"conflictsOfInterestStatement\":false,\"humanSubjectEthicalGuidelines\":true,\"humanSubjectConsent\":true,\"humanSubjectClinicalTrial\":false,\"humanSubjectCaseReport\":false,\"vertebrateSubjectEthicalGuidelines\":false},\"doi\":\"10.21203/rs.3.rs-6297944/v1\",\"doiUrl\":\"https://doi.org/10.21203/rs.3.rs-6297944/v1\",\"draftVersion\":[],\"editorialEvents\":[],\"editorialNote\":\"\",\"failedWorkflow\":false,\"files\":[{\"id\":79329450,\"identity\":\"547ed251-ae68-4c72-88a9-1c0d4a1d1a16\",\"added_by\":\"auto\",\"created_at\":\"2025-03-27 06:09:25\",\"extension\":\"png\",\"order_by\":1,\"title\":\"Figure 1\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":58241,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003eProcedure used to combine estimated risks from models that have previously been developed and reported\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"1.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-6297944/v1/f37f62136b6514d12d09a732.png\"},{\"id\":79329032,\"identity\":\"ab7a13ea-1541-4541-ad86-b167ebbfec0c\",\"added_by\":\"auto\",\"created_at\":\"2025-03-27 06:08:21\",\"extension\":\"png\",\"order_by\":2,\"title\":\"Figure 2\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":114633,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003eProportion of CPCSSN patients with diabetes with incident CKD over follow-up, overall and by subgroup\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"2.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-6297944/v1/4611a865f1d3bb9f0e2054ab.png\"},{\"id\":79327178,\"identity\":\"64b8110b-7c6f-4fdd-850a-1ad79bd7b036\",\"added_by\":\"auto\",\"created_at\":\"2025-03-27 05:44:04\",\"extension\":\"png\",\"order_by\":3,\"title\":\"Figure 3\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":308753,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003ea: AUROC of ensemble methods for CDK prediction, compared to best component model \\u003cem\\u003e(AUROC represented by solid horizontal line; 95% CI: dashed horizontal lines)\\u003c/em\\u003e\\u003c/p\\u003e\\n\\u003cp\\u003eb: AUPRC of ensemble methods for CDK prediction, compared to best component model \\u003cem\\u003e(AUPRC represented by solid horizontal line; 95% CI: dashed horizontal lines)\\u003c/em\\u003e\\u003c/p\\u003e\\n\\u003cp\\u003ec: NRI of ensemble methods for CDK prediction, compared to best component model\\u003c/p\\u003e\\n\\u003cp\\u003ed: IDI of ensemble methods for CDK prediction, compared to best component model\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"3.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-6297944/v1/ee67b3bfbf430c5b6b414eb9.png\"},{\"id\":79329455,\"identity\":\"6fa7f747-8470-4c42-ae4e-cbc90943005b\",\"added_by\":\"auto\",\"created_at\":\"2025-03-27 06:09:57\",\"extension\":\"png\",\"order_by\":4,\"title\":\"Figure 4\",\"display\":\"\",\"copyAsset\":false,\"role\":\"figure\",\"size\":29655,\"visible\":true,\"origin\":\"\",\"legend\":\"\\u003cp\\u003eCalibration plot for for the ensemble model with the best AUROC which weighted the predictions from all component models by their development cohort size\\u003c/p\\u003e\",\"description\":\"\",\"filename\":\"4.png\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-6297944/v1/343725bbbeec7b9d15b2a49f.png\"},{\"id\":79329473,\"identity\":\"fe7258f1-c66b-43e3-8cc4-d9f9fd462ce3\",\"added_by\":\"auto\",\"created_at\":\"2025-03-27 06:10:17\",\"extension\":\"pdf\",\"order_by\":0,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"manuscript-pdf\",\"size\":999834,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"manuscript.pdf\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-6297944/v1/1301f616-1aa6-4637-962f-9faea7585c91.pdf\"},{\"id\":79329452,\"identity\":\"2b64a109-64f0-4683-80ef-ea40ef1ecccd\",\"added_by\":\"auto\",\"created_at\":\"2025-03-27 06:09:26\",\"extension\":\"docx\",\"order_by\":1,\"title\":\"\",\"display\":\"\",\"copyAsset\":false,\"role\":\"supplement\",\"size\":211535,\"visible\":true,\"origin\":\"\",\"legend\":\"\",\"description\":\"\",\"filename\":\"Appendix.docx\",\"url\":\"https://assets-eu.researchsquare.com/files/rs-6297944/v1/252ee8529713e5481e693d83.docx\"}],\"financialInterests\":\"The authors declare no competing interests.\",\"formattedTitle\":\"\\u003cp\\u003e\\u003cstrong\\u003eCan ensemble methods improve predictive performance of existing models estimating chronic kidney disease among patients with diabetes?\\u003c/strong\\u003e\\u003c/p\\u003e\",\"fulltext\":[{\"header\":\"Introduction\",\"content\":\"\\u003cp\\u003eClinical prediction models are typically used to estimate disease risk in clinical settings and are often developed using a single data source, such as a health administrative database. However, research has consistently shown that models rarely perform as strongly in different settings and/or populations\\u003csup\\u003e\\u003cspan citationid=\\\"CR1\\\" class=\\\"CitationRef\\\"\\u003e1\\u003c/span\\u003e\\u003c/sup\\u003e. Poor model transportability (i.e., model performance in a new or different setting) can arise when the data source used to develop the model is not representative of the population where the model is to be used in practice\\u003csup\\u003e\\u003cspan citationid=\\\"CR2\\\" class=\\\"CitationRef\\\"\\u003e2\\u003c/span\\u003e\\u003c/sup\\u003e. Similarly, exclusion of important groups, such as females, older people, and those with multiple comorbidities, during model development may result in poor model performance when estimating disease risk among these groups.\\u003c/p\\u003e \\u003cp\\u003eCommonly, multiple clinical prediction models for the same outcome have been developed independently by different researchers from various countries and/or health systems. Thus, the transportability and performance of clinical prediction models might theoretically be improved using ensemble methods—a type of machine learning that can combine models. Like meta-analyses that synthesize information from multiple studies, ensemble methods have the potential to leverage predictive relationships from many existing models to improve predictive accuracy and reduce selection bias\\u003csup\\u003e\\u003cspan citationid=\\\"CR3\\\" class=\\\"CitationRef\\\"\\u003e3\\u003c/span\\u003e\\u003c/sup\\u003e.\\u003c/p\\u003e \\u003cp\\u003eEnsemble methods (also referred to as ensembles, ensemble models, or ensemble learning algorithms) operate under the assumption that many weak models operating in concert will outperform any single best-performing model\\u003csup\\u003e\\u003cspan citationid=\\\"CR4\\\" class=\\\"CitationRef\\\"\\u003e4\\u003c/span\\u003e\\u003c/sup\\u003e; this assumption has been confirmed in several studies\\u003csup\\u003e\\u003cspan additionalcitationids=\\\"CR6\\\" citationid=\\\"CR5\\\" class=\\\"CitationRef\\\"\\u003e5\\u003c/span\\u003e–\\u003cspan citationid=\\\"CR7\\\" class=\\\"CitationRef\\\"\\u003e7\\u003c/span\\u003e\\u003c/sup\\u003e. When developing an ensemble model traditionally, multiple models are fit to the same dataset while varying the type of model (e.g., logistic regression, decision tree, naïve Bayes), predictors considered by the model, and/or observations used to fit the model. When predicting the outcome for a new person whose outcome is unknown, each component model is evaluated and predictions are combined by some method, such as majority voting where the most frequently predicted outcome is assigned.\\u003c/p\\u003e \\u003cp\\u003eEnsemble methods have been successfully applied to develop novel clinical prediction models using a single data source [8]. However, it remains unknown whether ensemble methods can improve predictive performance by combining multiple existing clinical prediction models previously developed across a variety of data sources.\\u003c/p\\u003e \\u003cp\\u003e \\u003cem\\u003eCase study: Clinical prediction models estimating the risk of developing chronic kidney disease among people with diabetes\\u003c/em\\u003e \\u003c/p\\u003e \\u003cp\\u003eDiabetes is a common chronic condition associated with considerable morbidity worldwide\\u003csup\\u003e\\u003cspan citationid=\\\"CR8\\\" class=\\\"CitationRef\\\"\\u003e8\\u003c/span\\u003e\\u003c/sup\\u003e. As diabetes continues to increase in prevalence, the burden of complications resulting from this condition are also on the rise\\u003csup\\u003e\\u003cspan citationid=\\\"CR9\\\" class=\\\"CitationRef\\\"\\u003e9\\u003c/span\\u003e\\u003c/sup\\u003e. In particular, cardiovascular disease, as well as microvascular complications (including nephropathy leading to chronic kidney disease [CKD]) are known to result from all forms of diabetes. However, diabetes-self management resulting in optimized blood glucose levels, blood pressure, and cholesterol levels can reduce the risk of vascular complications. As most people with diabetes are managed by primary care practitioners, opportunities to optimize care and prevent diabetes complications exist in primary care\\u003csup\\u003e\\u003cspan citationid=\\\"CR10\\\" class=\\\"CitationRef\\\"\\u003e10\\u003c/span\\u003e\\u003c/sup\\u003e. Indeed, many clinical prediction models have been developed to estimate risk of developing CKD among people with diabetes to support preventative care\\u003csup\\u003e\\u003cspan citationid=\\\"CR11\\\" class=\\\"CitationRef\\\"\\u003e11\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR12\\\" class=\\\"CitationRef\\\"\\u003e12\\u003c/span\\u003e\\u003c/sup\\u003e.\\u003c/p\\u003e\\n\\u003ch3\\u003eObjectives\\u003c/h3\\u003e\\n\\u003cp\\u003eWe aimed to determine whether ensemble methods can improve predictive performance by combining existing clinical prediction models compared to the individual component models, using the risk of CKD development in people with diabetes as a case study. We assessed for improved performance overall and among subgroups defined by sex/gender, age group, number of comorbidities, and level of social and material deprivation.\\u003c/p\\u003e \"},{\"header\":\"Methods\",\"content\":\"\\u003ch2\\u003eStudy setting and data source\\u003c/h2\\u003e\\u003cp\\u003eWe used electronic medical record (EMR) data from the Canadian Primary Care Sentinel Surveillance Network (CPCSSN), which was established in 2008 as a Canada-wide repository of primary care records for more than 2\\u0026nbsp;million patients (excluding Saskatchewan)\\u003csup\\u003e\\u003cspan citationid=\\\"CR13\\\" class=\\\"CitationRef\\\"\\u003e13\\u003c/span\\u003e\\u003c/sup\\u003e. Patient information (e.g., diagnoses, procedures, laboratory tests, medication prescriptions, and referrals) as recorded by the primary care practitioner is de-identified and contributed to 14 regional networks that comprise CPCSSN. Prior work has found that patients in CPCSSN tend to be more likely to be female and older than the general population in Canada\\u003csup\\u003e\\u003cspan citationid=\\\"CR14\\\" class=\\\"CitationRef\\\"\\u003e14\\u003c/span\\u003e\\u003c/sup\\u003e, though this is typical of primary care patients\\u003csup\\u003e\\u003cspan citationid=\\\"CR15\\\" class=\\\"CitationRef\\\"\\u003e15\\u003c/span\\u003e\\u003c/sup\\u003e. CPCSSN facilitates research use by cleaning and standardising patient records (e.g., assigning diagnostic codes to free-text diagnoses)\\u003csup\\u003e\\u003cspan citationid=\\\"CR16\\\" class=\\\"CitationRef\\\"\\u003e16\\u003c/span\\u003e\\u003c/sup\\u003e. For this study, we used CPCSSN data from January 2014 to December 2019.\\u003c/p\\u003e\\u003ch3\\u003eModels\\u003c/h3\\u003e\\u003cp\\u003eTwo recent systematic reviews identified 47 clinical prediction models for incident CKD among people with diabetes\\u003csup\\u003e\\u003cspan citationid=\\\"CR11\\\" class=\\\"CitationRef\\\"\\u003e11\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR12\\\" class=\\\"CitationRef\\\"\\u003e12\\u003c/span\\u003e\\u003c/sup\\u003e. We previously evaluated the performance of 13 models for incident CKD with predictors available among Canadian primary care patients and found mixed performance when predicting incident CKD in this setting\\u003csup\\u003e\\u003cspan citationid=\\\"CR17\\\" class=\\\"CitationRef\\\"\\u003e17\\u003c/span\\u003e\\u003c/sup\\u003e. We considered these clinical prediction models to develop an ensemble model for incident CKD.\\u003c/p\\u003e\\u003ch3\\u003eParticipants\\u003c/h3\\u003e\\u003cp\\u003eWe identified adult (18+) patients with any form of diabetes (excluding gestational) in CPCSSN using validated case definitions\\u003csup\\u003e\\u003cspan citationid=\\\"CR18\\\" class=\\\"CitationRef\\\"\\u003e18\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR19\\\" class=\\\"CitationRef\\\"\\u003e19\\u003c/span\\u003e\\u003c/sup\\u003e. We identified baseline visits for each patient between January to December 2014, then included patients without an existing diagnosis of CKD (see \\u003cem\\u003e§Measures\\u003c/em\\u003e) prior to baseline, with at least 3 primary care visits in the 2 years prior to baseline, and with at least one visit within 5 years after baseline. For patients who were identified as having type 1 diabetes\\u003csup\\u003e\\u003cspan citationid=\\\"CR19\\\" class=\\\"CitationRef\\\"\\u003e19\\u003c/span\\u003e\\u003c/sup\\u003e, we excluded those who were diagnosed within 5 years before baseline as these patients were not yet recommended for CKD screening per clinical practice guidelines\\u003csup\\u003e\\u003cspan citationid=\\\"CR20\\\" class=\\\"CitationRef\\\"\\u003e20\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR21\\\" class=\\\"CitationRef\\\"\\u003e21\\u003c/span\\u003e\\u003c/sup\\u003e. We excluded patients from Quebec and Manitoba due to data quality issues in the recording of kidney function [estimated glomerular fitration rate (eGFR)] laboratory tests. We noted those with incident CKD diagnoses within 5 years from baseline and censored patients at their last visit within 5 years from baseline. Per clinical guidelines, we identified incident CKD as two or more serum creatinine laboratory values which correspond to an eGFR less than 60 mL/min/1.73 m\\u003csup\\u003e\\u003cspan citationid=\\\"CR2\\\" class=\\\"CitationRef\\\"\\u003e2\\u003c/span\\u003e\\u003c/sup\\u003e that were separated by at least 90 days but no more than 1 year apart\\u003csup\\u003e\\u003cspan citationid=\\\"CR22\\\" class=\\\"CitationRef\\\"\\u003e22\\u003c/span\\u003e\\u003c/sup\\u003e.\\u003c/p\\u003e\\u003ch3\\u003eMeasures\\u003c/h3\\u003e\\u003cp\\u003eWe identified predictors of CKD in CPCSSN using (in the following order): directly reported values (e.g., laboratory test results for hemoglobin A1c [HbA1c]); case definitions previously validated in CPCSSN (e.g., hypertension); case definitions previously applied in CPCSSN but not validated (e.g., macrovascular complications); or case definitions developed ad-hoc in consultation with clinical colleagues. See Appendix A for details on how we identified each predictor. We considered subgroups defined by sex/gender (female/woman and male/man), age group (18 to 39, 40 to 64, and 65 and older), number of comorbidities (none, one, two, or three or more), and social and material deprivation quintile. We identified sex/gender as recorded in the EMR by the practitioner—though we cannot be certain which was recorded. We calculated age at baseline visit based on patient birth dates. We identified the following comorbidities using validated case definitions: cardiovascular disease; liver cirrhosis; chronic obstructive pulmonary disease; dementia; depression; epilepsy; hypertension; osteoarthritis; Parkinson’s disease; obesity; and dyslipidemia\\u003csup\\u003e\\u003cspan citationid=\\\"CR18\\\" class=\\\"CitationRef\\\"\\u003e18\\u003c/span\\u003e,\\u003cspan additionalcitationids=\\\"CR24 CR25\\\" citationid=\\\"CR23\\\" class=\\\"CitationRef\\\"\\u003e23\\u003c/span\\u003e–\\u003cspan citationid=\\\"CR26\\\" class=\\\"CitationRef\\\"\\u003e26\\u003c/span\\u003e\\u003c/sup\\u003e. We estimated patient deprivation quintiles using the neighbourhood-level Pampalon social and material deprivation indices mapped to patient postal codes\\u003csup\\u003e\\u003cspan citationid=\\\"CR27\\\" class=\\\"CitationRef\\\"\\u003e27\\u003c/span\\u003e\\u003c/sup\\u003e.\\u003c/p\\u003e\\u003ch2\\u003eSample size\\u003c/h2\\u003e\\u003cp\\u003eWe estimated the required sample size for a reasonable range of anticipated areas under the receiver operating characteristic curve (AUROC) and incidences of CKD based on those of existing CKD clinical prediction models. We required from 689 to 21,925 patients to estimate 95% confidence intervals for the AUROC with a width of roughly 0.04 and the calibration intercept and slope both with widths of roughly 0.2\\u003csup\\u003e28\\u003c/sup\\u003e.\\u003c/p\\u003e\\u003ch3\\u003eMissing data\\u003c/h3\\u003e\\u003cp\\u003eWe accounted for missing data using single imputation. This approach enabled comparisons between point estimates while ensuring computation feasibility. However, precision of the confidence intervals for these estimates may be artificially inflated.\\u003c/p\\u003e\\u003ch2\\u003eStatistical analysis\\u003c/h2\\u003e\\u003cp\\u003eWe described the baseline predictors in our cohort using means (standard deviations), medians (first and third quartiles), and frequencies (percentages), as appropriate. We determined the incidence proportion (%) of CKD diagnosis over 5 years of follow-up.\\u003c/p\\u003e\\u003cp\\u003eWe combined risk estimates from models that were previously developed using the procedure presented in Fig.\\u0026nbsp;\\u003cspan refid=\\\"Fig1\\\" class=\\\"InternalRef\\\"\\u003e1\\u003c/span\\u003e. We restricted the ensemble methods we considered to those that did not estimate new model parameters (e.g., stacking) based on a dataset (i.e., CPCSSN diabetes patients). Instead, we relied on data-independent approaches (e.g., bagging) to combine models then validated the resulting ensemble model in CPCSSN patients with diabetes.\\u003c/p\\u003e\\u003cp\\u003eFirst, we considered the availability of predictors within CPCSSN. We excluded models without information on ≥2 predictors in CPCSSN to include many models while minimizing the loss of performance associated with missing predictors\\u003csup\\u003e\\u003cspan citationid=\\\"CR29\\\" class=\\\"CitationRef\\\"\\u003e29\\u003c/span\\u003e\\u003c/sup\\u003e. Next, we updated the models by re-estimating model intercepts and scaling model coefficients (i.e., reducing or increasing the magnitude of all coefficients by the same factor) using the CPCSSN cohort\\u003csup\\u003e\\u003cspan citationid=\\\"CR30\\\" class=\\\"CitationRef\\\"\\u003e30\\u003c/span\\u003e\\u003c/sup\\u003e. Model updating helps correct for differences in CKD incidence and prediction horizons (e.g., estimating 3- vs. 5-year CKD risks) between CPCSSN and the development cohort. For example, we assume that 3- and 5-year CKD risks should rank patients similarly, despite assigning different absolute values of risk. Updating corrects for differences in the mean and distribution of risks while retaining their ranking. For each patient, we estimated several 5-year CKD risks by using all updated models. We combined each patient’s risk estimates through an averaging and/or selection process, with or without weighting according to various metrics (Table\\u0026nbsp;\\u003cspan refid=\\\"Tab1\\\" class=\\\"InternalRef\\\"\\u003e1\\u003c/span\\u003e). We considered two types of means when averaging risk estimates: the \\u003cem\\u003earithmetic mean\\u003c/em\\u003e and the \\u003cem\\u003egeometric mean\\u003c/em\\u003e (i.e., the mean computed on the logarithmic scale converted back to the original scale). A \\u003cem\\u003emixture-of-experts\\u003c/em\\u003e process\\u003csup\\u003e\\u003cspan citationid=\\\"CR31\\\" class=\\\"CitationRef\\\"\\u003e31\\u003c/span\\u003e\\u003c/sup\\u003e selected the most appropriate models and corresponding risk estimates for each patient according to similarity or age-similarity and computed the arithmetic mean to determine their estimated risk. Similarly, the \\u003cem\\u003emedian-of-experts\\u003c/em\\u003e process selected the median estimated risk among the most appropriate models for each patient according to similarity or age-similarity to determine their estimated risk. We tested the \\u003cem\\u003emixture-\\u003c/em\\u003e and \\u003cem\\u003emedian-of-experts\\u003c/em\\u003e approaches using the 1, 3 and 5 most similar models. We weighted estimates or selected a model according to factors that may be associated with improved model performance (i.e., development cohort size; development AUROCs; similarity to the development cohort according to age, sex/gender, recorded HbA1c, and BMI; and similarity to the development cohort in age). We measured similarity based on the Euclidean distance after standardizing age, HbA1c, and BMI.\\u003c/p\\u003e\\u003cdiv class=\\\"gridtable\\\"\\u003e\\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c1\\\" colnum=\\\"1\\\"\\u003e\\u003c/div\\u003e\\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c2\\\" colnum=\\\"2\\\"\\u003e\\u003c/div\\u003e\\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c3\\\" colnum=\\\"3\\\"\\u003e\\u003c/div\\u003e\\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c4\\\" colnum=\\\"4\\\"\\u003e\\u003c/div\\u003e\\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c5\\\" colnum=\\\"5\\\"\\u003e\\u003c/div\\u003e\\u003cdiv align=\\\"left\\\" class=\\\"colspec\\\" colname=\\\"c6\\\" colnum=\\\"6\\\"\\u003e\\u003c/div\\u003e\\u003ctable float=\\\"Yes\\\" id=\\\"Tab1\\\" border=\\\"1\\\"\\u003e\\u003ccaption language=\\\"En\\\"\\u003e \\u003cdiv class=\\\"CaptionNumber\\\"\\u003eTable 1\\u003c/div\\u003e \\u003cdiv class=\\\"CaptionContent\\\"\\u003e \\u003cp\\u003eAveraging or selection processes and corresponding weighting techniques, where applicable\\u003c/p\\u003e \\u003c/div\\u003e \\u003c/caption\\u003e\\u003ccolgroup cols=\\\"6\\\"\\u003e\\u003c/colgroup\\u003e\\u003cthead\\u003e\\u003ctr\\u003e\\u003cth align=\\\"left\\\" colname=\\\"c1\\\"\\u003e\\u0026nbsp;\\u003c/th\\u003e\\u003cth align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eNone\\u003c/p\\u003e \\u003c/th\\u003e\\u003cth align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eDevelopment cohort size\\u003c/p\\u003e \\u003c/th\\u003e\\u003cth align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003eDevelopment AUROC\\u003c/p\\u003e \\u003c/th\\u003e\\u003cth align=\\\"left\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003eSimilarity\\u003c/p\\u003e \\u003c/th\\u003e\\u003cth align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003eAge-similarity\\u003c/p\\u003e \\u003c/th\\u003e\\u003c/tr\\u003e\\u003c/thead\\u003e\\u003ctbody\\u003e\\u003ctr\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eArithmetic mean\\u003c/p\\u003e \\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eX\\u003c/p\\u003e \\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eX\\u003c/p\\u003e \\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003eX\\u003c/p\\u003e \\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003eX\\u003c/p\\u003e \\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003eX\\u003c/p\\u003e \\u003c/td\\u003e\\u003c/tr\\u003e\\u003ctr\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eGeometric mean\\u003c/p\\u003e \\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e \\u003cp\\u003eX\\u003c/p\\u003e \\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e \\u003cp\\u003eX\\u003c/p\\u003e \\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e \\u003cp\\u003eX\\u003c/p\\u003e \\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003eX\\u003c/p\\u003e \\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003eX\\u003c/p\\u003e \\u003c/td\\u003e\\u003c/tr\\u003e\\u003ctr\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eMixture-of-experts\\u003c/p\\u003e \\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e\\u0026nbsp;\\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e\\u0026nbsp;\\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e\\u0026nbsp;\\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003eX\\u003c/p\\u003e \\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003eX\\u003c/p\\u003e \\u003c/td\\u003e\\u003c/tr\\u003e\\u003ctr\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c1\\\"\\u003e \\u003cp\\u003eMedian-of-experts\\u003c/p\\u003e \\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c2\\\"\\u003e\\u0026nbsp;\\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c3\\\"\\u003e\\u0026nbsp;\\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c4\\\"\\u003e\\u0026nbsp;\\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c5\\\"\\u003e \\u003cp\\u003eX\\u003c/p\\u003e \\u003c/td\\u003e\\u003ctd align=\\\"left\\\" colname=\\\"c6\\\"\\u003e \\u003cp\\u003eX\\u003c/p\\u003e \\u003c/td\\u003e\\u003c/tr\\u003e\\u003c/tbody\\u003e\\u003c/table\\u003e\\u003c/div\\u003e\\u003cp\\u003eWe evaluated the performance of the resulting ensemble models in terms of their discrimination (AUROC), precision and recall (area under the precision-recall curve), and calibration (calibration plots). We compared the ensemble results to the best performing component model (in terms of AUROC) by examining differences in discrimination, precision, recall, and calibration and estimating the category-free net reclassification index (NRI\\u003csub\\u003e\\u0026gt; 0\\u003c/sub\\u003e) and integrated discrimination improvement (IDI)\\u003csup\\u003e\\u003cspan citationid=\\\"CR32\\\" class=\\\"CitationRef\\\"\\u003e32\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR33\\\" class=\\\"CitationRef\\\"\\u003e33\\u003c/span\\u003e\\u003c/sup\\u003e. Comparing a new model to a reference model, the NRI\\u003csub\\u003e\\u0026gt; 0\\u003c/sub\\u003e measures whether individual estimated risks based on the new model increase for those who experience the outcome or decrease for those who do not experience the outcome. Similarly, IDI considers whether the average estimated risk based on the new model increases among those who experience the outcome and decreases among those who do not experience the outcome. From these comparisons, we determined 1) which averaging or selection process resulted in the best performing ensemble model and 2) whether ensemble models outperform their component models.\\u003c/p\\u003e\\u003ch2\\u003eSubgroup analysis\\u003c/h2\\u003e\\u003cp\\u003eWe evaluated the performance of each ensemble model among groups defined by sex/gender, age (18 to 39, 40 to 64, and 65 and older), number of comorbidities (none, one, two, or three or more), and material and social deprivation quintile based on the Pampalon index\\u003csup\\u003e\\u003cspan citationid=\\\"CR27\\\" class=\\\"CitationRef\\\"\\u003e27\\u003c/span\\u003e\\u003c/sup\\u003e.\\u003c/p\\u003e\\u003ch2\\u003eEthics approval\\u003c/h2\\u003e\\u003cp\\u003e Our study was approved by the University of Calgary Conjoint Health Research Ethics Board under study ID REB21-1741.\\u003c/p\\u003e\"},{\"header\":\"Results\",\"content\":\"\\u003cp\\u003eWe identified 37,604 patients with diabetes in CPCSSN to assess the performance of ensemble methods using several existing clinical prediction models for incident CKD. Patient characteristics are summarized in Table \\u003cspan class=\\\"InternalRef\\\"\\u003e2\\u003c/span\\u003e and Appendix B. CPCSSN patients with diabetes were on average 63.4 (SD: 12.2) years old, were more frequently male/men (53.0%) than female/women (47.0%), frequently had HbA1c values below 7.0% (median: 6.7%; first\\u0026mdash;third quartiles: 6.3\\u0026mdash;7.4), and had a mean body mass index (BMI) of 32.3 (SD: 7.4) kg/m\\u003csup\\u003e2\\u003c/sup\\u003e.\\u003c/p\\u003e\\n\\u003cdiv class=\\\"gridtable\\\"\\u003e\\u0026nbsp;\\u003ctable id=\\\"Tab2\\\" border=\\\"1\\\"\\u003e\\n \\u003ccaption language=\\\"En\\\"\\u003e\\n \\u003cdiv class=\\\"CaptionNumber\\\"\\u003eTable 2\\u003c/div\\u003e\\n \\u003cdiv class=\\\"CaptionContent\\\"\\u003e\\n \\u003cp\\u003eCharacteristics of CPCSSN diabetes cohort for validation of ensemble models\\u003c/p\\u003e\\n \\u003c/div\\u003e\\n \\u003c/caption\\u003e\\n \\u003ccolgroup cols=\\\"2\\\"\\u003e\\u003c/colgroup\\u003e\\n \\u003cthead\\u003e\\n \\u003ctr\\u003e\\n \\u003cth align=\\\"left\\\"\\u003e\\u0026nbsp;\\u003c/th\\u003e\\n \\u003cth align=\\\"left\\\"\\u003e\\n \\u003cp\\u003eN\\u0026thinsp;=\\u0026thinsp;37,604\\u003c/p\\u003e\\n \\u003c/th\\u003e\\n \\u003c/tr\\u003e\\n \\u003c/thead\\u003e\\n \\u003ctbody\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003eAge group, n (%)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\u0026nbsp;\\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e18 to 39\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e1,073 (2.9)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e40 to 64\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e18,370 (48.9)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e65 and older\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e18,161 (48.3)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003eSex/gender, n (%)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\u0026nbsp;\\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003eMale/men\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e23,174 (53.0)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003eFemale/women\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e20,516 (47.0)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003eSocial and material deprivation, n (%)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\u0026nbsp;\\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e1st quintile (least deprived)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e6,564 (17.5)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e2nd quintile\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e7,521 (20.0)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e3rd quintile\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e7,032 (18.7)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e4th quintile\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e7,123 (18.9)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e5th quintile (most deprived)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e7,039 (18.7)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e\\u003cem\\u003eMissing\\u003c/em\\u003e\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e2,325 (6.2)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003eUrban residence, n (%)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e29,088 (77.4)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003eHbA1c (%), median (first\\u0026ndash;third quartiles)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e6.7 (6.3\\u0026mdash;7.4)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003eDiabetes duration (years), median (first\\u0026ndash;third quartiles)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e2.8 (1.3\\u0026mdash;5.1)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003eNon-insulin antihyperglycemic drugs, n (%)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e21,460 (57.1)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003eInsulin, n (%)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e4,128 (11.0)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003eBMI (kg/m\\u003csup\\u003e2\\u003c/sup\\u003e), mean\\u0026thinsp;\\u0026plusmn;\\u0026thinsp;SD\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e32.3\\u0026thinsp;\\u0026plusmn;\\u0026thinsp;7.4\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003eBaseline eGFR (mL/min/1.73 m\\u003csup\\u003e\\u003cspan class=\\\"CitationRef\\\"\\u003e2\\u003c/span\\u003e\\u003c/sup\\u003e), mean\\u0026thinsp;\\u0026plusmn;\\u0026thinsp;SD\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e84\\u0026thinsp;\\u0026plusmn;\\u0026thinsp;18\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003eUrine Albumin-to-Creatinine Ratio (mg/mmol), median (first\\u0026ndash;third quartiles/IQR)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e1.3 (0.6\\u0026mdash;2.9)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003eComorbidities, n (%)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\u0026nbsp;\\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003eNone\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e3,692 (9.8)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003eOne\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e9,062 (24.1)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003eTwo\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e10,826 (28.8)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003eThree or more\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003ctd align=\\\"left\\\"\\u003e\\n \\u003cp\\u003e14,024 (37.3)\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003ctr\\u003e\\n \\u003ctd align=\\\"left\\\" colspan=\\\"2\\\"\\u003e\\n \\u003cp\\u003eRAS-antagonist: renin\\u0026ndash;angiotensin system antagonist.\\u003c/p\\u003e\\n \\u003c/td\\u003e\\n \\u003c/tr\\u003e\\n \\u003c/tbody\\u003e\\n \\u003c/table\\u003e\\n\\u003c/div\\u003e\\n\\u003cp\\u003eWe identified incident CKD among 14.6% of CPCSSN patients with diabetes over an average follow-up of 4.8 years, though we identified incident CKD more frequently among older patients and those with multiple comorbidities (Fig. \\u003cspan class=\\\"InternalRef\\\"\\u003e2\\u003c/span\\u003e).\\u003c/p\\u003e\\n\\u003cp\\u003eWe previously found that the performance of existing clinical prediction models for CKD among CPCSSN patients with diabetes was mixed, with AUROCs ranging from 0.492 to 0.826\\u003csup\\u003e17\\u003c/sup\\u003e. The best performing model according to AUROC and AUPRC was developed by Nelson et al. \\u003csup\\u003e\\u003cspan class=\\\"CitationRef\\\"\\u003e34\\u003c/span\\u003e\\u003c/sup\\u003e (AUROC: 0.826 [95% CI: 0.820 to 0.832]; AUPRC: 0.467 [95% CI: 0.462 to 0.472]).\\u003c/p\\u003e\\n\\u003cp\\u003eEnsemble methods displayed strong performance predicting CKD incidence in CPCSSN according to AUROC and AUPRC (Figs. \\u003cspan class=\\\"InternalRef\\\"\\u003e3\\u003c/span\\u003ea and \\u003cspan class=\\\"InternalRef\\\"\\u003e3\\u003c/span\\u003eb); however, no ensemble method had better performance than the best performing component model. Among ensemble methods, the averaging or selection process with the best performance weighted the predictions from all component models by their development cohort size (AUROC: 0.827 [95% CI: 0.821 to 0.833]). Based on the NRI\\u003csub\\u003e\\u0026gt;\\u0026thinsp;0\\u003c/sub\\u003e, estimated risks based on the ensemble methods were often worse (i.e., for those that developed CKD, risk estimates were further from 1, whereas for those that did not develop CKD, risk estimates were further from 0) than the best performing component model (Fig. \\u003cspan class=\\\"InternalRef\\\"\\u003e3\\u003c/span\\u003ec), except for the geometric mean for all component models without weighting (NRI\\u003csub\\u003e\\u0026gt;\\u0026thinsp;0\\u003c/sub\\u003e: 0.062 [95% CI: 0.052 to 0.071]). Similarly, the IDI for all ensemble methods showed that risk estimates were less accurate compared to the best component model: the average estimated risk for all patients that developed CKD was further from 1 and the average estimated risk for all patients that did not develop CKD was further from 0 (Fig. \\u003cspan class=\\\"InternalRef\\\"\\u003e3\\u003c/span\\u003ed). All ensemble models demonstrated some miscalibration: the calibration plot for the ensemble model with the best AUROC which weighted the predictions from all component models by their development cohort size showed overestimation of lower risks (Fig. \\u003cspan class=\\\"InternalRef\\\"\\u003e4\\u003c/span\\u003e). All calibration plots are available in Appendix C.\\u003c/p\\u003e\\n\\u003cp\\u003eWe found that \\u003cem\\u003emixture-\\u003c/em\\u003e and \\u003cem\\u003emedian-of-experts\\u003c/em\\u003e ensemble methods resulted in poorer performance when compared to weighting techniques that considered all component models, though performance improved as more component models were included. In most cases, the ensemble methods that relied on the geometric mean performed slightly better than the corresponding techniques that relied on the arithmetic mean.\\u003c/p\\u003e\\n\\u003cp\\u003ePerformance was not improved across subgroups when using ensemble methods. Instead, performance between subgroups followed the same patterns as when using the best performing component model (Appendices D to G).\\u003c/p\\u003e\"},{\"header\":\"Discussion\",\"content\":\"\\u003cp\\u003eWe assessed the performance of several models using ensemble methods to combine 13 previously developed clinical prediction models estimating the risk of incident CKD among patients with diabetes. While the performance of some ensemble methods equalled that of the best performing component model, no ensemble method was able to surpass this level of performance. The best ensemble method performance according to AUROC and AUPRC was achieved by computing the geometric mean of all component model predictions weighted based on development cohort size (AUROC: 0.827 [95% CI: 0.821 to 0.833]; AUPRC: 0.471 [95% CI: 0.466 to 0.476]). However, this ensemble method had similar performance to the best performing component model based on AUROC (i.e., the Nelson et al. model; AUROC: 0.826 [95% CI: 0.82 to 0.826]; AUPRC: 0.467 [95% CI: 0.462 to 0.472]). Overlapping confidence intervals suggested these performance measures were not statistically different. Indeed, due to the considerable size of the development cohort used by Nelson et al. compared to all others (Nelson et al. combined several datasets for a development cohort totalling 781,627 people, compared to 43,362 people among all other development cohorts combined), the ensemble predictions based on development cohort size weighting were almost the same as the Nelson et al. model predictions. According to NRI\\u003csub\\u003e\\u0026gt;\\u0026thinsp;0\\u003c/sub\\u003e, improved predictions compared to the best performing component model were only achieved by the geometric mean without weighting; however, this improvement was small (0.062 [95% CI: 0.052 to 0.071]). No ensemble methods displayed improvement compared to the best performing component model considering the IDI. Our results suggest that this application of ensemble methods may not provide improved predictive performance compared to its component models.\\u003c/p\\u003e \\u003cp\\u003eSeveral reasons may explain why ensemble methods failed to improve upon existing clinical prediction models for CKD in our analysis. Foremost, ensemble methods were designed to combine many models with poor performance (a.k.a., weak learners) to create a model that outperforms all its component models\\u003csup\\u003e\\u003cspan citationid=\\\"CR4\\\" class=\\\"CitationRef\\\"\\u003e4\\u003c/span\\u003e\\u003c/sup\\u003e. Applying existing clinical prediction models for CKD in CPCSSN diabetes patients, we observed mixed performances\\u0026mdash;many models performed poorly but some performed quite well (AUROC\\u0026thinsp;\\u0026gt;\\u0026thinsp;0.80; a.k.a., strong learners). As such, the addition of weak learners to strong learners may not have resulted in improvements upon the performance of the strong learners. Further, while 13 models were included in our ensemble methods, some ensemble methods employ more than 500 to 1,000 component models\\u003csup\\u003e\\u003cspan citationid=\\\"CR3\\\" class=\\\"CitationRef\\\"\\u003e3\\u003c/span\\u003e,\\u003cspan citationid=\\\"CR4\\\" class=\\\"CitationRef\\\"\\u003e4\\u003c/span\\u003e\\u003c/sup\\u003e. It is possible that more models are required to observe increased performance using ensemble methods; however, this limits the feasibility of this technique as each model must be operationalized within the validation dataset.\\u003c/p\\u003e \\u003cp\\u003eEnsemble methods have not frequently been utilized to combine existing clinical prediction models into a single ensemble model\\u0026mdash;instead, research has focused on developing novel clinical prediction models using ensemble methods, despite calls to externally validate existing models\\u003csup\\u003e\\u003cspan citationid=\\\"CR35\\\" class=\\\"CitationRef\\\"\\u003e35\\u003c/span\\u003e\\u003c/sup\\u003e. However, one study that used ensemble methods with existing models was performed by Wu et al.\\u003csup\\u003e\\u003cspan citationid=\\\"CR36\\\" class=\\\"CitationRef\\\"\\u003e36\\u003c/span\\u003e\\u003c/sup\\u003e, where they combined 7 previously developed clinical prediction models to identify individuals at high-risk of experiencing poor outcomes related to coronavirus disease (COVID-19). Wu et al. used similar ensemble methods, such as weighted averaging and mixture of experts, to combine predictions from each component model and assess their performance across 4 validation datasets. They found that ensemble methods could perform well; however, overlapping confidence intervals suggested they did not outperform the best performing component model in each validation dataset. These results are consistent with our findings evaluating ensemble methods to combine CKD clinical prediction models\\u0026mdash;though Wu et al. found consistent improvements in predictions from an ensemble method based on the NRI, whereas we only observed such an improvement for one ensemble method.\\u003c/p\\u003e \\u003cdiv id=\\\"Sec15\\\" class=\\\"Section2\\\"\\u003e \\u003ch2\\u003eStrengths and limitations\\u003c/h2\\u003e \\u003cp\\u003eOur study used a large cohort of more than 30,000 patients to test the performance of existing prediction models for CKD and determined whether ensemble methods might improve upon these performances. We leveraged real patient data to measure model performances, thus we can be confident that the performances we observed may approximate those expected in Canadian primary care. Further, we considered numerous models for CKD previously identified. We restricted to those that could be operationalized in CPCSSN based on predictor availability. In doing so, we ensured the models we included could be readily implemented in Canadian primary care, though this reduced the total amount of predictive information available for ensemble methods.\\u003c/p\\u003e \\u003cp\\u003eOur study had some limitations. Due to differences in the data and its structure, we could not measure some predictors in the same way as their development cohorts for the included models. However, model updating through intercept re-estimation and coefficient scaling may have accounted for some differences in measurement. Further, while we restricted to include only CKD models based on eGFR measurements, sometimes different eGFR thresholds were used. We used single imputation to account for missing data, as this was sufficient to compare point estimates of performance and did not require prohibitively intensive computational processes. As a result, our estimates may be artificially more precise; however, we did not observe a difference between the ensemble method performance and that of the best component model despite being more likely to do so based on the inflated precision of our estimates.\\u003c/p\\u003e \\u003c/div\\u003e\"},{\"header\":\"Conclusion\",\"content\":\"\\u003cp\\u003eClinical prediction models are only useful when they accurately estimate the risk of disease in their intended setting. We sought to understand how ensemble methods might increase the predictive performance of clinical prediction models, using CKD clinical prediction models as a case study. However, we found no improvement in performance using the ensemble methods we considered, though we observed strong performance among some component models that likely accounted for our null findings. When externally validating existing clinical prediction models, if one model performs better than all others, we encourage use of that best performing component model rather than combining models using ensemble methods. In situations where data are unavailable for external validation, ensemble methods may present a solution to promote strong performance without the ability to determine which component model performs best. However, these situations are likely uncommon and prone to issues with calibration, as model updating is not possible without data for external validation.\\u003c/p\\u003e \\u003cp\\u003eFuture work to combine prediction models may explore Bayesian approaches, where existing clinical prediction models may inform the priors in the development of a Bayesian prediction model\\u003csup\\u003e\\u003cspan citationid=\\\"CR37\\\" class=\\\"CitationRef\\\"\\u003e37\\u003c/span\\u003e\\u003c/sup\\u003e. Alternatively, simulation studies may explore the conditions under which model performance may be improved using ensemble methods. For example, perhaps when all models perform similarly (i.e., all models are weak learners) ensemble methods may result in improved performance compared to these weak learners. Nonetheless, this approach should be replicated in other settings and disease contexts to confirm whether ensemble methods can improve the performance of existing clinical prediction models.\\u003c/p\\u003e\"},{\"header\":\"Declarations\",\"content\":\"\\u003ch2\\u003eAcknowledgements\\u003c/h2\\u003e \\u003cp\\u003eThis research partly comprises Jason E. Black\\u0026rsquo;s doctoral work, which is supported by the Achievers in Medical Sciences, Alberta Innovates, and Artificial Intelligence for Public Health scholarships.\\u003c/p\\u003e\"},{\"header\":\"References\",\"content\":\"\\u003col\\u003e\\n \\u003cli\\u003eMoons KGM, Kengne AP, Grobbee DE, Royston P, Vergouwe Y, Altman DG, et al. Risk prediction models: II. External validation, model updating, and impact assessment. Heart Br Card Soc [Internet]. 2012 May;98(9):691\\u0026ndash;8.\\u003c/li\\u003e\\n \\u003cli\\u003ePajouheshnia R, Smeden M van, Peelen LM, Groenwold RHH. How variation in predictor measurement affects the discriminative ability and transportability of a prediction model. J Clin Epidemiol. 2019 Jan;105:136\\u0026ndash;41.\\u003c/li\\u003e\\n \\u003cli\\u003eZhi-Hua Z. Ensemble Methods: Foundations and Algorithms. Vol. 13, IEEE Intelligent Informatics Bulletin. Chapman and Hall; 2012.\\u003c/li\\u003e\\n \\u003cli\\u003ePolikar R. Ensemble Learning. In: Zhang C, Ma Y, editors. Ensemble Machine Learning: Methods and Applications [Internet]. Boston, MA: Springer US; 2012. p. 1\\u0026ndash;34.\\u003c/li\\u003e\\n \\u003cli\\u003eHu X, Madden LV, Edwards S, Xu X. Combining Models is More Likely to Give Better Predictions than Single Models. Httpdxdoiorg101094PHYTO-11-14-0315-R [Internet]. 2015 Aug;105(9):1174\\u0026ndash;82.\\u003c/li\\u003e\\n \\u003cli\\u003eBates JM, Granger CWJ. The Combination of Forecasts. OR [Internet]. 1969;20(4):451\\u0026ndash;68.\\u003c/li\\u003e\\n \\u003cli\\u003eKrogh A, Sollich P. Statistical mechanics of ensemble learning. Phys Rev E [Internet]. 1997 Jan 1;55(1):811\\u0026ndash;25.\\u003c/li\\u003e\\n \\u003cli\\u003eKhan MAB, Hashim MJ, King JK, Govender RD, Mustafa H, Al Kaabi J. Epidemiology of Type 2 Diabetes \\u0026ndash; Global Burden of Disease and Forecasted Trends. J Epidemiol Glob Health [Internet]. 2020 Mar;10(1):107\\u0026ndash;11.\\u003c/li\\u003e\\n \\u003cli\\u003eHarding JL, Pavkov ME, Magliano DJ, Shaw JE, Gregg EW. Global trends in diabetes complications: a review of current evidence. Diabetologia [Internet]. 2019 Jan 1;62(1):3\\u0026ndash;16.\\u003c/li\\u003e\\n \\u003cli\\u003eAmerican Diabetes Association Professional Practice Committee. 4. Comprehensive Medical Evaluation and Assessment of Comorbidities: Standards of Medical Care in Diabetes\\u0026mdash;2022. Diabetes Care [Internet]. 2021 Dec 16;45(Supplement_1):S46\\u0026ndash;59.\\u003c/li\\u003e\\n \\u003cli\\u003eNdjaboue R, Ngueta G, Rochefort-Brihay C, Delorme S, Guay D, Ivers N, et al. Prediction models of diabetes complications: a scoping review. J Epidemiol Community Health [Internet]. 2022 Jun 30;\\u003c/li\\u003e\\n \\u003cli\\u003eSlieker RC, van der Heijden AAWA, Siddiqui MK, Langendoen-Gort M, Nijpels G, Herings R, et al. Performance of prediction models for nephropathy in people with type 2 diabetes: systematic review and external validation study. BMJ. 2021 Sep 28;374:n2134.\\u003c/li\\u003e\\n \\u003cli\\u003eGaries S, Birtwhistle R, Drummond N, Queenan J, Williamson T. Data Resource Profile: National electronic medical record data from the Canadian Primary Care Sentinel Surveillance Network (CPCSSN). Int J Epidemiol [Internet]. 2017 Aug 1;46(4):1091\\u0026ndash;1092f.\\u003c/li\\u003e\\n \\u003cli\\u003eQueenan JA, Williamson T, Khan S, Drummond N, Garies S, Morkem R, et al. Representativeness of patients and providers in the Canadian Primary Care Sentinel Surveillance Network: a cross-sectional study. CMAJ Open [Internet]. 2016;4(1):E28-32.\\u003c/li\\u003e\\n \\u003cli\\u003eNie JX, Wang L, Tracy CS, Moineddin R, Upshur RE. Health care service utilization among the elderly: findings from the Study to Understand the Chronic Condition Experience of the Elderly and the Disabled (SUCCEED project). J Eval Clin Pract [Internet]. 2008 Dec;14(6):1044\\u0026ndash;9.\\u003c/li\\u003e\\n \\u003cli\\u003eMorkem R, Salman A, Herman C, Shah R, Wong S, Barber D. CPCSSN Data Quality: An Opportunity for Enhancing Canadian Primary Care Data. 2023 Apr.\\u003c/li\\u003e\\n \\u003cli\\u003eBlack JE, Campbell DJ, Ronksley PE, McBrien KA, Williamson TS. Performance of clinical prediction models for chronic kidney disease among people with diabetes: External validation using the Canadian Primary Care Sentinel Surveillance Network (CPCSSN) [Internet]. Research Square; 2025.\\u003c/li\\u003e\\n \\u003cli\\u003eWilliamson T, Green ME, Birtwhistle R, Khan S, Garies S, Wong ST, et al. Validating the 8 CPCSSN Case Definitions for Chronic Disease Surveillance in a Primary Care Database of Electronic Health Records. Ann Fam Med [Internet]. 2014 Jul;12(4):367\\u0026ndash;72.\\u003c/li\\u003e\\n \\u003cli\\u003eLethebe BC, Williamson T, Garies S, McBrien K, Leduc C, Butalia S, et al. Developing a case definition for type 1 diabetes mellitus in a primary care electronic medical record database: an exploratory study. Can Med Assoc Open Access J [Internet]. 2019 Apr;7(2):E246\\u0026ndash;51.\\u003c/li\\u003e\\n \\u003cli\\u003eMcFarlane P, Cherney D, Gilbert R, Senior P. Diabetes Canada 2018 Clinical Practice Guidelines for the Prevention and Management of Diabetes in Canada: Chronic Kidney Disease in Diabetes. Can J Diabetes. 2018;\\u003c/li\\u003e\\n \\u003cli\\u003eBoer IH de, Caramori ML, Chan JCN, Heerspink HJL, Hurst C, Khunti K, et al. KDIGO 2020 Clinical Practice Guideline for Diabetes Management in Chronic Kidney Disease. Kidney Int [Internet]. 2020 Oct 1;98(4):S1\\u0026ndash;115.\\u003c/li\\u003e\\n \\u003cli\\u003eCommittee; CDACPGE, Cheng AYY. Canadian Diabetes Association 2013 clinical practice guidelines for the prevention and management of diabetes in Canada. Can J Diabetes [Internet]. 2013 Apr;37:S1\\u0026ndash;3.\\u003c/li\\u003e\\n \\u003cli\\u003eThomas RD, Kosowan L, Rabey M, Bell A, Connelly KA, Hawkins NM, et al. Validation of a case definition to identify patients diagnosed with cardiovascular disease in Canadian primary care practices. CJC Open [Internet]. 2023 Apr 22;\\u003c/li\\u003e\\n \\u003cli\\u003eFaisal N, Kosowan L, Zafari H, Zulkernine F, Lix L, Mahar A, et al. Development and validation of a case definition to estimate the prevalence and incidence of cirrhosis in pan-Canadian primary care databases. Can Liver J [Internet]. 2023 Oct 19;e20230002.\\u003c/li\\u003e\\n \\u003cli\\u003eRigobon AV, Birtwhistle R, Khan S, Barber D, Biro S, Morkem R, et al. Adult obesity prevalence in primary care users: An exploration using Canadian Primary Care Sentinel Surveillance Network (CPCSSN) data. Can J Public Health Rev Can Sante Publique. 2015 Apr 30;106(5):e283-289.\\u003c/li\\u003e\\n \\u003cli\\u003eSpohn O, Morkem R, Singer AG, Barber D. Prevalence and management of dyslipidemia in primary care practices in Canada. Can Fam Physician [Internet]. 2024 Mar 1;70(3):187\\u0026ndash;96.\\u003c/li\\u003e\\n \\u003cli\\u003ePampalon R, Hamel D, Gamache P, Raymond G. A deprivation index for health planning in Canada. Chronic Dis Can. 2009;29(4):178\\u0026ndash;91.\\u003c/li\\u003e\\n \\u003cli\\u003ePavlou M, Qu C, Omar RZ, Seaman SR, Steyerberg EW, White IR, et al. Estimation of required sample size for external validation of risk models for binary outcomes. Stat Methods Med Res. 2021 Oct;30(10):2187\\u0026ndash;206.\\u003c/li\\u003e\\n \\u003cli\\u003eJanssen KJM, Vergouwe Y, Donders ART, Harrell FE Jr, Chen Q, Grobbee DE, et al. Dealing with Missing Predictor Values When Applying Clinical Prediction Models. Clin Chem [Internet]. 2009 May 1;55(5):994\\u0026ndash;1001.\\u003c/li\\u003e\\n \\u003cli\\u003eVergouwe Y, Nieboer D, Oostenbrink R, Debray TPA, Murray GD, Kattan MW, et al. A closed testing procedure to select an appropriate method for updating prediction models. Stat Med. 2017 Dec;36(28):4529\\u0026ndash;39.\\u003c/li\\u003e\\n \\u003cli\\u003eJacobs RA, Jordan MI, Nowlan SJ, Hinton GE. Adaptive mixtures of local experts. Neural Comput. 1991;3(1):79\\u0026ndash;87.\\u003c/li\\u003e\\n \\u003cli\\u003ePencina MJ, D\\u0026rsquo; Agostino Sr RB, D\\u0026rsquo; Agostino Jr RB, Vasan RS. Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond. Stat Med [Internet]. 2008;27(2):157\\u0026ndash;72.\\u003c/li\\u003e\\n \\u003cli\\u003ePencina MJ, D\\u0026rsquo;Agostino RB, Steyerberg EW. Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Stat Med [Internet]. 2011 Jan;30(1):11\\u0026ndash;21.\\u003c/li\\u003e\\n \\u003cli\\u003eNelson RG, Grams ME, Ballew SH, Sang Y, Azizi F, Chadban SJ, et al. Development of Risk Prediction Equations for Incident Chronic Kidney Disease. JAMA [Internet]. 2019 Dec 3;322(21):2104\\u0026ndash;14.\\u003c/li\\u003e\\n \\u003cli\\u003eCollins GS, Groot JA de, Dutton S, Omar O, Shanyinde M, Tajar A, et al. External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC Med Res Methodol [Internet]. 2014;14:40.\\u003c/li\\u003e\\n \\u003cli\\u003eWu H, Zhang H, Karwath A, Ibrahim Z, Shi T, Zhang X, et al. Ensemble learning for poor prognosis predictions: A case study on SARS-CoV-2. J Am Med Inform Assoc JAMIA. 2021 Mar 18;28(4):791\\u0026ndash;800.\\u003c/li\\u003e\\n \\u003cli\\u003eArora P, Boyne D, Slater JJ, Gupta A, Brenner DR, Druzdzel MJ. Bayesian Networks for Risk Prediction Using Real-World Data: A Tool for Precision Medicine. Value Health [Internet]. 2019 Apr 1;22(4):439\\u0026ndash;45.\\u003c/li\\u003e\\n\\u003c/ol\\u003e\"}],\"fulltextSource\":\"\",\"fullText\":\"\",\"funders\":[],\"hasAdminPriorityOnWorkflow\":false,\"hasManuscriptDocX\":true,\"hasOptedInToPreprint\":true,\"hasPassedJournalQc\":\"\",\"hasAnyPriority\":true,\"hideJournal\":true,\"highlight\":\"\",\"institution\":\"University of Calgary\",\"isAcceptedByJournal\":false,\"isAuthorSuppliedPdf\":false,\"isDeskRejected\":\"\",\"isHiddenFromSearch\":false,\"isInQc\":false,\"isInWorkflow\":false,\"isPdf\":false,\"isPdfUpToDate\":true,\"isWithdrawnOrRetracted\":false,\"journal\":{\"display\":true,\"email\":\"info@researchsquare.com\",\"identity\":\"researchsquare\",\"isNatureJournal\":false,\"hasQc\":true,\"allowDirectSubmit\":true,\"externalIdentity\":\"\",\"sideBox\":\"\",\"snPcode\":\"\",\"submissionUrl\":\"/submission\",\"title\":\"Research Square\",\"twitterHandle\":\"researchsquare\",\"acdcEnabled\":true,\"dfaEnabled\":false,\"editorialSystem\":\"\",\"reportingPortfolio\":\"\",\"inReviewEnabled\":false,\"inReviewRevisionsEnabled\":true},\"keywords\":\"\",\"lastPublishedDoi\":\"10.21203/rs.3.rs-6297944/v1\",\"lastPublishedDoiUrl\":\"https://doi.org/10.21203/rs.3.rs-6297944/v1\",\"license\":{\"name\":\"CC BY 4.0\",\"url\":\"https://creativecommons.org/licenses/by/4.0/\"},\"manuscriptAbstract\":\"\\u003cp\\u003e\\u003cem\\u003eClinical prediction models often suffer from poor model transportability and/or subgroup performance resulting from using a single data source. We aimed to determine whether ensemble methods can combine multiple existing models to improve predictive performance when compared to component models. As a case study, we used electronic medical records from the Canadian Primary Care Sentinel Surveillance Network (CPCSSN) to test ensemble methods for models estimating the risk of developing chronic kidney disease (CKD) among people with diabetes in a cohort of 37,604 individuals. We considered 13 models identified from prior systematic reviews and combined their unique risk estimates using many strategies (e.g., averaging or mixture-of-experts). We assessed discrimination, precision, recall, calibration, net reclassification index, and integrated discrimination improvement. Ensemble methods performed well, but no better than the best performing component model. This study suggests ensemble methods may not improve predictive performance, though further research should confirm these findings.\\u003c/em\\u003e\\u003c/p\\u003e\",\"manuscriptTitle\":\"Can ensemble methods improve predictive performance of existing models estimating chronic kidney disease among patients with diabetes?\",\"msid\":\"\",\"msnumber\":\"\",\"nonDraftVersions\":[{\"code\":1,\"date\":\"2025-03-27 05:43:59\",\"doi\":\"10.21203/rs.3.rs-6297944/v1\",\"editorialEvents\":[{\"type\":\"communityComments\",\"content\":0}],\"status\":\"published\",\"journal\":{\"display\":true,\"email\":\"info@researchsquare.com\",\"identity\":\"researchsquare\",\"isNatureJournal\":false,\"hasQc\":true,\"allowDirectSubmit\":true,\"externalIdentity\":\"\",\"sideBox\":\"\",\"snPcode\":\"\",\"submissionUrl\":\"/submission\",\"title\":\"Research Square\",\"twitterHandle\":\"researchsquare\",\"acdcEnabled\":true,\"dfaEnabled\":false,\"editorialSystem\":\"\",\"reportingPortfolio\":\"\",\"inReviewEnabled\":false,\"inReviewRevisionsEnabled\":true}}],\"origin\":\"\",\"ownerIdentity\":\"289a6afe-a67b-4f3c-8d9b-7f8c0d3b0136\",\"owner\":[],\"postedDate\":\"March 27th, 2025\",\"published\":true,\"recentEditorialEvents\":[],\"rejectedJournal\":[],\"revision\":\"\",\"amendment\":\"\",\"status\":\"posted\",\"subjectAreas\":[],\"tags\":[],\"updatedAt\":\"2025-03-27T05:43:59+00:00\",\"versionOfRecord\":[],\"versionCreatedAt\":\"2025-03-27 05:43:59\",\"video\":\"\",\"vorDoi\":\"\",\"vorDoiUrl\":\"\",\"workflowStages\":[]},\"version\":\"v1\",\"identity\":\"rs-6297944\",\"journalConfig\":\"researchsquare\"},\"__N_SSP\":true},\"page\":\"/article/[identity]/[[...version]]\",\"query\":{\"redirect\":\"/article/rs-6297944\",\"identity\":\"rs-6297944\",\"version\":[\"v1\"]},\"buildId\":\"8U1c8b4HqxoKbykW_rLl7\",\"isFallback\":false,\"isExperimentalCompile\":false,\"dynamicIds\":[84888],\"gssp\":true,\"scriptLoader\":[]}","source_license":"CC-BY-4.0","license_restricted":false}