On the importance of a clear definition of time horizon for time-to-event dynamic predictions: a systematic review and a concrete illustration in kidney transplantation

On the importance of a clear definition of time horizon for time-to-event dynamic predictions: a systematic review and a concrete illustration in kidney transplantation | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article On the importance of a clear definition of time horizon for time-to-event dynamic predictions: a systematic review and a concrete illustration in kidney transplantation Lucas Chabeau, Vincent Bonnemains, Pierre Rinder, Magali Giral, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-3938204/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Background. Time-to-event dynamic predictions are defined as the probability to survive until a defined time horizon given being event-free at landmark times and given available predictive variables at such prediction times. From two different mathematical formulations, dynamic predictions can either predict the survival probability until a final time horizon or until the end of a sliding horizon window. We aim to illustrate the need to clearly define the time horizon to correctly interpret the prognostic performances. Methods. First, following the PRISMA, CHARMS and TRIPOD recommendations, we conducted a systematic review of articles concerning dynamic predictions to assess how the time horizon was reported in the literature. Second, using a sample of 2,523 kidney recipients, we assessed the prognostic capacities of the Dynamic predictions of Patient and kidney Graft survival (DynPG) using either a final time horizon or a sliding horizon window . Results. Of 172 references retrieved about dynamic predictions, 102 articles were included in the systematic review. We notably observed that 71 (69.6%) used a sliding horizon window to assess the prognostic performance while 18 (17.7%) used a final time horizon . We also identified 13 articles (12.7%) where the time horizon was not defined clearly (or at all). Our concrete application in kidney transplantation shows that discrimination and calibration are not the same when comparing the two time horizon definitions. On one hand, for a 5-year sliding horizon window , the discrimination slightly increased as the landmark times increased, and we also observed that DynPG is reasonably well calibrated, particularly for the earliest landmark times. On the other hand, for an 11-year final time horizon , the discrimination was high for the earliest landmark times and increased over time, while the calibration plot revealed predictions were underestimated for the earliest landmark times and overestimated for later ones. Conclusions. Our systematic review identified a clear heterogeneity in the time horizon definition used, and an absence of a clear time horizon definition in a part of published articles. Our study advocates for improving the reporting when studying dynamic prediction scoring systems since the prognostic performances and interpretation differ according to the time horizon definition. Time-to-event dynamic predictions landmark times horizon window time horizon discrimination calibration Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 1. Background In the personalized medicine era [ 1 ], time-to-event dynamic predictions are becoming more widespread. They are defined as the probability to survive until a defined time horizon given being event-free at the time of making a prediction (landmark time) and given available predictive variables at such prediction times [ 2 – 5 ]. Dynamic predictions enable to take into account the valuable information consisting of the entire marker trajectory known at landmark time and have shown their importance in improving time-fixed predictions [ 6 – 8 ]. The dynamic predictions are updated predictions whenever additional longitudinal data becomes available during the patient follow-up. The time horizon deserves precisions in reporting since it can be defined in several ways [ 3 , 6 ]. One objective could be to predict the survival probability until a final time horizon . For instance, Teramukai et al. were interested in dynamically predicting cardiovascular endpoints of hypertensive patients until 3 years after inclusion using repeated on-treatment blood pressure measurements [ 9 ]. An alternative objective could be to predict the survival probability until the end of a sliding horizon window [ 6 , 10 , 11 ]. Ben-Hassen et al. proposed to dynamically predict dementia from longitudinal neurocognitive tests on a 5-year sliding horizon window [ 12 ]. The literature on dynamic predictions is growing. But, to our knowledge, a clear definition of these two objectives does not exist yet and the comparison between the two approaches have not been investigated. To be useful, prognostic scores require good prognostic performances [ 13 ]. They can be studied through global performances using time-dependent Brier Score or R²-curve [ 10 , 14 ], discrimination property using time-dependent AUC of ROC curves [ 14 , 15 ] and calibration property using time-dependent calibration plots [ 16 ]. Since the estimated dynamic predictions would not be the same given the definition of the time horizon ( final time horizon or sliding horizon window ), this is of major importance when assessing the prognostic performances. The objective was to illustrate that the prognostic performances and their interpretations differ given the two time horizon definitions. We presented the mathematical framework to obtain dynamic predictions according the two definitions (Section 2). We then conducted a systematic review of articles concerning dynamic predictions to objectively assess how the time horizon was reported in the literature (Section 3). We illustrated that prognostic performances of dynamic predictions of kidney graft failure differ between the time horizon definitions (Section 4). Finally, section 5 offers a discussion and conclusions. 2. Methods 2.1 Notations Let’s assume a learning sample of \(n\) independent and identically distributed patients with the observed data \(\left\{{Y}_{i},{t}_{i}, {T}_{i},{\delta }_{i},{A}_{i};i=1,\dots , n\right\}\) . Here, \({Y}_{i}=\left\{{Y}_{ij};j=1,\dots ,{n}_{i}\right\}\) corresponds to the set of \({n}_{i}\) longitudinal marker values measured at the corresponding times \({t}_{i}=\left\{{t}_{ij};j=1,\dots ,{n}_{i}\right\}\) in individual \(i\) . Let’s consider \({T}_{i}=\text{min}\left({T}_{i}^{*}, {C}_{i}\right)\) with \({T}_{i}^{*}\) the true time-to-event, \({C}_{i}\) the censoring time for subject \(i\) , and \({\delta }_{i}=1\left\{{T}_{i}^{*}\le {C}_{i}\right\}\) the event indicator function taking 1 if the event is not censored and 0 otherwise. We note \({A}_{i}=\left\{{A}_{ik};k=1,\dots ,K\right\}\) the \(K\) baseline variables. From a learning sample of \(n\) patients, we can estimate a prediction model of parameters \(\theta\) . Among those, landmarking and joint modeling of longitudinal and survival data are popular approaches [ 5 , 17 , 18 ]. 2.2. Dynamic predictions definitions The dynamic prediction \({\pi }_{l}\) for a patient \(l\) is defined as the probability of being event-free until a time horizon \(u\) given being event-free at the landmark time \(s \left(s<u\right)\) , given baseline variables \({A}_{l}\) and the longitudinal marker history available at time \(s\) (i.e. \({\stackrel{\sim}{Y}}_{l}\left(s\right)=\left\{{Y}_{l}\left({s}_{1}\right),\dots ,{Y}_{l}\left({s}_{p}\right);0\le {s}_{1}<\dots <{s}_{p}u|{T}_{l}^{*}>s,{A}_{l},{\stackrel{\sim}{Y}}_{l}\left(s\right);\theta \right)\) \(\left(3\right)\) We report two different uses of such generic definition of dynamic predictions [ 2 – 5 , 7 ], that depend on the definition of time horizon \(u\) and that would not have the same clinical objective. Dynamic predictions given a final time horizon. The objective would be to dynamically predict the survival probability \({\pi }_{l}\) for a patient \(l\) until a fixed final time horizon \(U\) (Fig. 1 ). It corresponds to the probability to not suffer the event on a horizon window defined as the delay between the landmark time \(s\) and the final time horizon \(U\) : $${\pi }_{l}\left(s,U;\theta \right)=\text{Pr}\left({T}_{l}^{*}>U|{T}_{l}^{*}>s,{A}_{l},{\stackrel{\sim}{Y}}_{l}(s);\theta \right)$$ Using the estimated parameters \(\widehat{\theta }\) of the prediction model, the dynamic predictions considering a final time horizon can be estimated as the ratio between the survival probability until the final time horizon \(U\) and the survival probability until the time of making prediction \(s\) : $${\widehat{\pi }}_{l}\left(s,U;\widehat{\theta }\right) = \frac{Pr\left({T}_{l}^{*}>U|{A}_{l},{\stackrel{\sim}{Y}}_{l}\left(s\right);\widehat{\theta }\right)}{Pr\left({T}_{l}^{*}>s|{A}_{l},{\stackrel{\sim}{Y}}_{l}\left(s\right);\widehat{\theta }\right)}$$ Following this, the horizon window \(\left\{U-s\right\}\) is progressively reduced when landmark time \(s\) increases (Fig. 1 ). We note that for a fixed \({\stackrel{\sim}{Y}}_{l}\left(s\right)\) , the smaller the difference between \(s\) and \(U\) , the bigger \({\widehat{\pi }}_{l}\left(s,U\right)\) will be. Indeed, \({\pi }_{l}\left(s,U;\theta \right)\) is bounded with: $$\underset{s\to 0}{\text{lim}}{\widehat{\pi }}_{l}\left(s,U;\widehat{\theta }\right) = \frac{Pr\left({T}_{l}^{*}>U|{A}_{l},{\stackrel{\sim}{Y}}_{l}(0);\widehat{\theta }\right)}{1} = Pr\left({T}_{l}^{*}>U|{A}_{l},{\stackrel{\sim}{Y}}_{l}(0);\widehat{\theta }\right)$$ $$\underset{s\to U}{\text{lim}}{\widehat{\pi }}_{l}\left(s,U;\widehat{\theta }\right) = \frac{Pr\left({T}_{l}^{*}>U|{A}_{l},{\stackrel{\sim}{Y}}_{l}(U);\widehat{\theta }\right)}{Pr\left({T}_{l}^{*}>U|{A}_{l},{\stackrel{\sim}{Y}}_{l}(U);\widehat{\theta }\right)} = 1$$ Dynamic predictions given a sliding horizon window. Some authors have proposed an alternative definition of the dynamic predictions [ 2 , 10 , 11 ]. The target of inference of a patient \(l\) is to predict the survival probability until the end of a horizon window of length \(\varDelta t\) : $${\pi }_{l}\left(s,s+\varDelta t;\theta \right)=\text{Pr}\left({T}_{l}^{*}>s+\varDelta t|{T}_{l}^{*}>s,{A}_{l},{\stackrel{\sim}{Y}}_{l}(s);\theta \right)$$ Following this definition, the horizon window \(\varDelta t\) is thus sliding when updating the prediction, i.e. the landmark time increases (Fig. 2 ). Using the estimated parameters \(\widehat{\theta }\) of the prediction model, the dynamic predictions given a sliding horizon window can be decomposed as the ratio between the survival probability until the end of the horizon window \(s+\varDelta t\) and the survival probability until the time of making prediction \(s\) : $${\widehat{\pi }}_{l}\left(s,s+\varDelta t;\widehat{\theta }\right) = \frac{Pr\left({T}_{l}^{*}>s+\varDelta t|{A}_{l},{\stackrel{\sim}{Y}}_{l}\left(s\right);\widehat{\theta }\right)}{Pr\left({T}_{l}^{*}>s|{A}_{l},{\stackrel{\sim}{Y}}_{l}\left(s\right);\widehat{\theta }\right)}$$ 2.3. Accuracy measures for dynamic predictions Following these two definitions, the horizon window is either reduced or sliding as the landmark time increases. Therefore, the at-risk population at landmark times would be identical given the two definitions, but the incidence rate on the prediction window would not. Consequently, the prognostic performances and the corresponding interpretation will differ. The global prognostic performance of dynamic predictions can be assessed by measuring the prediction error [ 19 ] with the popular Brier Score metric, for instance. The time-dependent expected Brier Score is defined as follows [ 14 , 17 ]: $$BS\left(s,u\right)=E\left[\left(1\left({T}^{*}>u\right)-\pi \left(s,u\right)\right)²|{T}^{*}>s\right]$$ The Brier Score is a mean square error term. The closer the prediction is to the observation, the closer the Brier Score is to 0. When considering the final time horizon , it means that the survival probability until a final time horizon is expected to be close to the observed event indicator after the final time horizon \(\left(BS\left(s,U\right)=E\left[\left(1\left({T}^{*}>U\right)-\pi \left(s,U\right)\right)²|{T}^{*}>s\right]\right)\) . When considering the sliding horizon window , it means that the survival probability after an horizon window \(\varDelta t\) is expected to be close to the observed event indicator after this horizon window \(\left(BS\left(s,s+\varDelta t\right)=E\left[\left(1\left({T}^{*}>s+\varDelta t\right)-\pi \left(s,s+\varDelta t\right)\right)²|{T}^{*}>s\right]\right)\) . In a dynamic context, one limit of the Brier Score is that it is sensitive to the marginal event probability that could change as the landmark time increases. van Houwelingen et al. proposed studying the relative error reduction [ 19 ], also named R²-curve [ 10 ]: $${R}^{2}\left(s,u\right)=1-\frac{BS\left(s,u\right)}{B{S}_{0}\left(s,u\right)}$$ where \(BS\left(s,u\right)\) is the Brier Score obtained using the model of interest and \(B{S}_{0}\left(s,u\right)\) the Brier Score of a reference model (Kaplan-Meier estimator for instance) allowing to assess the global performance of dynamic predictions whatever the evolution of the marginal event probability along the landmark times. Since the Brier Score computation would be impacted by the consideration of the final time horizon or the sliding horizon window , the R²-curve would also not be the same between the two approaches. The discrimination is the ability of a prognostic tool to order the risk between subjects. To assess discrimination, the well-known Area Under the Receiver Operating Characteristics Curve (AUC) [ 14 , 15 ] can be explicitly defined in a dynamic context as: $$AUC\left(s,u\right)=\text{Pr}\left({\pi }_{l}\left(s,u\right)>{\pi }_{{l}^{{\prime }}}\left(s,u\right)|{T}_{l}^{*}>u,{s<T}_{{l}^{{\prime }}}^{*}<u\right)$$ When considering the final time horizon , the AUC corresponds to the probability that a subject \(l\) at-risk at landmark time \(s\) and who would not suffer the event before the final time horizon \(U\) would have a higher survival prediction than a subject \({l}^{{\prime }}\) at-risk at landmark time \(s\) and who would suffer the event before the final time horizon \(U\) \(\left(AUC\left(s,U\right)=\text{Pr}\left({\pi }_{l}\left(s,U\right)>{\pi }_{{l}^{{\prime }}}\left(s,U\right)|{T}_{l}^{*}>U,{s<T}_{{l}^{{\prime }}}^{*}<U\right)\right)\) . When considering the sliding horizon window , the AUC corresponds to the probability that a subject \(l\) at-risk at landmark time \(s\) and who would not suffer the event before the end of the horizon window \(s+\varDelta t\) would have a higher survival prediction than a subject \({l}^{{\prime }}\) at-risk at landmark time \(s\) and who would suffer the event between \(s\) and \(s+\varDelta t\) \(\left(AUC\left(s,s+\varDelta t\right)=\text{Pr}\left({\pi }_{l}\left(s,s+\varDelta t\right)>{\pi }_{{l}^{{\prime }}}\left(s,s+\varDelta t\right)|{T}_{l}^{*}>s+\varDelta t,{s<T}_{{l}^{{\prime }}}^{*}<s+\varDelta t\right)\right)\) . The calibration property assesses the ability of a prognostic tool to provide a prediction close to the observed outcome. Usually assessed through calibration plots, the calibration is described by comparing the predicted values within subgroups (defined from quantiles of predictions) to the observed survival probabilities (computed using the Kaplan–Meier estimator for instance). When considering the final time horizon , a good calibration means that, for any given \(x\) value, we expect that among all subjects who are predicted to survive with a \(x\) % probability, \(x\) out of 100 will not experience the event before the final time horizon . When considering the sliding horizon window , a good calibration means that, for any given \(x\) value, we expect that among all subjects who are predicted to survive with a \(x\) % probability, \(x\) out of 100 will not experience the event before the end of the horizon window . 3. Systematic review of dynamic predictions 3.1. Search strategy We conducted a systematic review following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement - PRISMA (Web supplementary materials Table S1 ) [ 20 ]. We searched the Medline database on 25th August 2022, without date or language restriction. The search equation used is reported in Appendix 1 of Web supplementary materials. 3.2. Study selection As this systematic review questioned the definition and the choice of time horizon, we included all articles focusing on the development or the validation of individual dynamic prediction scoring systems as well as methodological papers concerning dynamic predictions. Two reviewers (LC - ED) independently screened references by title and abstract. All eligible texts were assessed independently by the two reviewers (LC - ED) and discussions resolved any discrepancies. Our non-inclusion criteria were reviews or meta-analyses, full texts not found or conference abstracts and editorials. 3.3. Data extraction Each study was randomly allocated to two of the five reviewers (LC – VB – PR – SD – ED). Following the Checklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) [ 21 ], we predefined a standardized form for data extraction. Whether it was a methodological article or an article concerning the development or the validation of dynamic predictions, for each eligible article, we collected the author’s name, year and journal, the data source, the study design, the inclusion and exclusion criteria, the sample size, the population characteristics, the predicted outcomes, the prediction times (i.e. the time when one calculates the prediction), the time horizon (i.e. the end of the prediction time window), the predictive tools and the development details (such as the methodology used, the variables of the scoring systems). For each article, the time horizon was qualified as either a final time horizon or a sliding horizon window from an explicit equation, explicit texts and/or a clear graphic representing individual dynamic prediction, or qualified as unclear information in case of lack of such explicit information or contradictory information. The TRIPOD statement recommends to develop a prognostic score from a learning sample and to validate the prognostic performances from an independent internal and/or external validation sample to avoid reporting the prognostic performances on the learning sample only, which would lead to overestimating the performances [ 22 , 23 ]. We thus extracted information on internal and external validations as well. Whenever necessary, any discrepancies in data extraction were resolved by discussion with another reviewer to reach a consensus. 3.4. Results The search identified 172 unique articles. Screening of titles and abstracts identified 135 papers eligible for full-text review. As detailed on the flow diagram (Fig. 3 ), 33 articles were excluded. Finally, we included 102 articles (Appendix 2 of Web supplementary materials). Dynamic predictions appear to be a relatively recent research topic in biostatistics with 100 (98%) articles published after 2010. Fifty-one (50%) of the retained articles are methodological studies that proposed new modeling approach. This systematic review confirmed the dominance of joint modeling for longitudinal and survival data (n = 42, 41%) and landmarking (n = 44, 43%) to develop dynamic predictions. To a lesser extent, we identified the emergence of machine learning approaches including deep learning (n = 14, 14%), with random survival forest [ 24 – 26 ] or neural networks [ 27 , 28 ] for instance. All articles referring to machine learning approaches were published after 2018. This will probably increase in the future with the rapid development of such modeling approaches. Among the applied articles (n = 51, 50%), we also observed that 12 studies (24%) only presented a development modeling of dynamic prediction without validation of their predictive performance on an independent data set (Table 1 ). This does not respect the TRIPOD recommendations about the right process to develop and validate prognostic performances. Nevertheless, 28 articles included internal validation, 5 an external validation and 5 both types of validation, leading to 76% of the applied articles in agreement with the TRIPOD. Table 1 Description of the articles included in the systematic review All included articles N = 102 Methodological articles N = 51 Applied articles N = 51 Time prediction horizon definition Final time horizon 18 (18%) 5 (10%) 13 (25%) Sliding horizon window 71 (69%) 44 (86%) 27 (53%) Unclear 13 (13%) 2 (4%) 11 (22%) Published after 2010 100 (98%) 51 (100%) 49 (96%) Main statistical analyses for model development Joint model 42 (41%) 26 (51%) 16 (32%) Landmarking 44 (43%) 18 (35%) 26 (52%) Machine Learning 14 (14%) 6 (12%) 8 (16%) Others 9 (9%) 5 (10%) 4 (8%) Global performance 46 (45%) 32 (74%) 14 (28%) Brier Score 28 (61% a ) 21 (66% a ) 7 (% a ) Prediction error 20 (43% a ) 12 (38% a ) 8 (57% a ) Others 1 (2% a ) 1 (3% a ) 0 (0% a ) Discrimination 84 (82%) 34 (79%) 50 (100%) AUC 65 (77% a ) 31 (91% a ) 34 (68% a ) C-index 18 (21% a ) 3 (9% a ) 15 (30% a ) Others 6 (7% a ) 0 (0% a ) 0 (0% a ) Calibration 24 (24%) 6 (14%) 18 (36%) Calibration plot 19 (79% a ) 5 (83% a ) 14 (78% a ) Calibration slope 2 (8% a ) 0 (0% a ) 2 (11% a ) Heuristic shrinkage factor 5 (21% a ) 0 (0% a ) 5 (28% a ) Others 2 (8% a ) 1 (17% a ) 1 (6% a ) Type of study Development alone - - 12 (24%) Development & Internal validation - - 28 (55%) Development & External validation - - 5 (10%) Development, Internal & External validation - - 5 (10%) External validation alone - - 1 (1%) a Percentage among the count of articles reporting the prognostic ability Among the included articles, we identified 18 (17.7%) articles defining dynamic predictions using the final time horizon , 71 (69.6%) articles using sliding horizon windows and 13 (12.7%) articles that were unclear about the time horizon definition (Table 1 ). Methodological articles mainly used a sliding horizon window approach to consider dynamic predictions (n = 44, 86%), and to a lesser extent final time horizon (n = 5, 10%) while only 2 (4%) articles did not clearly define the considered approach. This repartition was more heterogeneous when studying the applied articles: 27 (53%) used sliding horizon windows , 13 (25%) used the final time horizon and 11 (22%) appeared unclear as to which approach was used. Beyond the description of the time horizon considered, our systematic review allows to appraise the reporting quality on dynamic predictions. As clearly recommended in the literature [ 13 ], assessing prognostic performances is of major importance for prediction tools to propose useful scores. Discrimination capacity is well reported (n = 84, 82%) with AUC as the principal performance indicator. Global performances (n = 46, 45%) with a Brier Score are less often provided, as are calibration (n = 24, 24%) with a calibration plot. Details of such prognostic performances are also not homogeneous between methodological and applied articles (Table 1 ). 4. Application 4.1. Context In kidney transplantation, patients are particularly interested by their kidney graft survival, before any risk of adverse outcomes or infections [ 29 ]. In this context, we developed and internally and externally validated ‘Dynamic predictions of Patient and kidney Graft survival’ (DynPG) for kidney recipients alive with a functioning graft at 1-year post-transplantation [ 16 , 30 ]. The main outcome was the delay from 1-year post-transplantation to patient and kidney graft failure defined as the first event between return-to-dialysis, pre-emptive retransplantation and death with a functioning graft. Such dynamic predictions can be obtained from six baseline variables: recipient age, graft rank, cardiovascular histories, pretransplantation anti-HLA class I immunization, serum creatinine at 3-months post-transplantation, occurrence of acute rejection in the first year post-transplantation, and also the complete longitudinal serum creatinine trajectory available at the time of prediction. Our aim was to provide to patients and physicians an information on the possible future on a sliding horizon window of 5 years. In line with this clinical objective, the DynPG was defined as the probability of being graft failure-free over the next 5 years after the landmark time, for each prediction time from 1 to 6 years post-transplantation. We retained 6 years post-transplantation as the maximal landmark time since there were 178 patients still at risk of patient and kidney graft failure at 11 years post-transplantation in the validation sample. We aimed to illustrate that the prognostic performances would not be the same under the assumptions of a sliding horizon window or a final time horizon. While a sliding horizon window may be relevant to monitor the patient risk along his/her follow-up with always the same horizon window of 5 years, the clinical objective is different under the prism of a final time horizon . Considering the final time horizon of 11 years post-transplantation, the dynamic predictions aim to predict with good confidence until this final time horizon . Benefiting from the previously estimated joint model and the same internal validation sample, we compared the prognostic performances of the DynPG under the assumption of a 5-year sliding horizon window and under the assumption of a 11 years final time horizon . 4.2. Study population Data were extracted from the French multicentric observational and prospective DIVAT cohort (Données Informatisées et VAlidées en Transplantation; www.divat.fr , CNIL no. 914184, ClinicalTrials.gov recording NCT02900040). All participants gave informed consent. The inclusion criteria were adult recipients who received a first or second renal graft transplanted after January 2000 from a living or heart-beating deceased donor, alive with a functioning graft at 1 year post-transplantation and maintained under tacrolimus and mycophenolate. The extracted DIVAT cohort data consisted of a learning set of 2,749 patients, initially used to estimate the shared-random joint model, and an internal validation set of 2,589 patients. Due to missing data for retained predictors, 66 patients were excluded from the analysis of the validation set (n = 2,523). More details on this validation sample are reported in Fournier et al. [ 16 ]. 4.3. Results Under the 5-year sliding horizon window setting, the results were as previously published [ 16 ]. The global prognostic performance of the DynPG appeared relatively stable along the landmark times with R² values ranging from 14% (95% CI 7–21%) to 15% (95% CI -2–33%) at 1 and 6 years post-transplantation, respectively (Fig. 4 A). The discrimination slightly increased along the prediction times with the AUC values ranging from 0.72 (95% CI 0.67–0.78) to 0.76 (95% CI 0.68–0.85) at 1 and 6 years post-transplantation (Fig. 4 B). Since the discrimination performances increased and the global prognostic performances remained stable, the calibration properties decreased for the late landmark times (Figure S1 ). Under the 11 years final time horizon assumption, the global prognostic performances of the DynPG were at a higher level for the earliest landmark times, but decreased along the landmark times. We estimated R² values of 22% (95% CI 12–32%), 15% (95% CI -2–31%) and − 2% (95% CI -41–37%) at 1, 6 and 10 years post-transplantation, respectively (Fig. 5 A). The discrimination was high for the earliest landmark times and increased over times with AUC values of 0.76 (95% CI 0.70–0.82), 0.76 (95% CI 0.67–0.84) and 0.88 (95% CI 0.79–0.98) at 1, 6 and 10 years post-transplantation, respectively (Fig. 5 B). On the other hand, we observed a very poor calibration plot (Figure S2): the predictions appeared underestimated for the earliest landmark times and overestimated for the late ones. The number of at-risk patients at landmark times are the same for the two time horizon definitions. However, the number of observed events and the number of censored subjects on each window differs between the two approaches due different horizon windows, resulting in a number of at-risk patients at the end of the horizon window that are not comparable (Figures S3, S4). The incidence rate on the prediction window, that is either reduced or sliding given the landmark time increases, are thus not the same between the two approaches. Consequently, the dynamic predictions cannot be compared and they tell a different story. For instance, under the sliding horizon window assumption, at 1 year post-transplantation, we estimated a 72% probability that the predicted survival of a patient who actually experienced a graft failure within the 5 years was lower than that of a patient who did not (Fig. 4 – part B). Under the final time horizon assumption, at 1 year post-transplantation, we estimated a 76% probability that the predicted survival of a patient who actually experienced a graft failure before 11 years post-transplantation was lower than that of a patient who did not (Fig. 5 – part B). In terms of calibration properties, we may reasonably accept that DynPG was sufficiently well calibrated for the earliest landmark times under the 5-year sliding horizon window assumption (Figure S1 ). In contrast, under the 11-year final time horizon assumption, the calibration is quite poor (Figure S2). 5. Discussion In this study, we distinguished two types of time prediction horizon - final time horizon or sliding horizon window - for dynamic predictions. We conducted a systematic review that stated the heterogeneity of the used time prediction horizons in the literature about dynamic predictions. While the two definitions are similar, a specific definition is of major importance since the prognostic performances obtained are different given the nature of the prediction window. This is also well illustrated by our concrete application in kidney transplantation. The concept of P4-medicine (predictive, preventive, personalized and participatory) is now largely developed in the literature, but still difficult to apply in clinical practice [ 1 ]. Dynamic predictive tools can be beneficial for a such health policy and promote shared medical decision making, provided that the prognostic performances are sufficiently good. While dynamic predictions are updated predictions whenever additional information is available during the patient follow-up [ 2 , 6 ], their associated prognostic performances depend on the at-risk population at the time of making a prediction and can evolve as the landmark time increases. In this work, we insist on the fact that prognostic performances are also related to the prediction window. We showed that the incidence rates of the event would not be identical between the two time horizon definitions due to the nature of the window that can be either reduced or sliding, resulting in prognostic performances that are not comparable. The corresponding interpretations of prognostic performances should therefore be formulated with caution since they do not tell the same story in the two contexts. In our opinion, the dynamic predictions obtained with the sliding horizon window framework can be used to follow the patient health evolution accross the landmark times and thus present interesting properties for individualized and personalized medicine [ 16 , 30 ]. With a fixed length of prediction window, it allows to appreciate the prognostic performances all along the follow-up. In our concrete application in kidney transplantation, the ‘Dynamic predictions of patient and kidney graft survival’ has a good discrimination property useful for stratified medicine. The good calibration properties from 1-year until 6-year landmark times are important point for personalized care. We argued that this dynamic predictive tool can be useful for instance to inform kidney transplant recipients of their prognosis over the next 5 years [ 16 ]. This may result in increased patient adherence to their treatment, to increase patient empowerment, as patients recognize having an active role in their chronic disease evolution [ 31 , 32 ]. This could also help patients to better manage their feelings of uncertainty about the survival of their transplant in the not-too-distant future. When considering the final time horizon (i.e. reducing window), the comparison of prognostic performances across the landmark times is difficult since the lengths of the prediction window are not the same. Despite the bounded character of dynamic predictions for a final time horizon , it is important to note that such predictions are not always monotonic along the landmark times because of the actualization of the longitudinal marker. The amelioration or deterioration of the prognosis between two landmark times is not necessarily due to a difference in health state characterized by the new marker measurement. This is noised by the mathematical artefact brought by the reduction of the prediction window. Since it is difficult to know to what extent the prognosis evolution is due to the reduction of the window rather than to the marker evolution, a comparison cannot be done between predictions realized at two different landmark times. For this reason, the final time horizon approach does not seem suited to follow the patient health evolution through survival probabilities and cannot be envisaged to support patient personalized care. In our kidney transplantation application, the poor calibration property across the landmark times for a final time horizon of 11 years post-transplantation do not allow to consider that individual survival probability are correctly predicted. The more the prediction time approaches 11 years, the more we predicted patient and kidney survival probabilities close to 1, while the observed event repartition appear not. This inevitably results in poor calibration. Such an approach cannot be used to personalize the kidney recipient taking care based on the predicted patient and kidney graft survival probabilities. Nevertheless, in a context in which the objective is not the personalization of patient care, but rather the stratification of the studied population, considering a final time horizon may be of interest. This should be clearly stated in the clinical objective. In kidney transplantation, we may envisage to guide patients through the care organization given their stratified survival probabilities. In our application, we observed an increase of the discrimination performance of the DynPG up to 11 years post-transplantation. We may consider to better organize the healthcare visit by reducing the patient visit schedule or consider telemedicine consultations for recipients at higher patient and kidney graft survival probabilities and by reinforcing the patient follow-up for patients with lower survival predictions [ 33 , 34 ]. All the same, we recognize that it may be difficult to determine a final time horizon of clinical interest, particularly in the context of chronic disease. In our application, the horizon of 11 years is debatable from a patient-centered perspective should difficulty be bounded by a terminal horizon. This may be more appropriate in a public health perspective where efficient allocation of resources could provide economic benefits. Other clinical contexts may be of interest when a final time horizon is already known as for medical devices, with for instance the prediction of aortic valve duration or the prediction of hip replacement duration up to a maximum announced by the supplier [ 35 ]. Considering these methodological differences between the two time horizon definitions, the studies’ clinical objective should be clearly anticipated to correctly assess the prognostic performances. Indirectly, the heterogeneity that we observed in the reporting of the time horizon window in our systematic review reveals the lack of clarity regarding the clinical objectives justifying dynamic predictions development. Our systematic review also informed about the methodological quality of dynamic predictions. A non negligible proportion of the literature did not sufficiently respect the TRIPOD recommendation [ 22 ] with a lack of validation studies for instance. We also noted that discrimination properties were mainly reported and referred to the stratification of the studied population, but calibration metrics that are essential for personalized care were not reported in numerous studies. In the literature, numerous developments concern improvements of the joint modeling of the longitudinal and the survival processes and thus participate to the improvement of dynamic predictions performances. For instance, considering competing risks [ 36 , 37 ], to refine the longitudinal evolution with flexible modeling [ 38 , 39 ], or incorporating multiple longitudinal markers [ 36 , 40 ], would help to better predict the clinical event of interest. The mathematical notations introduced and the following concrete application concerns the case of one longitudinal marker (e.g. serum creatinine) and one clinical event (e.g. patient and kidney graft survival). Nevertheless, notations can be adapted to more complex modeling situations. Our conclusion about the attention to bring to time prediction windows would still be valid to broader modeling frameworks. 6. Conclusions In conclusion, our study identified that the choice of the time prediction window is crucial and should depend on the clinical objectives. Our work advocates for clarifying the clinical objective of the dynamic predictions and improving the reporting when studying dynamic prediction scoring systems since prognostic performances and their interpretation will differ given the time horizon definition. Abbreviations 95% CI: 95% Confidence Interval; AUC: Area Under the Curve; CHARMS: CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies; DIVAT: Données Informatisées et VAlidées en Transplantation; DynPG : Dynamic predictions of Patient and kidney Graft survival; HLA: Human Leukocyte Antigen, PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses; ROC: Receiver Operating Characteristic ; TRIPOD: Transparent Reporting of multivariable prediction model for Individual Prognosis Or Diagnosis statements. Declarations Ethics approval and consent to participate Data confidentiality was ensured following the recommendations of the French commission for data protection (Commission Nationale Informatique et Liberté, CNIL no. 914184, ClinicalTrials.gov recording NCT02900040). The present study was approved by the DIVAT scientific committee (Données Informatisées et VAlidées en Transplantation; www.divat.fr). Consent for publication Not applicable Availability of data and materials The data underlying this article could be shared on reasonable request to the corresponding author. Competing interests Lucas Chabeau and Pierre Rinder report financial support as employee of semeia. Other authors declare no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. Funding This work was partly supported by SÊMEIA and also by the French National Research Agency [grant number ANR PRC 2022 JM-QALYs]. Authors' contributions Lucas Chabeau : methodology, formal analysis, systematic review, writing - original draft, review and editing. Vincent Bonnemains : methodology, systematic review, writing - review and editing. Pierre Rinder : methodology, systematic review, writing - review and editing. Magali Giral : scientific coordinator of the DIVAT cohort, clinical mentorship, data collection, writing - review and editing. Solène Desmée: methodology, systematic review, writing - review and editing. Etienne Dantan: conceptualization, methodology, formal analysis, systematic review, writing - original draft, review and editing. Acknowledgments We thank members of the DIVAT consortium* and the clinical research assistant team (S. Le Floch, A. Petit, J. Posson, C. Scellier, V. Eschbach, K. Zurbonsen, C. Dagot, F. M’Raiagh, V. Godel, X. Longy, P. Przednowed). We are also grateful to Roche Pharma, Novartis, Sanofi and Astellas laboratories for supporting the DIVAT cohort (www.divat.fr) as the CENTAURE foundation (www.fondation-centaure.org). * Données Informatisées et VAlidées en Transplantation, DIVAT Cohort Collaborators (Medical Doctors, Surgeons, HLA Biologists) Nantes : Gilles Blancho, Julien Branchereau, Diego Cantarovich, Agnès Chapelet, Jacques Dantal, Clément Deltombe, Lucile Figueres, Claire Garandeau, Magali Giral, Caroline Gourraud-Vercel, Maryvonne Hourmant, Georges Karam, Clarisse Kerleau, Aurélie Meurette, Simon Ville, Christine Kandell, Anne Moreau, Karine Renaudin, Anne Cesbron, Florent Delbos, Alexandre Walencik, Anne Devis ; Paris-Necker : Lucile Amrouche, Dany Anglicheau, Olivier Aubert, Lynda Bererhi, Christophe Legendre, Alexandre Loupy, Frank Martinez, Rébecca Sberro-Soussan, Anne Scemla, Claire Tinel, Julien Zuber ; Nancy : Pascal Eschwege, Luc Frimat, Sophie Girerd, Jacques Hubert, Marc Ladriere, Emmanuelle Laurain, Louis Leblanc, Pierre Lecoanet, Jean-Louis Lemelle ; Lyon E. Hériot : Lionel Badet, Maria Brunet, Fanny Buron, Rémi Cahen, Sameh Daoud, Coralie Fournie, Arnaud Grégoire, Alice Koenig, Charlène Lévi, Emmanuel Morelon, Claire Pouteil-Noble, Thomas Rimmelé, Olivier Thaunat ; Montpellier : Sylvie Delmas, Valérie Garrigue, Moglie Le Quintrec, Vincent Pernin, Jean-Emmanuel Serre. References Flores M, Glusman G, Brogaard K, Price ND, Hood L. P4 medicine: how systems medicine will transform the healthcare sector and society. Pers Med. 2013;10:565–76. Proust-Lima C, Blanche P. Dynamic Predictions. Wiley StatsRef: Statistics Reference Online. John Wiley & Sons, Ltd; 2016. pp. 1–6. Andrinopoulou E-R, Harhay MO, Ratcliffe SJ, Rizopoulos D. Reflection on modern methods: Dynamic prediction using joint models of longitudinal and time-to-event data. Int J Epidemiol. 2021;50:1731–43. Asgari S, Khalili D, Zayeri F, Azizi F, Hadaegh F. Dynamic prediction models improved the risk classification of type 2 diabetes compared with classical static models. J Clin Epidemiol. 2021;140:33–43. Rizopoulos D. Joint Models for Longitudinal and Time-to-Event Data: With Applications in R. CRC; 2012. Rizopoulos D. Dynamic predictions and prospective accuracy in joint models for longitudinal and time-to-event data. Biometrics. 2011;67:819–29. Barrett J, Su L. Dynamic predictions using flexible joint models of longitudinal and time-to-event data. Stat Med. 2017;36:1447–60. Li K, Luo S. Dynamic predictions in Bayesian functional joint models for longitudinal and time-to-event data: An application to Alzheimer’s disease. Stat Methods Med Res. 2019;28:327–42. Teramukai S, Okuda Y, Miyazaki S, Kawamori R, Shirayama M, Teramoto T. Dynamic prediction model and risk assessment chart for cardiovascular disease based on on-treatment blood pressure and baseline risk factors. Hypertens Res Off J Jpn Soc Hypertens. 2016;39:113–8. Fournier M-C, Dantan E, Blanche P. An R2 -curve for evaluating the accuracy of dynamic predictions. Stat Med. 2018;37:1125–33. Asar Ö, Fournier M-C, Dantan E. Dynamic predictions of kidney graft survival in the presence of longitudinal outliers. Stat Methods Med Res. 2021;30:185–203. Ben-Hassen C, Helmer C, Berr C, Jacqmin-Gadda H. Five-Year Dynamic Prediction of Dementia Using Repeated Measures of Cognitive Tests and a Dependency Scale. Am J Epidemiol. 2022;191:453–64. Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiol Camb Mass. 2010;21:128–38. Blanche P, Proust-Lima C, Loubère L, Berr C, Dartigues J-F, Jacqmin-Gadda H. Quantifying and comparing dynamic predictive accuracy of joint models for longitudinal marker and time-to-event in presence of censoring and competing risks. Biometrics. 2015;71:102–13. Kamarudin AN, Cox T, Kolamunnage-Dona R. Time-dependent ROC curve analysis in medical research: current methods and applications. BMC Med Res Methodol. 2017;17:53. Fournier M-C, Foucher Y, Blanche P, Legendre C, Girerd S, Ladrière M, et al. Dynamic predictions of long-term kidney graft failure: an information tool promoting patient-centred care. Nephrol Dial Transplant Off Publ Eur Dial Transpl Assoc -. Eur Ren Assoc. 2019;34:1961–9. Ferrer L, Putter H, Proust-Lima C. Individual dynamic predictions using landmarking and joint modelling: Validation of estimators and robustness assessment. Stat Methods Med Res. 2019;28:3649–66. Goldstein BA, Pomann GM, Winkelmayer WC, Pencina MJ. A comparison of risk prediction methods using repeated observations: An application to Electronic Health Records for Hemodialysis. Stat Med. 2017;36:2750–63. van Houwelingen HC, Putter H. Dynamic Prediction in Clinical Survival Analysis. CRC; 2012. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151:264–9. W64. Moons KGM, de Groot JAH, Bouwmeester W, Vergouwe Y, Mallett S, Altman DG, et al. Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist. PLoS Med. 2014;11:e1001744. Moons KGM, Altman DG, Reitsma JB, Ioannidis JPA, Macaskill P, Steyerberg EW, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162:W1–73. Steyerberg EW, Bleeker SE, Moll HA, Grobbee DE, Moons KGM. Internal and external validation of predictive models: a simulation study of bias and precision in small samples. J Clin Epidemiol. 2003;56:441–7. Devaux A, Genuer R, Peres K, Proust-Lima C. Individual dynamic prediction of clinical endpoint from large dimensional longitudinal biomarker history: a landmark approach. BMC Med Res Methodol. 2022;22:1–14. Yang T, Yang Y, Jia Y, Li X. Dynamic prediction of hospital admission with medical claim data. BMC Med Inf Decis Mak. 2019;19(Suppl 1):18. Lin J, Li K, Luo S. Functional survival forests for multivariate longitudinal outcomes: Dynamic prediction of Alzheimer’s disease progression. Stat Methods Med Res. 2021;30:99–111. Fahimi F, Guo Y, Tong SC, Ng A, Bing SOY, Choo B et al. A Vital Signs Telemonitoring Programme Improves the Dynamic Prediction of Readmission Risk in Patients with Heart Failure. AMIA Annu Symp Proc AMIA Symp. 2020;2020:432–41. Zeng Z, Tang X, Liu Y, He Z, Gong X. Interpretable recurrent neural network models for dynamic prediction of the extubation failure risk in patients with invasive mechanical ventilation in the intensive care unit. BioData Min. 2022;15:21. Howell M, Wong G, Rose J, Tong A, Craig JC, Howard K. Patient Preferences for Outcomes After Kidney Transplantation: A Best-Worst Scaling Survey. Transplantation. 2017;101:2765–73. Lenain R, Dantan E, Giral M, Foucher Y, Asar Ö, Naesens M, et al. External Validation of the DynPG for Kidney Transplant Recipients. Transplantation. 2021;105:396–403. Gordon EJ, Butt Z, Jensen SE, Lok-Ming Lehr A, Franklin J, Becker Y, et al. Opportunities for shared decision making in kidney transplantation. Am J Transpl Off J Am Soc Transpl Am Soc Transpl Surg. 2013;13:1149–58. Vandecasteele SJ, Kurella Tamura M. A patient-centered vision of care for ESRD: dialysis as a bridging treatment or as a final destination? J Am Soc Nephrol JASN. 2014;25:1647–51. Jörres A, John S, Lewington A, ter Wee PM, Vanholder R, Van Biesen W, et al. A European Renal Best Practice (ERBP) position statement on the Kidney Disease Improving Global Outcomes (KDIGO) Clinical Practice Guidelines on Acute Kidney Injury: part 2: renal replacement therapy. Nephrol Dial Transplant Off Publ Eur Dial Transpl Assoc -. Eur Ren Assoc. 2013;28:2940–5. Foucher Y, Meurette A, Daguin P, Bonnaud-Antignac A, Hardouin J-B, Chailan S, et al. A personalized follow-up of kidney transplant recipients using video conferencing based on a 1-year scoring system predictive of long term graft failure (TELEGRAFT study): protocol for a randomized controlled trial. BMC Nephrol. 2015;16:6. Evans JT, Evans JP, Walker RW, Blom AW, Whitehouse MR, Sayers A. How long does a hip replacement last? A systematic review and meta-analysis of case series and national registry reports with more than 15 years of follow-up. Lancet. 2019;393:647–54. Andrinopoulou E-R, Rizopoulos D, Takkenberg JJ, Lesaffre E. Combined dynamic predictions using joint models of two longitudinal outcomes and competing risk data. Stat Methods Med Res. 2017;26:1787–801. Suresh K, Taylor JMG, Spratt DE, Daignault S, Tsodikov A. Comparison of joint modeling and landmarking for dynamic prediction under an illness-death model. Biom J Biom Z. 2017;59:1277–300. Yang M, Luo S, DeSantis S. Bayesian quantile regression joint models: Inference and dynamic predictions. Stat Methods Med Res. 2019;28:2524–37. Desmée S, Mentré F, Veyrat-Follet C, Sébastien B, Guedj J. Nonlinear joint models for individual dynamic prediction of risk of death using Hamiltonian Monte Carlo: application to metastatic prostate cancer. BMC Med Res Methodol. 2017;17:105. Li N, Liu Y, Li S, Elashoff RM, Li G. A flexible joint model for multiple longitudinal biomarkers and a time-to-event outcome: With applications to dynamic prediction using highly correlated biomarkers. Biom J Biom Z. 2021;63:1575–86. Additional Declarations Competing interest reported. Lucas Chabeau and Pierre Rinder report financial support as employee of semeia. Other authors declare no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. Supplementary Files Supplementarymaterial.docx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-3938204","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":272512852,"identity":"10790b71-66a9-4453-bbab-19d75bc905a2","order_by":0,"name":"Lucas Chabeau","email":"","orcid":"","institution":"Nantes Université, Univ Tours, INSERM, MethodS in Patients-centered outcomes and HEalth Research, SPHERE","correspondingAuthor":false,"prefix":"","firstName":"Lucas","middleName":"","lastName":"Chabeau","suffix":""},{"id":272512854,"identity":"b908410b-f090-4032-b8f8-91bc8e20bb5d","order_by":1,"name":"Vincent Bonnemains","email":"","orcid":"","institution":"Nantes Université, Univ Tours, INSERM, MethodS in Patients-centered outcomes and HEalth Research, SPHERE","correspondingAuthor":false,"prefix":"","firstName":"Vincent","middleName":"","lastName":"Bonnemains","suffix":""},{"id":272512855,"identity":"c2c0cb57-4a86-446a-b6e5-b553cf4d9422","order_by":2,"name":"Pierre Rinder","email":"","orcid":"","institution":"SEMEIA","correspondingAuthor":false,"prefix":"","firstName":"Pierre","middleName":"","lastName":"Rinder","suffix":""},{"id":272512856,"identity":"c5df1faa-5dec-48dd-a787-8f48a2ed3721","order_by":3,"name":"Magali Giral","email":"","orcid":"","institution":"Nantes Université, CHU Nantes, INSERM, Center for Research in Transplantation and Translational Immunology, UMR 1064, ITUN, F-44000 Nantes, France","correspondingAuthor":false,"prefix":"","firstName":"Magali","middleName":"","lastName":"Giral","suffix":""},{"id":272512857,"identity":"febc3e06-00d2-41b5-8b5e-4405617a7be4","order_by":4,"name":"Solène Desmée","email":"","orcid":"","institution":"Univ Tours, Nantes Université, INSERM, MethodS in Patients-centered outcomes and HEalth Research, SPHERE, Tours, France","correspondingAuthor":false,"prefix":"","firstName":"Solène","middleName":"","lastName":"Desmée","suffix":""},{"id":272512858,"identity":"b34fe44c-c84e-40b0-86c5-d500fbf50b03","order_by":5,"name":"Etienne Dantan","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABKElEQVRIie3Pv0oDMRzA8YTAdclx6xWpvkJC19J7lQuFuhyim4NIQiBO6nriS3SzblcC3nJ660EdLIJTC4ogSPFP6tnikHZ2yBcOfpfwIQkALtc/LOKQ10P93wEA/dl+thDi/RLKwWLqr0hoPphayXLIfohe7awnO1I8Hgw7gOS3+cP7sGRXDUzQ/tH9cRBkUB5a3uJB2b4o+oAUe0ycFmN2LQ1Jb57CZhpDUVjforZ8pc3FEsp9NWYD7e3OsadDUoHGhFvJydxXX4CUUyo+1N2CxAh/6jAqMyjsRCFfZYBUCZVmMARlZsWcAuI1hElzsR6m6ZRetlSvXZMz3UwrJmwkkvnk1Vfd7ShIyMtMdVuDcsQRftNBcK5HNrIM2xbhBuByuVyuTX0DMoxpdLrNng8AAAAASUVORK5CYII=","orcid":"","institution":"Nantes Université, Univ Tours, INSERM, MethodS in Patients-centered outcomes and HEalth Research, SPHERE","correspondingAuthor":true,"prefix":"","firstName":"Etienne","middleName":"","lastName":"Dantan","suffix":""}],"badges":[],"createdAt":"2024-02-07 22:29:26","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-3938204/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-3938204/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":51135509,"identity":"6fdc5ef6-5f94-4776-9aea-c8fbb5374c59","added_by":"auto","created_at":"2024-02-14 18:30:01","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":119402,"visible":true,"origin":"","legend":"\u003cp\u003eScheme of dynamic predictions of patient and kidney graft survival until a final time horizon 𝑈 (red line) given a landmark time 𝑠 (Part A) and given a landmark time 𝑠′ (𝑠′\u0026gt;𝑠) (Part B), with longitudinal measures (blue crosses) and predicted marker evolution available before the landmark time (blue line).\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-3938204/v1/caf733f41d104cda6d1226e3.png"},{"id":51135506,"identity":"71eef457-e959-482b-8cbd-48bd321a50c0","added_by":"auto","created_at":"2024-02-14 18:30:00","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":122880,"visible":true,"origin":"","legend":"\u003cp\u003eScheme of dynamic predictions of patient and kidney graft survival for a fixed horizon window Δ𝑡 (red line) given landmark time 𝑠 (Part A) and given landmark time 𝑠′ (𝑠′\u0026gt;𝑠) (Part B), with longitudinal measures (blue crosses) and predicted marker evolution available before the landmark time (blue line).\u003c/p\u003e","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-3938204/v1/c7e420ff67792edc581ef7f8.png"},{"id":51135510,"identity":"79843a64-e24b-4c09-a096-78c21c52d429","added_by":"auto","created_at":"2024-02-14 18:30:01","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":64782,"visible":true,"origin":"","legend":"\u003cp\u003ePRISMA flow diagram, selection of included studies in the systematic review.\u003c/p\u003e","description":"","filename":"floatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-3938204/v1/5383395d9ef8959bfd06ea2c.png"},{"id":51135511,"identity":"951303a1-93a7-40c2-93dd-af9b1779a845","added_by":"auto","created_at":"2024-02-14 18:30:01","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":117892,"visible":true,"origin":"","legend":"\u003cp\u003ePrognostic capacities of the dynamic predictions obtained from the DIVAT internal validation sample (n=2,523, 66 observations deleted due to missing data concerning covariates) for landmark times varying from 1 to 6 years post-transplantation for a given 5-year horizon window; R² evaluated global performance (A) and the AUC appraised the discrimination accuracy (B). Estimations are drawn as solid lines and the corresponding 95% CIs are drawn as dashed lines.\u003c/p\u003e","description":"","filename":"floatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-3938204/v1/62f943551e88979562843df4.png"},{"id":51135508,"identity":"e4fb38dd-ed9e-476e-853a-43c1803e64b5","added_by":"auto","created_at":"2024-02-14 18:30:00","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":140904,"visible":true,"origin":"","legend":"\u003cp\u003ePrognostic capacities of the dynamic predictions obtained from the DIVAT internal validation sample (n=2,523, 66 observations deleted due to missing data concerning covariates) for landmark times varying from 1 to 10 years post-transplantation for a final time horizon of 11 years post-transplantation; R2 evaluated global performance (A) and the AUC appraised the discrimination accuracy (B). Estimations are drawn as solid lines and the corresponding 95% CIs are drawn as dashed lines.\u003c/p\u003e","description":"","filename":"floatimage5.png","url":"https://assets-eu.researchsquare.com/files/rs-3938204/v1/5f7fad69b1d87f66b835bef5.png"},{"id":62111070,"identity":"1e9fbc4f-86cd-45c1-b46f-87b6b911b42f","added_by":"auto","created_at":"2024-08-09 11:56:17","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1117672,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3938204/v1/8a8ff44c-dc23-49ff-8644-869b67aef20a.pdf"},{"id":51135507,"identity":"8b19536c-0520-4e7a-a43f-d2bfbf35a2bb","added_by":"auto","created_at":"2024-02-14 18:30:00","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":784796,"visible":true,"origin":"","legend":"","description":"","filename":"Supplementarymaterial.docx","url":"https://assets-eu.researchsquare.com/files/rs-3938204/v1/5930971aa33866ae145011f8.docx"}],"financialInterests":"Competing interest reported. Lucas Chabeau and Pierre Rinder report financial support as employee of semeia. Other authors declare no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.","formattedTitle":"On the importance of a clear definition of time horizon for time-to-event dynamic predictions: a systematic review and a concrete illustration in kidney transplantation","fulltext":[{"header":"1. Background","content":"\u003cp\u003eIn the personalized medicine era [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e], time-to-event dynamic predictions are becoming more widespread. They are defined as the probability to survive until a defined time horizon given being event-free at the time of making a prediction (landmark time) and given available predictive variables at such prediction times [\u003cspan additionalcitationids=\"CR3 CR4\" citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. Dynamic predictions enable to take into account the valuable information consisting of the entire marker trajectory known at landmark time and have shown their importance in improving time-fixed predictions [\u003cspan additionalcitationids=\"CR7\" citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]. The dynamic predictions are updated predictions whenever additional longitudinal data becomes available during the patient follow-up.\u003c/p\u003e \u003cp\u003eThe \u003cem\u003etime horizon\u003c/em\u003e deserves precisions in reporting since it can be defined in several ways [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e]. One objective could be to predict the survival probability until a \u003cem\u003efinal time horizon\u003c/em\u003e. For instance, Teramukai et al. were interested in dynamically predicting cardiovascular endpoints of hypertensive patients until 3 years after inclusion using repeated on-treatment blood pressure measurements [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. An alternative objective could be to predict the survival probability until the end of a \u003cem\u003esliding horizon window\u003c/em\u003e [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]. Ben-Hassen et al. proposed to dynamically predict dementia from longitudinal neurocognitive tests on a 5-year \u003cem\u003esliding horizon window\u003c/em\u003e [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. The literature on dynamic predictions is growing. But, to our knowledge, a clear definition of these two objectives does not exist yet and the comparison between the two approaches have not been investigated.\u003c/p\u003e \u003cp\u003eTo be useful, prognostic scores require good prognostic performances [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e]. They can be studied through global performances using time-dependent Brier Score or R\u0026sup2;-curve [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e], discrimination property using time-dependent AUC of ROC curves [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e, \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e] and calibration property using time-dependent calibration plots [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e]. Since the estimated dynamic predictions would not be the same given the definition of the \u003cem\u003etime horizon\u003c/em\u003e (\u003cem\u003efinal time horizon\u003c/em\u003e or \u003cem\u003esliding horizon window\u003c/em\u003e), this is of major importance when assessing the prognostic performances.\u003c/p\u003e \u003cp\u003eThe objective was to illustrate that the prognostic performances and their interpretations differ given the two \u003cem\u003etime horizon\u003c/em\u003e definitions. We presented the mathematical framework to obtain dynamic predictions according the two definitions (Section 2). We then conducted a systematic review of articles concerning dynamic predictions to objectively assess how the time horizon was reported in the literature (Section 3). We illustrated that prognostic performances of dynamic predictions of kidney graft failure differ between the time horizon definitions (Section 4). Finally, section 5 offers a discussion and conclusions.\u003c/p\u003e"},{"header":"2. Methods","content":"\u003cp\u003e\u003cspan\u003e\u003c/span\u003e\u003c/p\u003e\n\u003ch2\u003e\u003cem\u003e2.1 Notations\u003c/em\u003e\u003c/h2\u003e\n\u003cp\u003e\u003c/p\u003e\n\u003cp\u003eLet\u0026rsquo;s assume a learning sample of \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(n\\)\u003c/span\u003e\u003c/span\u003e independent and identically distributed patients with the observed data \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\left\\{{Y}_{i},{t}_{i}, {T}_{i},{\\delta }_{i},{A}_{i};i=1,\\dots , n\\right\\}\\)\u003c/span\u003e\u003c/span\u003e. Here, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({Y}_{i}=\\left\\{{Y}_{ij};j=1,\\dots ,{n}_{i}\\right\\}\\)\u003c/span\u003e\u003c/span\u003e corresponds to the set of \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({n}_{i}\\)\u003c/span\u003e\u003c/span\u003e longitudinal marker values measured at the corresponding times \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({t}_{i}=\\left\\{{t}_{ij};j=1,\\dots ,{n}_{i}\\right\\}\\)\u003c/span\u003e\u003c/span\u003e in individual \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(i\\)\u003c/span\u003e\u003c/span\u003e. Let\u0026rsquo;s consider \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({T}_{i}=\\text{min}\\left({T}_{i}^{*}, {C}_{i}\\right)\\)\u003c/span\u003e\u003c/span\u003e with \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({T}_{i}^{*}\\)\u003c/span\u003e\u003c/span\u003e the true time-to-event, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({C}_{i}\\)\u003c/span\u003e\u003c/span\u003e the censoring time for subject \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(i\\)\u003c/span\u003e\u003c/span\u003e, and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\delta }_{i}=1\\left\\{{T}_{i}^{*}\\le {C}_{i}\\right\\}\\)\u003c/span\u003e\u003c/span\u003e the event indicator function taking 1 if the event is not censored and 0 otherwise. We note \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({A}_{i}=\\left\\{{A}_{ik};k=1,\\dots ,K\\right\\}\\)\u003c/span\u003e\u003c/span\u003e the \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(K\\)\u003c/span\u003e\u003c/span\u003e baseline variables. From a learning sample of \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(n\\)\u003c/span\u003e\u003c/span\u003e patients, we can estimate a prediction model of parameters \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\theta\\)\u003c/span\u003e\u003c/span\u003e. Among those, landmarking and joint modeling of longitudinal and survival data are popular approaches [\u003cspan class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e17\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e18\u003c/span\u003e].\u003c/p\u003e\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e\u003ch2\u003e2.2. Dynamic predictions definitions\u003c/h2\u003e\u003cp\u003eThe dynamic prediction \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\pi }_{l}\\)\u003c/span\u003e\u003c/span\u003e for a patient \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(l\\)\u003c/span\u003e\u003c/span\u003e is defined as the probability of being event-free until a \u003cem\u003etime horizon\u003c/em\u003e \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(u\\)\u003c/span\u003e\u003c/span\u003e given being event-free at the landmark time \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(s \\left(s\u0026lt;u\\right)\\)\u003c/span\u003e\u003c/span\u003e, given baseline variables \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({A}_{l}\\)\u003c/span\u003e\u003c/span\u003e and the longitudinal marker history available at time \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(s\\)\u003c/span\u003e\u003c/span\u003e (i.e. \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\stackrel{\\sim}{Y}}_{l}\\left(s\\right)=\\left\\{{Y}_{l}\\left({s}_{1}\\right),\\dots ,{Y}_{l}\\left({s}_{p}\\right);0\\le {s}_{1}\u0026lt;\\dots \u0026lt;{s}_{p}\u0026lt;s\\right\\}\\)\u003c/span\u003e\u003c/span\u003e):\u003c/p\u003e\u003cp\u003e\u003cspan class=\"InlineEquation\"\u003e\u0026nbsp;\u003cspan class=\"mathinline\"\u003e\\({\\pi }_{l}\\left(s,u;\\theta \\right)=\\text{Pr}\\left({T}_{l}^{*}\u0026gt;u|{T}_{l}^{*}\u0026gt;s,{A}_{l},{\\stackrel{\\sim}{Y}}_{l}\\left(s\\right);\\theta \\right)\\)\u003c/span\u003e\u0026nbsp;\u003c/span\u003e \u003cspan class=\"InlineEquation\"\u003e\u0026nbsp;\u003cspan class=\"mathinline\"\u003e\\(\\left(3\\right)\\)\u003c/span\u003e\u0026nbsp;\u003c/span\u003e\u003c/p\u003e\n\u003cp\u003eWe report two different uses of such generic definition of dynamic predictions [\u003cspan class=\"CitationRef\"\u003e2\u003c/span\u003e\u0026ndash;\u003cspan class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e7\u003c/span\u003e], that depend on the definition of \u003cem\u003etime horizon\u003c/em\u003e \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(u\\)\u003c/span\u003e\u003c/span\u003e and that would not have the same clinical objective.\u003c/p\u003e\n\u003cp\u003e\u003cspan type=\"ItalicUnderline\" class=\"ItalicUnderline\" name=\"Emphasis\"\u003eDynamic predictions given a final time horizon.\u003c/span\u003e The objective would be to dynamically predict the survival probability \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\pi }_{l}\\)\u003c/span\u003e\u003c/span\u003e for a patient\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(l\\)\u003c/span\u003e\u003c/span\u003e until a fixed \u003cem\u003efinal time horizon\u003c/em\u003e \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(U\\)\u003c/span\u003e\u003c/span\u003e (Fig. \u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e). It corresponds to the probability to not suffer the event on a horizon window defined as the delay between the landmark time\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(s\\)\u003c/span\u003e\u003c/span\u003e and the \u003cem\u003efinal time horizon\u003c/em\u003e \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(U\\)\u003c/span\u003e\u003c/span\u003e:\u003c/p\u003e\n\u003cdiv id=\"Equa\" class=\"Equation\"\u003e\n \u003cdiv class=\"mathdisplay\" id=\"FileID_Equa\" name=\"EquationSource\"\u003e$${\\pi }_{l}\\left(s,U;\\theta \\right)=\\text{Pr}\\left({T}_{l}^{*}\u0026gt;U|{T}_{l}^{*}\u0026gt;s,{A}_{l},{\\stackrel{\\sim}{Y}}_{l}(s);\\theta \\right)$$\u003c/div\u003e\n\u003c/div\u003e\n\u003cp\u003eUsing the estimated parameters \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\widehat{\\theta }\\)\u003c/span\u003e\u003c/span\u003e of the prediction model, the dynamic predictions considering a \u003cem\u003efinal time horizon\u003c/em\u003e can be estimated as the ratio between the survival probability until the \u003cem\u003efinal time horizon\u003c/em\u003e\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(U\\)\u003c/span\u003e\u003c/span\u003e and the survival probability until the time of making prediction \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(s\\)\u003c/span\u003e\u003c/span\u003e:\u003c/p\u003e\n\u003cdiv id=\"Equb\" class=\"Equation\"\u003e\n \u003cdiv class=\"mathdisplay\" id=\"FileID_Equb\" name=\"EquationSource\"\u003e$${\\widehat{\\pi }}_{l}\\left(s,U;\\widehat{\\theta }\\right) = \\frac{Pr\\left({T}_{l}^{*}\u0026gt;U|{A}_{l},{\\stackrel{\\sim}{Y}}_{l}\\left(s\\right);\\widehat{\\theta }\\right)}{Pr\\left({T}_{l}^{*}\u0026gt;s|{A}_{l},{\\stackrel{\\sim}{Y}}_{l}\\left(s\\right);\\widehat{\\theta }\\right)}$$\u003c/div\u003e\n\u003c/div\u003e\n\u003cp\u003eFollowing this, the horizon window \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\left\\{U-s\\right\\}\\)\u003c/span\u003e\u003c/span\u003e is progressively reduced when landmark time \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(s\\)\u003c/span\u003e\u003c/span\u003e increases (Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e). We note that for a fixed \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\stackrel{\\sim}{Y}}_{l}\\left(s\\right)\\)\u003c/span\u003e\u003c/span\u003e, the smaller the difference between \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(s\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(U\\)\u003c/span\u003e\u003c/span\u003e, the bigger \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\widehat{\\pi }}_{l}\\left(s,U\\right)\\)\u003c/span\u003e\u003c/span\u003e will be. Indeed, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\pi }_{l}\\left(s,U;\\theta \\right)\\)\u003c/span\u003e\u003c/span\u003e is bounded with:\u003c/p\u003e\n\u003cdiv id=\"Equc\" class=\"Equation\"\u003e\n \u003cdiv class=\"mathdisplay\" id=\"FileID_Equc\" name=\"EquationSource\"\u003e$$\\underset{s\\to 0}{\\text{lim}}{\\widehat{\\pi }}_{l}\\left(s,U;\\widehat{\\theta }\\right) = \\frac{Pr\\left({T}_{l}^{*}\u0026gt;U|{A}_{l},{\\stackrel{\\sim}{Y}}_{l}(0);\\widehat{\\theta }\\right)}{1} = Pr\\left({T}_{l}^{*}\u0026gt;U|{A}_{l},{\\stackrel{\\sim}{Y}}_{l}(0);\\widehat{\\theta }\\right)$$\u003c/div\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Equd\" class=\"Equation\"\u003e\n \u003cdiv class=\"mathdisplay\" id=\"FileID_Equd\" name=\"EquationSource\"\u003e$$\\underset{s\\to U}{\\text{lim}}{\\widehat{\\pi }}_{l}\\left(s,U;\\widehat{\\theta }\\right) = \\frac{Pr\\left({T}_{l}^{*}\u0026gt;U|{A}_{l},{\\stackrel{\\sim}{Y}}_{l}(U);\\widehat{\\theta }\\right)}{Pr\\left({T}_{l}^{*}\u0026gt;U|{A}_{l},{\\stackrel{\\sim}{Y}}_{l}(U);\\widehat{\\theta }\\right)} = 1$$\u003c/div\u003e\n\u003c/div\u003e\n\u003cp\u003e\u003cspan type=\"ItalicUnderline\" class=\"ItalicUnderline\" name=\"Emphasis\"\u003eDynamic predictions given a sliding horizon window.\u003c/span\u003e Some authors have proposed an alternative definition of the dynamic predictions [\u003cspan class=\"CitationRef\"\u003e2\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e11\u003c/span\u003e]. The target of inference of a patient \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(l\\)\u003c/span\u003e\u003c/span\u003e is to predict the survival probability until the end of a \u003cem\u003ehorizon window\u003c/em\u003e of length \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\varDelta t\\)\u003c/span\u003e\u003c/span\u003e:\u003c/p\u003e\n\u003cdiv id=\"Eque\" class=\"Equation\"\u003e\n \u003cdiv class=\"mathdisplay\" id=\"FileID_Eque\" name=\"EquationSource\"\u003e$${\\pi }_{l}\\left(s,s+\\varDelta t;\\theta \\right)=\\text{Pr}\\left({T}_{l}^{*}\u0026gt;s+\\varDelta t|{T}_{l}^{*}\u0026gt;s,{A}_{l},{\\stackrel{\\sim}{Y}}_{l}(s);\\theta \\right)$$\u003c/div\u003e\n\u003c/div\u003e\n\u003cp\u003eFollowing this definition, the \u003cem\u003ehorizon window\u003c/em\u003e \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\varDelta t\\)\u003c/span\u003e\u003c/span\u003e is thus sliding when updating the prediction, i.e. the landmark time increases (Fig. \u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e). Using the estimated parameters \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\widehat{\\theta }\\)\u003c/span\u003e\u003c/span\u003e of the prediction model, the dynamic predictions given a \u003cem\u003esliding horizon window\u003c/em\u003e can be decomposed as the ratio between the survival probability until the end of the \u003cem\u003ehorizon window\u003c/em\u003e \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(s+\\varDelta t\\)\u003c/span\u003e\u003c/span\u003e and the survival probability until the time of making prediction \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(s\\)\u003c/span\u003e\u003c/span\u003e:\u003c/p\u003e\n\u003cdiv id=\"Equf\" class=\"Equation\"\u003e\n \u003cdiv class=\"mathdisplay\" id=\"FileID_Equf\" name=\"EquationSource\"\u003e$${\\widehat{\\pi }}_{l}\\left(s,s+\\varDelta t;\\widehat{\\theta }\\right) = \\frac{Pr\\left({T}_{l}^{*}\u0026gt;s+\\varDelta t|{A}_{l},{\\stackrel{\\sim}{Y}}_{l}\\left(s\\right);\\widehat{\\theta }\\right)}{Pr\\left({T}_{l}^{*}\u0026gt;s|{A}_{l},{\\stackrel{\\sim}{Y}}_{l}\\left(s\\right);\\widehat{\\theta }\\right)}$$\u003c/div\u003e\n\u003c/div\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec4\" class=\"Section2\"\u003e\n \u003ch2\u003e2.3. Accuracy measures for dynamic predictions\u003c/h2\u003e\n \u003cp\u003eFollowing these two definitions, the \u003cem\u003ehorizon window\u003c/em\u003e is either reduced or sliding as the landmark time increases. Therefore, the at-risk population at landmark times would be identical given the two definitions, but the incidence rate on the prediction window would not. Consequently, the prognostic performances and the corresponding interpretation will differ.\u003c/p\u003e\n \u003cp\u003eThe global prognostic performance of dynamic predictions can be assessed by measuring the prediction error [\u003cspan class=\"CitationRef\"\u003e19\u003c/span\u003e] with the popular Brier Score metric, for instance. The time-dependent expected Brier Score is defined as follows [\u003cspan class=\"CitationRef\"\u003e14\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e17\u003c/span\u003e]:\u003c/p\u003e\n \u003cdiv id=\"Equg\" class=\"Equation\"\u003e\n \u003cdiv class=\"mathdisplay\" id=\"FileID_Equg\" name=\"EquationSource\"\u003e$$BS\\left(s,u\\right)=E\\left[\\left(1\\left({T}^{*}\u0026gt;u\\right)-\\pi \\left(s,u\\right)\\right)\u0026sup2;|{T}^{*}\u0026gt;s\\right]$$\u003c/div\u003e\n \u003c/div\u003e\n \u003cp\u003eThe Brier Score is a mean square error term. The closer the prediction is to the observation, the closer the Brier Score is to 0. When considering the \u003cem\u003efinal time horizon\u003c/em\u003e, it means that the survival probability until a \u003cem\u003efinal time horizon\u003c/em\u003e is expected to be close to the observed event indicator after the final \u003cem\u003etime horizon\u003c/em\u003e \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\left(BS\\left(s,U\\right)=E\\left[\\left(1\\left({T}^{*}\u0026gt;U\\right)-\\pi \\left(s,U\\right)\\right)\u0026sup2;|{T}^{*}\u0026gt;s\\right]\\right)\\)\u003c/span\u003e\u003c/span\u003e. When considering the \u003cem\u003esliding horizon window\u003c/em\u003e, it means that the survival probability after an \u003cem\u003ehorizon window\u003c/em\u003e \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\varDelta t\\)\u003c/span\u003e\u003c/span\u003e is expected to be close to the observed event indicator after this \u003cem\u003ehorizon window\u003c/em\u003e \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\left(BS\\left(s,s+\\varDelta t\\right)=E\\left[\\left(1\\left({T}^{*}\u0026gt;s+\\varDelta t\\right)-\\pi \\left(s,s+\\varDelta t\\right)\\right)\u0026sup2;|{T}^{*}\u0026gt;s\\right]\\right)\\)\u003c/span\u003e\u003c/span\u003e. In a dynamic context, one limit of the Brier Score is that it is sensitive to the marginal event probability that could change as the landmark time increases. van Houwelingen et al. proposed studying the relative error reduction [\u003cspan class=\"CitationRef\"\u003e19\u003c/span\u003e], also named R\u0026sup2;-curve [\u003cspan class=\"CitationRef\"\u003e10\u003c/span\u003e]:\u003c/p\u003e\n \u003cdiv id=\"Equh\" class=\"Equation\"\u003e\n \u003cdiv class=\"mathdisplay\" id=\"FileID_Equh\" name=\"EquationSource\"\u003e$${R}^{2}\\left(s,u\\right)=1-\\frac{BS\\left(s,u\\right)}{B{S}_{0}\\left(s,u\\right)}$$\u003c/div\u003e\n \u003c/div\u003e\n \u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(BS\\left(s,u\\right)\\)\u003c/span\u003e\u003c/span\u003e is the Brier Score obtained using the model of interest and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(B{S}_{0}\\left(s,u\\right)\\)\u003c/span\u003e\u003c/span\u003e the Brier Score of a reference model (Kaplan-Meier estimator for instance) allowing to assess the global performance of dynamic predictions whatever the evolution of the marginal event probability along the landmark times. Since the Brier Score computation would be impacted by the consideration of the \u003cem\u003efinal time horizon\u003c/em\u003e or the \u003cem\u003esliding horizon window\u003c/em\u003e, the R\u0026sup2;-curve would also not be the same between the two approaches.\u003c/p\u003e\n \u003cp\u003eThe discrimination is the ability of a prognostic tool to order the risk between subjects. To assess discrimination, the well-known Area Under the Receiver Operating Characteristics Curve (AUC) [\u003cspan class=\"CitationRef\"\u003e14\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e15\u003c/span\u003e] can be explicitly defined in a dynamic context as:\u003c/p\u003e\n \u003cdiv id=\"Equi\" class=\"Equation\"\u003e\n \u003cdiv class=\"mathdisplay\" id=\"FileID_Equi\" name=\"EquationSource\"\u003e$$AUC\\left(s,u\\right)=\\text{Pr}\\left({\\pi }_{l}\\left(s,u\\right)\u0026gt;{\\pi }_{{l}^{{\\prime }}}\\left(s,u\\right)|{T}_{l}^{*}\u0026gt;u,{s\u0026lt;T}_{{l}^{{\\prime }}}^{*}\u0026lt;u\\right)$$\u003c/div\u003e\n \u003c/div\u003e\n \u003cp\u003eWhen considering the \u003cem\u003efinal time horizon\u003c/em\u003e, the AUC corresponds to the probability that a subject \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(l\\)\u003c/span\u003e\u003c/span\u003e at-risk at landmark time\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(s\\)\u003c/span\u003e\u003c/span\u003e and who would not suffer the event before the \u003cem\u003efinal time horizon\u003c/em\u003e \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(U\\)\u003c/span\u003e\u003c/span\u003e would have a higher survival prediction than a subject \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({l}^{{\\prime }}\\)\u003c/span\u003e\u003c/span\u003e at-risk at landmark time\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(s\\)\u003c/span\u003e\u003c/span\u003e and who would suffer the event before the \u003cem\u003efinal time horizon\u003c/em\u003e \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(U\\)\u003c/span\u003e\u003c/span\u003e \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\left(AUC\\left(s,U\\right)=\\text{Pr}\\left({\\pi }_{l}\\left(s,U\\right)\u0026gt;{\\pi }_{{l}^{{\\prime }}}\\left(s,U\\right)|{T}_{l}^{*}\u0026gt;U,{s\u0026lt;T}_{{l}^{{\\prime }}}^{*}\u0026lt;U\\right)\\right)\\)\u003c/span\u003e\u003c/span\u003e. When considering the \u003cem\u003esliding horizon window\u003c/em\u003e, the AUC corresponds to the probability that a subject \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(l\\)\u003c/span\u003e\u003c/span\u003e at-risk at landmark time\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(s\\)\u003c/span\u003e\u003c/span\u003e and who would not suffer the event before the end of the \u003cem\u003ehorizon window\u003c/em\u003e \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(s+\\varDelta t\\)\u003c/span\u003e\u003c/span\u003e would have a higher survival prediction than a subject \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({l}^{{\\prime }}\\)\u003c/span\u003e\u003c/span\u003e at-risk at landmark time\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(s\\)\u003c/span\u003e\u003c/span\u003e and who would suffer the event between \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(s\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(s+\\varDelta t\\)\u003c/span\u003e\u003c/span\u003e \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\left(AUC\\left(s,s+\\varDelta t\\right)=\\text{Pr}\\left({\\pi }_{l}\\left(s,s+\\varDelta t\\right)\u0026gt;{\\pi }_{{l}^{{\\prime }}}\\left(s,s+\\varDelta t\\right)|{T}_{l}^{*}\u0026gt;s+\\varDelta t,{s\u0026lt;T}_{{l}^{{\\prime }}}^{*}\u0026lt;s+\\varDelta t\\right)\\right)\\)\u003c/span\u003e\u003c/span\u003e.\u003c/p\u003e\n \u003cp\u003eThe calibration property assesses the ability of a prognostic tool to provide a prediction close to the observed outcome. Usually assessed through calibration plots, the calibration is described by comparing the predicted values within subgroups (defined from quantiles of predictions) to the observed survival probabilities (computed using the Kaplan\u0026ndash;Meier estimator for instance). When considering the \u003cem\u003efinal time horizon\u003c/em\u003e, a good calibration means that, for any given \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(x\\)\u003c/span\u003e\u003c/span\u003e value, we expect that among all subjects who are predicted to survive with a \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(x\\)\u003c/span\u003e\u003c/span\u003e % probability, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(x\\)\u003c/span\u003e\u003c/span\u003e out of 100 will not experience the event before the \u003cem\u003efinal time horizon\u003c/em\u003e. When considering the \u003cem\u003esliding horizon window\u003c/em\u003e, a good calibration means that, for any given \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(x\\)\u003c/span\u003e\u003c/span\u003e value, we expect that among all subjects who are predicted to survive with a \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(x\\)\u003c/span\u003e\u003c/span\u003e % probability, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(x\\)\u003c/span\u003e\u003c/span\u003e out of 100 will not experience the event before the end of the \u003cem\u003ehorizon window\u003c/em\u003e.\u003c/p\u003e\n\u003c/div\u003e"},{"header":"3. Systematic review of dynamic predictions","content":"\u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003e3.1. Search strategy\u003c/h2\u003e \u003cp\u003eWe conducted a systematic review following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement - PRISMA (Web supplementary materials Table \u003cspan refid=\"MOESM1\" class=\"InternalRef\"\u003eS1\u003c/span\u003e) [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e]. We searched the Medline database on 25th August 2022, without date or language restriction. The search equation used is reported in Appendix 1 of Web supplementary materials.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003e3.2. Study selection\u003c/h2\u003e \u003cp\u003e As this systematic review questioned the definition and the choice of time horizon, we included all articles focusing on the development or the validation of individual dynamic prediction scoring systems as well as methodological papers concerning dynamic predictions. Two reviewers (LC - ED) independently screened references by title and abstract. All eligible texts were assessed independently by the two reviewers (LC - ED) and discussions resolved any discrepancies. Our non-inclusion criteria were reviews or meta-analyses, full texts not found or conference abstracts and editorials.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003e3.3. Data extraction\u003c/h2\u003e \u003cp\u003eEach study was randomly allocated to two of the five reviewers (LC \u0026ndash; VB \u0026ndash; PR \u0026ndash; SD \u0026ndash; ED). Following the Checklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e], we predefined a standardized form for data extraction. Whether it was a methodological article or an article concerning the development or the validation of dynamic predictions, for each eligible article, we collected the author\u0026rsquo;s name, year and journal, the data source, the study design, the inclusion and exclusion criteria, the sample size, the population characteristics, the predicted outcomes, the prediction times (i.e. the time when one calculates the prediction), the time horizon (i.e. the end of the prediction time window), the predictive tools and the development details (such as the methodology used, the variables of the scoring systems). For each article, the time horizon was qualified as either a \u003cem\u003efinal time horizon\u003c/em\u003e or a \u003cem\u003esliding horizon window\u003c/em\u003e from an explicit equation, explicit texts and/or a clear graphic representing individual dynamic prediction, or qualified as unclear information in case of lack of such explicit information or contradictory information. The TRIPOD statement recommends to develop a prognostic score from a learning sample and to validate the prognostic performances from an independent internal and/or external validation sample to avoid reporting the prognostic performances on the learning sample only, which would lead to overestimating the performances [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e, \u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e]. We thus extracted information on internal and external validations as well. Whenever necessary, any discrepancies in data extraction were resolved by discussion with another reviewer to reach a consensus.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec9\" class=\"Section2\"\u003e \u003ch2\u003e3.4. Results\u003c/h2\u003e \u003cp\u003eThe search identified 172 unique articles. Screening of titles and abstracts identified 135 papers eligible for full-text review. As detailed on the flow diagram (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e), 33 articles were excluded. Finally, we included 102 articles (Appendix 2 of Web supplementary materials).\u003c/p\u003e \u003cp\u003e\u003c/p\u003e \u003cp\u003e Dynamic predictions appear to be a relatively recent research topic in biostatistics with 100 (98%) articles published after 2010. Fifty-one (50%) of the retained articles are methodological studies that proposed new modeling approach. This systematic review confirmed the dominance of joint modeling for longitudinal and survival data (n\u0026thinsp;=\u0026thinsp;42, 41%) and landmarking (n\u0026thinsp;=\u0026thinsp;44, 43%) to develop dynamic predictions. To a lesser extent, we identified the emergence of machine learning approaches including deep learning (n\u0026thinsp;=\u0026thinsp;14, 14%), with random survival forest [\u003cspan additionalcitationids=\"CR25\" citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e] or neural networks [\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e, \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e] for instance. All articles referring to machine learning approaches were published after 2018. This will probably increase in the future with the rapid development of such modeling approaches. Among the applied articles (n\u0026thinsp;=\u0026thinsp;51, 50%), we also observed that 12 studies (24%) only presented a development modeling of dynamic prediction without validation of their predictive performance on an independent data set (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). This does not respect the TRIPOD recommendations about the right process to develop and validate prognostic performances. Nevertheless, 28 articles included internal validation, 5 an external validation and 5 both types of validation, leading to 76% of the applied articles in agreement with the TRIPOD.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eDescription of the articles included in the systematic review\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAll included articles\u003c/p\u003e \u003cp\u003eN\u0026thinsp;=\u0026thinsp;102\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eMethodological articles\u003c/p\u003e \u003cp\u003eN\u0026thinsp;=\u0026thinsp;51\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eApplied articles\u003c/p\u003e \u003cp\u003eN\u0026thinsp;=\u0026thinsp;51\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"1\" nameend=\"c5\" namest=\"c5\"\u003e\u0026nbsp;\u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTime prediction horizon definition\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colspan=\"1\" nameend=\"c5\" namest=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFinal time horizon\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e18 (18%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e5 (10%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e13 (25%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"1\" nameend=\"c5\" namest=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSliding horizon window\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e71 (69%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e44 (86%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e27 (53%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"1\" nameend=\"c5\" namest=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eUnclear\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e13 (13%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e2 (4%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e11 (22%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"1\" nameend=\"c5\" namest=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePublished after 2010\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e100 (98%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e51 (100%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e49 (96%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"1\" nameend=\"c5\" namest=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMain statistical analyses for model development\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colspan=\"1\" nameend=\"c5\" namest=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eJoint model\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e42 (41%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e26 (51%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e16 (32%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"1\" nameend=\"c5\" namest=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLandmarking\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e44 (43%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e18 (35%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e26 (52%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"1\" nameend=\"c5\" namest=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMachine Learning\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e14 (14%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e6 (12%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e8 (16%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"1\" nameend=\"c5\" namest=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOthers\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e9 (9%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e5 (10%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e4 (8%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"1\" nameend=\"c5\" namest=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGlobal performance\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e46 (45%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e32 (74%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e14 (28%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"1\" nameend=\"c5\" namest=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBrier Score\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e28 (61%\u003csup\u003ea\u003c/sup\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e21 (66%\u003csup\u003ea\u003c/sup\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e7 (%\u003csup\u003ea\u003c/sup\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"1\" nameend=\"c5\" namest=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePrediction error\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e20 (43%\u003csup\u003ea\u003c/sup\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e12 (38%\u003csup\u003ea\u003c/sup\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e8 (57%\u003csup\u003ea\u003c/sup\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"1\" nameend=\"c5\" namest=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOthers\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e1 (2%\u003csup\u003ea\u003c/sup\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e1 (3%\u003csup\u003ea\u003c/sup\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0 (0%\u003csup\u003ea\u003c/sup\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"1\" nameend=\"c5\" namest=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDiscrimination\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e84 (82%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e34 (79%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e50 (100%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"1\" nameend=\"c5\" namest=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAUC\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e65 (77%\u003csup\u003ea\u003c/sup\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e31 (91%\u003csup\u003ea\u003c/sup\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e34 (68%\u003csup\u003ea\u003c/sup\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"1\" nameend=\"c5\" namest=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eC-index\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e18 (21%\u003csup\u003ea\u003c/sup\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e3 (9%\u003csup\u003ea\u003c/sup\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e15 (30%\u003csup\u003ea\u003c/sup\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"1\" nameend=\"c5\" namest=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOthers\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e6 (7%\u003csup\u003ea\u003c/sup\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0 (0%\u003csup\u003ea\u003c/sup\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0 (0%\u003csup\u003ea\u003c/sup\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"1\" nameend=\"c5\" namest=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCalibration\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e24 (24%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e6 (14%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e18 (36%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"1\" nameend=\"c5\" namest=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCalibration plot\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e19 (79%\u003csup\u003ea\u003c/sup\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e5 (83%\u003csup\u003ea\u003c/sup\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e14 (78%\u003csup\u003ea\u003c/sup\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"1\" nameend=\"c5\" namest=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCalibration slope\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2 (8%\u003csup\u003ea\u003c/sup\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0 (0%\u003csup\u003ea\u003c/sup\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e2 (11%\u003csup\u003ea\u003c/sup\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"1\" nameend=\"c5\" namest=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHeuristic shrinkage factor\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e5 (21%\u003csup\u003ea\u003c/sup\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0 (0%\u003csup\u003ea\u003c/sup\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e5 (28%\u003csup\u003ea\u003c/sup\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"1\" nameend=\"c5\" namest=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOthers\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2 (8%\u003csup\u003ea\u003c/sup\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e1 (17%\u003csup\u003ea\u003c/sup\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e1 (6%\u003csup\u003ea\u003c/sup\u003e)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"1\" nameend=\"c5\" namest=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eType of study\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colspan=\"1\" nameend=\"c5\" namest=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDevelopment alone\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e12 (24%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"1\" nameend=\"c5\" namest=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDevelopment \u0026amp; Internal validation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e28 (55%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"1\" nameend=\"c5\" namest=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDevelopment \u0026amp; External validation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e5 (10%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"1\" nameend=\"c5\" namest=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDevelopment, Internal \u0026amp; External validation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e5 (10%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"1\" nameend=\"c5\" namest=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eExternal validation alone\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e-\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e1 (1%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"1\" nameend=\"c5\" namest=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"5\" nameend=\"c5\" namest=\"c1\"\u003e \u003cp\u003e\u003csup\u003ea\u003c/sup\u003e Percentage among the count of articles reporting the prognostic ability\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eAmong the included articles, we identified 18 (17.7%) articles defining dynamic predictions using the \u003cem\u003efinal time horizon\u003c/em\u003e, 71 (69.6%) articles using \u003cem\u003esliding horizon windows\u003c/em\u003e and 13 (12.7%) articles that were unclear about the time horizon definition (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). Methodological articles mainly used a \u003cem\u003esliding horizon window\u003c/em\u003e approach to consider dynamic predictions (n\u0026thinsp;=\u0026thinsp;44, 86%), and to a lesser extent \u003cem\u003efinal time horizon\u003c/em\u003e (n\u0026thinsp;=\u0026thinsp;5, 10%) while only 2 (4%) articles did not clearly define the considered approach. This repartition was more heterogeneous when studying the applied articles: 27 (53%) used \u003cem\u003esliding horizon windows\u003c/em\u003e, 13 (25%) used the \u003cem\u003efinal time horizon\u003c/em\u003e and 11 (22%) appeared unclear as to which approach was used.\u003c/p\u003e \u003cp\u003eBeyond the description of the time horizon considered, our systematic review allows to appraise the reporting quality on dynamic predictions. As clearly recommended in the literature [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e], assessing prognostic performances is of major importance for prediction tools to propose useful scores. Discrimination capacity is well reported (n\u0026thinsp;=\u0026thinsp;84, 82%) with AUC as the principal performance indicator. Global performances (n\u0026thinsp;=\u0026thinsp;46, 45%) with a Brier Score are less often provided, as are calibration (n\u0026thinsp;=\u0026thinsp;24, 24%) with a calibration plot. Details of such prognostic performances are also not homogeneous between methodological and applied articles (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e \u003c/div\u003e"},{"header":"4. Application","content":"\u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003e4.1. Context\u003c/h2\u003e \u003cp\u003eIn kidney transplantation, patients are particularly interested by their kidney graft survival, before any risk of adverse outcomes or infections [\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e]. In this context, we developed and internally and externally validated \u0026lsquo;Dynamic predictions of Patient and kidney Graft survival\u0026rsquo; (DynPG) for kidney recipients alive with a functioning graft at 1-year post-transplantation [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e, \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e]. The main outcome was the delay from 1-year post-transplantation to patient and kidney graft failure defined as the first event between return-to-dialysis, pre-emptive retransplantation and death with a functioning graft. Such dynamic predictions can be obtained from six baseline variables: recipient age, graft rank, cardiovascular histories, pretransplantation anti-HLA class I immunization, serum creatinine at 3-months post-transplantation, occurrence of acute rejection in the first year post-transplantation, and also the complete longitudinal serum creatinine trajectory available at the time of prediction. Our aim was to provide to patients and physicians an information on the possible future on a \u003cem\u003esliding horizon window\u003c/em\u003e of 5 years. In line with this clinical objective, the DynPG was defined as the probability of being graft failure-free over the next 5 years after the landmark time, for each prediction time from 1 to 6 years post-transplantation. We retained 6 years post-transplantation as the maximal landmark time since there were 178 patients still at risk of patient and kidney graft failure at 11 years post-transplantation in the validation sample.\u003c/p\u003e \u003cp\u003eWe aimed to illustrate that the prognostic performances would not be the same under the assumptions of a \u003cem\u003esliding horizon window\u003c/em\u003e or a \u003cem\u003efinal time horizon.\u003c/em\u003e While a \u003cem\u003esliding horizon window\u003c/em\u003e may be relevant to monitor the patient risk along his/her follow-up with always the same \u003cem\u003ehorizon window\u003c/em\u003e of 5 years, the clinical objective is different under the prism of a \u003cem\u003efinal time horizon\u003c/em\u003e. Considering the \u003cem\u003efinal time horizon\u003c/em\u003e of 11 years post-transplantation, the dynamic predictions aim to predict with good confidence until this \u003cem\u003efinal time horizon\u003c/em\u003e. Benefiting from the previously estimated joint model and the same internal validation sample, we compared the prognostic performances of the DynPG under the assumption of a 5-year \u003cem\u003esliding horizon window\u003c/em\u003e and under the assumption of a 11 years \u003cem\u003efinal time horizon\u003c/em\u003e.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003e4.2. Study population\u003c/h2\u003e \u003cp\u003eData were extracted from the French multicentric observational and prospective DIVAT cohort (Donn\u0026eacute;es Informatis\u0026eacute;es et VAlid\u0026eacute;es en Transplantation; \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e\u003ca href=\"http://www.divat.fr\" target=\"_blank\"\u003ewww.divat.fr\u003c/a\u003e\u003c/span\u003e\u003cspan address=\"http://www.divat.fr\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e, CNIL no. 914184, ClinicalTrials.gov recording NCT02900040). All participants gave informed consent. The inclusion criteria were adult recipients who received a first or second renal graft transplanted after January 2000 from a living or heart-beating deceased donor, alive with a functioning graft at 1 year post-transplantation and maintained under tacrolimus and mycophenolate. The extracted DIVAT cohort data consisted of a learning set of 2,749 patients, initially used to estimate the shared-random joint model, and an internal validation set of 2,589 patients. Due to missing data for retained predictors, 66 patients were excluded from the analysis of the validation set (n\u0026thinsp;=\u0026thinsp;2,523). More details on this validation sample are reported in Fournier et al. [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e].\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003e4.3. Results\u003c/h2\u003e \u003cp\u003eUnder the 5-year \u003cem\u003esliding horizon window\u003c/em\u003e setting, the results were as previously published [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e]. The global prognostic performance of the DynPG appeared relatively stable along the landmark times with R\u0026sup2; values ranging from 14% (95% CI 7\u0026ndash;21%) to 15% (95% CI -2\u0026ndash;33%) at 1 and 6 years post-transplantation, respectively (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003eA). The discrimination slightly increased along the prediction times with the AUC values ranging from 0.72 (95% CI 0.67\u0026ndash;0.78) to 0.76 (95% CI 0.68\u0026ndash;0.85) at 1 and 6 years post-transplantation (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003eB). Since the discrimination performances increased and the global prognostic performances remained stable, the calibration properties decreased for the late landmark times (Figure \u003cspan refid=\"MOESM1\" class=\"InternalRef\"\u003eS1\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e\u003c/p\u003e \u003cp\u003eUnder the 11 years \u003cem\u003efinal time horizon\u003c/em\u003e assumption, the global prognostic performances of the DynPG were at a higher level for the earliest landmark times, but decreased along the landmark times. We estimated R\u0026sup2; values of 22% (95% CI 12\u0026ndash;32%), 15% (95% CI -2\u0026ndash;31%) and \u0026minus;\u0026thinsp;2% (95% CI -41\u0026ndash;37%) at 1, 6 and 10 years post-transplantation, respectively (Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003eA). The discrimination was high for the earliest landmark times and increased over times with AUC values of 0.76 (95% CI 0.70\u0026ndash;0.82), 0.76 (95% CI 0.67\u0026ndash;0.84) and 0.88 (95% CI 0.79\u0026ndash;0.98) at 1, 6 and 10 years post-transplantation, respectively (Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003eB). On the other hand, we observed a very poor calibration plot (Figure S2): the predictions appeared underestimated for the earliest landmark times and overestimated for the late ones.\u003c/p\u003e \u003cp\u003e\u003c/p\u003e \u003cp\u003eThe number of at-risk patients at landmark times are the same for the two time horizon definitions. However, the number of observed events and the number of censored subjects on each window differs between the two approaches due different horizon windows, resulting in a number of at-risk patients at the end of the horizon window that are not comparable (Figures S3, S4). The incidence rate on the prediction window, that is either reduced or sliding given the landmark time increases, are thus not the same between the two approaches. Consequently, the dynamic predictions cannot be compared and they tell a different story. For instance, under the \u003cem\u003esliding horizon window\u003c/em\u003e assumption, at 1 year post-transplantation, we estimated a 72% probability that the predicted survival of a patient who actually experienced a graft failure within the 5 years was lower than that of a patient who did not (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e \u0026ndash; part B). Under the \u003cem\u003efinal time horizon\u003c/em\u003e assumption, at 1 year post-transplantation, we estimated a 76% probability that the predicted survival of a patient who actually experienced a graft failure before 11 years post-transplantation was lower than that of a patient who did not (Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003e \u0026ndash; part B). In terms of calibration properties, we may reasonably accept that DynPG was sufficiently well calibrated for the earliest landmark times under the 5-year \u003cem\u003esliding horizon window\u003c/em\u003e assumption (Figure \u003cspan refid=\"MOESM1\" class=\"InternalRef\"\u003eS1\u003c/span\u003e). In contrast, under the 11-year \u003cem\u003efinal time horizon\u003c/em\u003e assumption, the calibration is quite poor (Figure S2).\u003c/p\u003e \u003c/div\u003e"},{"header":"5. Discussion","content":"\u003cp\u003eIn this study, we distinguished two types of time prediction horizon - \u003cem\u003efinal time horizon\u003c/em\u003e or \u003cem\u003esliding horizon window\u003c/em\u003e - for dynamic predictions. We conducted a systematic review that stated the heterogeneity of the used time prediction horizons in the literature about dynamic predictions. While the two definitions are similar, a specific definition is of major importance since the prognostic performances obtained are different given the nature of the prediction window. This is also well illustrated by our concrete application in kidney transplantation.\u003c/p\u003e \u003cp\u003eThe concept of P4-medicine (predictive, preventive, personalized and participatory) is now largely developed in the literature, but still difficult to apply in clinical practice [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]. Dynamic predictive tools can be beneficial for a such health policy and promote shared medical decision making, provided that the prognostic performances are sufficiently good. While dynamic predictions are updated predictions whenever additional information is available during the patient follow-up [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e, \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e], their associated prognostic performances depend on the at-risk population at the time of making a prediction and can evolve as the landmark time increases. In this work, we insist on the fact that prognostic performances are also related to the prediction window. We showed that the incidence rates of the event would not be identical between the two time horizon definitions due to the nature of the window that can be either reduced or sliding, resulting in prognostic performances that are not comparable. The corresponding interpretations of prognostic performances should therefore be formulated with caution since they do not tell the same story in the two contexts.\u003c/p\u003e \u003cp\u003eIn our opinion, the dynamic predictions obtained with the \u003cem\u003esliding horizon window\u003c/em\u003e framework can be used to follow the patient health evolution accross the landmark times and thus present interesting properties for individualized and personalized medicine [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e, \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e]. With a fixed length of prediction window, it allows to appreciate the prognostic performances all along the follow-up. In our concrete application in kidney transplantation, the \u0026lsquo;Dynamic predictions of patient and kidney graft survival\u0026rsquo; has a good discrimination property useful for stratified medicine. The good calibration properties from 1-year until 6-year landmark times are important point for personalized care. We argued that this dynamic predictive tool can be useful for instance to inform kidney transplant recipients of their prognosis over the next 5 years [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e]. This may result in increased patient adherence to their treatment, to increase patient empowerment, as patients recognize having an active role in their chronic disease evolution [\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e, \u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e]. This could also help patients to better manage their feelings of uncertainty about the survival of their transplant in the not-too-distant future.\u003c/p\u003e \u003cp\u003eWhen considering the \u003cem\u003efinal time horizon\u003c/em\u003e (i.e. reducing window), the comparison of prognostic performances across the landmark times is difficult since the lengths of the prediction window are not the same. Despite the bounded character of dynamic predictions for a \u003cem\u003efinal time horizon\u003c/em\u003e, it is important to note that such predictions are not always monotonic along the landmark times because of the actualization of the longitudinal marker. The amelioration or deterioration of the prognosis between two landmark times is not necessarily due to a difference in health state characterized by the new marker measurement. This is noised by the mathematical artefact brought by the reduction of the prediction window. Since it is difficult to know to what extent the prognosis evolution is due to the reduction of the window rather than to the marker evolution, a comparison cannot be done between predictions realized at two different landmark times. For this reason, the \u003cem\u003efinal time horizon\u003c/em\u003e approach does not seem suited to follow the patient health evolution through survival probabilities and cannot be envisaged to support patient personalized care. In our kidney transplantation application, the poor calibration property across the landmark times for a \u003cem\u003efinal time horizon\u003c/em\u003e of 11 years post-transplantation do not allow to consider that individual survival probability are correctly predicted. The more the prediction time approaches 11 years, the more we predicted patient and kidney survival probabilities close to 1, while the observed event repartition appear not. This inevitably results in poor calibration. Such an approach cannot be used to personalize the kidney recipient taking care based on the predicted patient and kidney graft survival probabilities. Nevertheless, in a context in which the objective is not the personalization of patient care, but rather the stratification of the studied population, considering a \u003cem\u003efinal time horizon\u003c/em\u003e may be of interest. This should be clearly stated in the clinical objective. In kidney transplantation, we may envisage to guide patients through the care organization given their stratified survival probabilities. In our application, we observed an increase of the discrimination performance of the DynPG up to 11 years post-transplantation. We may consider to better organize the healthcare visit by reducing the patient visit schedule or consider telemedicine consultations for recipients at higher patient and kidney graft survival probabilities and by reinforcing the patient follow-up for patients with lower survival predictions [\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e, \u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e]. All the same, we recognize that it may be difficult to determine a \u003cem\u003efinal time horizon\u003c/em\u003e of clinical interest, particularly in the context of chronic disease. In our application, the horizon of 11 years is debatable from a patient-centered perspective should difficulty be bounded by a terminal horizon. This may be more appropriate in a public health perspective where efficient allocation of resources could provide economic benefits. Other clinical contexts may be of interest when a final time horizon is already known as for medical devices, with for instance the prediction of aortic valve duration or the prediction of hip replacement duration up to a maximum announced by the supplier [\u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eConsidering these methodological differences between the two time horizon definitions, the studies\u0026rsquo; clinical objective should be clearly anticipated to correctly assess the prognostic performances. Indirectly, the heterogeneity that we observed in the reporting of the time horizon window in our systematic review reveals the lack of clarity regarding the clinical objectives justifying dynamic predictions development. Our systematic review also informed about the methodological quality of dynamic predictions. A non negligible proportion of the literature did not sufficiently respect the TRIPOD recommendation [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e] with a lack of validation studies for instance. We also noted that discrimination properties were mainly reported and referred to the stratification of the studied population, but calibration metrics that are essential for personalized care were not reported in numerous studies.\u003cdiv class=\"BlockQuote\"\u003e\u003cp\u003eIn the literature, numerous developments concern improvements of the joint modeling of the longitudinal and the survival processes and thus participate to the improvement of dynamic predictions performances. For instance, considering competing risks [\u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e, \u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e], to refine the longitudinal evolution with flexible modeling [\u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e38\u003c/span\u003e, \u003cspan citationid=\"CR39\" class=\"CitationRef\"\u003e39\u003c/span\u003e], or incorporating multiple longitudinal markers [\u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e, \u003cspan citationid=\"CR40\" class=\"CitationRef\"\u003e40\u003c/span\u003e], would help to better predict the clinical event of interest. The mathematical notations introduced and the following concrete application concerns the case of one longitudinal marker (e.g. serum creatinine) and one clinical event (e.g. patient and kidney graft survival). Nevertheless, notations can be adapted to more complex modeling situations. Our conclusion about the attention to bring to time prediction windows would still be valid to broader modeling frameworks.\u003c/p\u003e\u003c/div\u003e\u003c/p\u003e"},{"header":"6. Conclusions","content":"\u003cp\u003eIn conclusion, our study identified that the choice of the time prediction window is crucial and should depend on the clinical objectives. Our work advocates for clarifying the clinical objective of the dynamic predictions and improving the reporting when studying dynamic prediction scoring systems since prognostic performances and their interpretation will differ given the time horizon definition.\u003c/p\u003e"},{"header":"Abbreviations","content":"\u003cp\u003e95% CI: 95% Confidence Interval; AUC: Area Under the Curve; CHARMS: CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies; DIVAT: Donn\u0026eacute;es Informatis\u0026eacute;es et VAlid\u0026eacute;es en Transplantation; DynPG : Dynamic predictions of Patient and kidney Graft survival; HLA: Human Leukocyte Antigen, PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses; ROC: Receiver Operating Characteristic ; TRIPOD: Transparent Reporting of multivariable prediction model for Individual Prognosis Or Diagnosis statements.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cem\u003eEthics approval and consent to participate\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eData confidentiality was ensured following the recommendations of the French commission for data protection (Commission Nationale Informatique et Libert\u0026eacute;, CNIL no. 914184, ClinicalTrials.gov recording NCT02900040). The present study was approved by the DIVAT scientific committee (Donn\u0026eacute;es Informatis\u0026eacute;es et VAlid\u0026eacute;es en Transplantation; www.divat.fr).\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eConsent for publication\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eNot applicable\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eAvailability of data and materials\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eThe data underlying this article could be shared on reasonable request to the corresponding author.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eCompeting interests\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eLucas Chabeau and Pierre Rinder report financial support as employee of semeia. Other authors declare no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eFunding\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eThis work was partly supported by S\u0026Ecirc;MEIA and also by the French National Research Agency [grant number ANR PRC 2022 JM-QALYs].\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eAuthors\u0026apos; contributions\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eLucas Chabeau\u0026nbsp;: methodology, formal analysis, systematic review, writing - original draft, review and editing.\u003c/p\u003e\n\u003cp\u003eVincent Bonnemains\u0026nbsp;: methodology, systematic review, writing - review and editing.\u003c/p\u003e\n\u003cp\u003ePierre Rinder\u0026nbsp;: methodology, systematic review, writing - review and editing.\u003c/p\u003e\n\u003cp\u003eMagali Giral\u0026nbsp;: scientific coordinator of the DIVAT cohort, clinical mentorship, data collection, writing - review and editing.\u003c/p\u003e\n\u003cp\u003eSol\u0026egrave;ne Desm\u0026eacute;e: methodology, systematic review, writing - review and editing.\u003c/p\u003e\n\u003cp\u003eEtienne Dantan: conceptualization, methodology, formal analysis, systematic review, writing - original draft, review and editing.\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eAcknowledgments\u0026nbsp;\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003eWe thank members of the DIVAT consortium* and the clinical research assistant team (S. Le Floch, A. Petit, J. Posson, C. Scellier, V. Eschbach, K. Zurbonsen, C. Dagot, F. M\u0026rsquo;Raiagh, V. Godel, X. Longy, P. Przednowed). We are also grateful to Roche Pharma, Novartis, Sanofi and Astellas laboratories for supporting the DIVAT cohort (www.divat.fr) as the CENTAURE foundation (www.fondation-centaure.org).\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e* Donn\u0026eacute;es Informatis\u0026eacute;es et VAlid\u0026eacute;es en Transplantation, DIVAT Cohort Collaborators (Medical Doctors, Surgeons, HLA Biologists)\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eNantes :\u0026nbsp;\u003c/strong\u003e\u003cem\u003eGilles Blancho, Julien Branchereau, Diego Cantarovich, Agn\u0026egrave;s Chapelet, Jacques Dantal, Cl\u0026eacute;ment Deltombe, Lucile Figueres, Claire Garandeau, Magali Giral, Caroline Gourraud-Vercel, Maryvonne Hourmant, Georges Karam, Clarisse Kerleau, Aur\u0026eacute;lie Meurette, Simon Ville, Christine Kandell, Anne Moreau, Karine Renaudin, Anne Cesbron, Florent Delbos, Alexandre Walencik, Anne Devis\u003c/em\u003e\u003cstrong\u003e\u0026nbsp;; Paris-Necker :\u0026nbsp;\u003c/strong\u003e\u003cem\u003eLucile Amrouche, Dany Anglicheau, Olivier Aubert, Lynda Bererhi, Christophe Legendre, Alexandre Loupy, Frank Martinez, R\u0026eacute;becca Sberro-Soussan, Anne Scemla, Claire Tinel, Julien Zuber\u003c/em\u003e\u003cstrong\u003e\u0026nbsp;; Nancy :\u0026nbsp;\u003c/strong\u003e\u003cem\u003ePascal Eschwege, Luc Frimat, Sophie Girerd, Jacques Hubert, Marc Ladriere, Emmanuelle Laurain, Louis Leblanc, Pierre Lecoanet, Jean-Louis Lemelle\u003c/em\u003e\u003cstrong\u003e\u0026nbsp;; Lyon E. H\u0026eacute;riot :\u0026nbsp;\u003c/strong\u003e\u003cem\u003eLionel Badet, Maria Brunet, Fanny Buron, R\u0026eacute;mi Cahen, Sameh Daoud, Coralie Fournie, Arnaud Gr\u0026eacute;goire, Alice Koenig, Charl\u0026egrave;ne L\u0026eacute;vi, Emmanuel Morelon, Claire Pouteil-Noble, Thomas Rimmel\u0026eacute;, Olivier Thaunat\u003c/em\u003e\u003cstrong\u003e\u0026nbsp;; Montpellier :\u0026nbsp;\u003c/strong\u003e\u003cem\u003eSylvie Delmas, Val\u0026eacute;rie Garrigue, Moglie Le Quintrec, Vincent Pernin, Jean-Emmanuel Serre.\u003c/em\u003e\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eFlores M, Glusman G, Brogaard K, Price ND, Hood L. P4 medicine: how systems medicine will transform the healthcare sector and society. Pers Med. 2013;10:565\u0026ndash;76.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eProust-Lima C, Blanche P. Dynamic Predictions. Wiley StatsRef: Statistics Reference Online. John Wiley \u0026amp; Sons, Ltd; 2016. pp. 1\u0026ndash;6.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAndrinopoulou E-R, Harhay MO, Ratcliffe SJ, Rizopoulos D. Reflection on modern methods: Dynamic prediction using joint models of longitudinal and time-to-event data. Int J Epidemiol. 2021;50:1731\u0026ndash;43.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAsgari S, Khalili D, Zayeri F, Azizi F, Hadaegh F. Dynamic prediction models improved the risk classification of type 2 diabetes compared with classical static models. J Clin Epidemiol. 2021;140:33\u0026ndash;43.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRizopoulos D. Joint Models for Longitudinal and Time-to-Event Data: With Applications in R. CRC; 2012.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRizopoulos D. Dynamic predictions and prospective accuracy in joint models for longitudinal and time-to-event data. Biometrics. 2011;67:819\u0026ndash;29.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBarrett J, Su L. Dynamic predictions using flexible joint models of longitudinal and time-to-event data. Stat Med. 2017;36:1447\u0026ndash;60.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLi K, Luo S. Dynamic predictions in Bayesian functional joint models for longitudinal and time-to-event data: An application to Alzheimer\u0026rsquo;s disease. Stat Methods Med Res. 2019;28:327\u0026ndash;42.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTeramukai S, Okuda Y, Miyazaki S, Kawamori R, Shirayama M, Teramoto T. Dynamic prediction model and risk assessment chart for cardiovascular disease based on on-treatment blood pressure and baseline risk factors. Hypertens Res Off J Jpn Soc Hypertens. 2016;39:113\u0026ndash;8.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFournier M-C, Dantan E, Blanche P. An R2 -curve for evaluating the accuracy of dynamic predictions. Stat Med. 2018;37:1125\u0026ndash;33.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAsar \u0026Ouml;, Fournier M-C, Dantan E. Dynamic predictions of kidney graft survival in the presence of longitudinal outliers. Stat Methods Med Res. 2021;30:185\u0026ndash;203.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBen-Hassen C, Helmer C, Berr C, Jacqmin-Gadda H. Five-Year Dynamic Prediction of Dementia Using Repeated Measures of Cognitive Tests and a Dependency Scale. Am J Epidemiol. 2022;191:453\u0026ndash;64.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSteyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiol Camb Mass. 2010;21:128\u0026ndash;38.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBlanche P, Proust-Lima C, Loub\u0026egrave;re L, Berr C, Dartigues J-F, Jacqmin-Gadda H. Quantifying and comparing dynamic predictive accuracy of joint models for longitudinal marker and time-to-event in presence of censoring and competing risks. Biometrics. 2015;71:102\u0026ndash;13.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKamarudin AN, Cox T, Kolamunnage-Dona R. Time-dependent ROC curve analysis in medical research: current methods and applications. BMC Med Res Methodol. 2017;17:53.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFournier M-C, Foucher Y, Blanche P, Legendre C, Girerd S, Ladri\u0026egrave;re M, et al. Dynamic predictions of long-term kidney graft failure: an information tool promoting patient-centred care. Nephrol Dial Transplant Off Publ Eur Dial Transpl Assoc -. Eur Ren Assoc. 2019;34:1961\u0026ndash;9.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFerrer L, Putter H, Proust-Lima C. Individual dynamic predictions using landmarking and joint modelling: Validation of estimators and robustness assessment. Stat Methods Med Res. 2019;28:3649\u0026ndash;66.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGoldstein BA, Pomann GM, Winkelmayer WC, Pencina MJ. A comparison of risk prediction methods using repeated observations: An application to Electronic Health Records for Hemodialysis. Stat Med. 2017;36:2750\u0026ndash;63.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003evan Houwelingen HC, Putter H. Dynamic Prediction in Clinical Survival Analysis. CRC; 2012.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMoher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151:264\u0026ndash;9. W64.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMoons KGM, de Groot JAH, Bouwmeester W, Vergouwe Y, Mallett S, Altman DG, et al. Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist. PLoS Med. 2014;11:e1001744.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMoons KGM, Altman DG, Reitsma JB, Ioannidis JPA, Macaskill P, Steyerberg EW, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162:W1\u0026ndash;73.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSteyerberg EW, Bleeker SE, Moll HA, Grobbee DE, Moons KGM. Internal and external validation of predictive models: a simulation study of bias and precision in small samples. J Clin Epidemiol. 2003;56:441\u0026ndash;7.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDevaux A, Genuer R, Peres K, Proust-Lima C. Individual dynamic prediction of clinical endpoint from large dimensional longitudinal biomarker history: a landmark approach. BMC Med Res Methodol. 2022;22:1\u0026ndash;14.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYang T, Yang Y, Jia Y, Li X. Dynamic prediction of hospital admission with medical claim data. BMC Med Inf Decis Mak. 2019;19(Suppl 1):18.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLin J, Li K, Luo S. Functional survival forests for multivariate longitudinal outcomes: Dynamic prediction of Alzheimer\u0026rsquo;s disease progression. Stat Methods Med Res. 2021;30:99\u0026ndash;111.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFahimi F, Guo Y, Tong SC, Ng A, Bing SOY, Choo B et al. A Vital Signs Telemonitoring Programme Improves the Dynamic Prediction of Readmission Risk in Patients with Heart Failure. AMIA Annu Symp Proc AMIA Symp. 2020;2020:432\u0026ndash;41.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZeng Z, Tang X, Liu Y, He Z, Gong X. Interpretable recurrent neural network models for dynamic prediction of the extubation failure risk in patients with invasive mechanical ventilation in the intensive care unit. BioData Min. 2022;15:21.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHowell M, Wong G, Rose J, Tong A, Craig JC, Howard K. Patient Preferences for Outcomes After Kidney Transplantation: A Best-Worst Scaling Survey. Transplantation. 2017;101:2765\u0026ndash;73.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLenain R, Dantan E, Giral M, Foucher Y, Asar \u0026Ouml;, Naesens M, et al. External Validation of the DynPG for Kidney Transplant Recipients. Transplantation. 2021;105:396\u0026ndash;403.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGordon EJ, Butt Z, Jensen SE, Lok-Ming Lehr A, Franklin J, Becker Y, et al. Opportunities for shared decision making in kidney transplantation. Am J Transpl Off J Am Soc Transpl Am Soc Transpl Surg. 2013;13:1149\u0026ndash;58.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eVandecasteele SJ, Kurella Tamura M. A patient-centered vision of care for ESRD: dialysis as a bridging treatment or as a final destination? J Am Soc Nephrol JASN. 2014;25:1647\u0026ndash;51.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJ\u0026ouml;rres A, John S, Lewington A, ter Wee PM, Vanholder R, Van Biesen W, et al. A European Renal Best Practice (ERBP) position statement on the Kidney Disease Improving Global Outcomes (KDIGO) Clinical Practice Guidelines on Acute Kidney Injury: part 2: renal replacement therapy. Nephrol Dial Transplant Off Publ Eur Dial Transpl Assoc -. Eur Ren Assoc. 2013;28:2940\u0026ndash;5.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFoucher Y, Meurette A, Daguin P, Bonnaud-Antignac A, Hardouin J-B, Chailan S, et al. A personalized follow-up of kidney transplant recipients using video conferencing based on a 1-year scoring system predictive of long term graft failure (TELEGRAFT study): protocol for a randomized controlled trial. BMC Nephrol. 2015;16:6.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eEvans JT, Evans JP, Walker RW, Blom AW, Whitehouse MR, Sayers A. How long does a hip replacement last? A systematic review and meta-analysis of case series and national registry reports with more than 15 years of follow-up. Lancet. 2019;393:647\u0026ndash;54.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAndrinopoulou E-R, Rizopoulos D, Takkenberg JJ, Lesaffre E. Combined dynamic predictions using joint models of two longitudinal outcomes and competing risk data. Stat Methods Med Res. 2017;26:1787\u0026ndash;801.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSuresh K, Taylor JMG, Spratt DE, Daignault S, Tsodikov A. Comparison of joint modeling and landmarking for dynamic prediction under an illness-death model. Biom J Biom Z. 2017;59:1277\u0026ndash;300.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYang M, Luo S, DeSantis S. Bayesian quantile regression joint models: Inference and dynamic predictions. Stat Methods Med Res. 2019;28:2524\u0026ndash;37.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDesm\u0026eacute;e S, Mentr\u0026eacute; F, Veyrat-Follet C, S\u0026eacute;bastien B, Guedj J. Nonlinear joint models for individual dynamic prediction of risk of death using Hamiltonian Monte Carlo: application to metastatic prostate cancer. BMC Med Res Methodol. 2017;17:105.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLi N, Liu Y, Li S, Elashoff RM, Li G. A flexible joint model for multiple longitudinal biomarkers and a time-to-event outcome: With applications to dynamic prediction using highly correlated biomarkers. Biom J Biom Z. 2021;63:1575\u0026ndash;86.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Time-to-event dynamic predictions, landmark times, horizon window, time horizon, discrimination, calibration","lastPublishedDoi":"10.21203/rs.3.rs-3938204/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-3938204/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eBackground.\u003c/h2\u003e \u003cp\u003eTime-to-event dynamic predictions are defined as the probability to survive until a defined time horizon given being event-free at landmark times and given available predictive variables at such prediction times. From two different mathematical formulations, dynamic predictions can either predict the survival probability until a \u003cem\u003efinal time horizon\u003c/em\u003e or until the end of a \u003cem\u003esliding horizon window.\u003c/em\u003e We aim to illustrate the need to clearly define the time horizon to correctly interpret the prognostic performances.\u003c/p\u003e\u003ch2\u003eMethods.\u003c/h2\u003e \u003cp\u003e First, following the PRISMA, CHARMS and TRIPOD recommendations, we conducted a systematic review of articles concerning dynamic predictions to assess how the time horizon was reported in the literature. Second, using a sample of 2,523 kidney recipients, we assessed the prognostic capacities of the Dynamic predictions of Patient and kidney Graft survival (DynPG) using either a \u003cem\u003efinal time horizon\u003c/em\u003e or a \u003cem\u003esliding horizon window\u003c/em\u003e.\u003c/p\u003e\u003ch2\u003eResults.\u003c/h2\u003e \u003cp\u003eOf 172 references retrieved about dynamic predictions, 102 articles were included in the systematic review. We notably observed that 71 (69.6%) used a \u003cem\u003esliding horizon window\u003c/em\u003e to assess the prognostic performance while 18 (17.7%) used a \u003cem\u003efinal time horizon\u003c/em\u003e. We also identified 13 articles (12.7%) where the time horizon was not defined clearly (or at all). Our concrete application in kidney transplantation shows that discrimination and calibration are not the same when comparing the two time horizon definitions. On one hand, for a 5-year \u003cem\u003esliding horizon window\u003c/em\u003e, the discrimination slightly increased as the landmark times increased, and we also observed that DynPG is reasonably well calibrated, particularly for the earliest landmark times. On the other hand, for an 11-year \u003cem\u003efinal time horizon\u003c/em\u003e, the discrimination was high for the earliest landmark times and increased over time, while the calibration plot revealed predictions were underestimated for the earliest landmark times and overestimated for later ones.\u003c/p\u003e\u003ch2\u003eConclusions.\u003c/h2\u003e \u003cp\u003eOur systematic review identified a clear heterogeneity in the time horizon definition used, and an absence of a clear time horizon definition in a part of published articles. Our study advocates for improving the reporting when studying dynamic prediction scoring systems since the prognostic performances and interpretation differ according to the time horizon definition.\u003c/p\u003e","manuscriptTitle":"On the importance of a clear definition of time horizon for time-to-event dynamic predictions: a systematic review and a concrete illustration in kidney transplantation","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-02-14 18:29:54","doi":"10.21203/rs.3.rs-3938204/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"eedda178-8f88-4781-8a78-2df2f0b42036","owner":[],"postedDate":"February 14th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2024-08-09T11:48:08+00:00","versionOfRecord":[],"versionCreatedAt":"2024-02-14 18:29:54","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-3938204","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-3938204","identity":"rs-3938204","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}