Assessing predictive abilities of hazard-based regression models for survival data: a tutorial for prognosis modelling

Assessing predictive abilities of hazard-based regression models for survival data: a tutorial for prognosis modelling | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Assessing predictive abilities of hazard-based regression models for survival data: a tutorial for prognosis modelling Marguerite Fournier, Francisco Javier Rubio, Loïc Chartier, Camille Maringe, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-3866618/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Predicting the occurrence of an event over time for a newly diagnosed individual is a common aim in medical statistics. For time-to-event outcomes, this prediction is typically based on a regression model. The Cox proportional hazard (PH) model represents one of the most popular regression models for analysing time-to-event data. However, several flexible models that go beyond the assumption of proportionality of hazards have been recently developed. These include flexible hazard-based models using splines or models based on more general hazard structures. In these 2 types of models, non-linear associations and time-varying regression coefficient(s) can be easily included. Assessing the predictive ability of a hazard-based regression model is necessary to validate a predictive model but it might prove difficult for models other than the Cox PH model. We present a tutorial which explains how the predictive ability of hazard-based regression models can be assessed, focusing on the 3 commonly used performance measures. We report (i) the overall prediction ability using prediction error curve and the Brier score, (ii) the discriminative ability using the cumulative/dynamic area under the receiving operator characteristic curve, and (iii) the calibration ability, i.e., the agreement between observed and predicted probabilities, using calibration plots and graphical comparison between predicted and observed survival. We provide an implementation of these methods in R together with an illustrative example using a publicly available data set. Figures Figure 1 Figure 2 Figure 3 Figure 4 Introduction Predicting the occurrence of a time-to-event of interest since cancer diagnosis is an important part of medical research in oncology ( 1 ) ( 2 ) ( 3 ). Predictions are typically based on survival regression models formulated in terms of the hazard function. The main goal of these models is to equip physicians and their patients with accurate tools to develop an optimal care strategy based on the available information from patients (such as age, sex, cancer stage, co-morbidities, frailty, et cetera). The semi-parametric Cox model ( 4 ) is the most popular model in time-to-event data analysis. The Cox model has many advantages, such as being easily interpretable and relying on a semi-parametric structure as the baseline hazard is not specified (and therefore not forced to have a particular shape). However, to predict survival probabilities from the Cox model, the baseline hazard function needs to be estimated. Moreover, the classical Cox model relies on the proportional hazards (PH) assumption, where the hazard ratio (HR) associated with a covariate is assumed to be constant over time. This assumption may be reasonable only for a short follow-up period after diagnosis. Consequently, the Cox model may be unable to describe more complex scenarios where the proportionality of hazards assumption is not reasonable. Several regression models have been proposed to relax the PH assumption and thus allow for non-proportional hazards. For instance, alternative regression models have been proposed using flexible functions for modelling the baseline hazard and/or time-dependent regression coefficients for covariates ( 5 )( 6 )( 7 )( 8 )( 9 ). One possible choice for such flexible functions are spline functions ( 10 ), in particular B-splines ( 11 ). Models based on B-splines allow great flexibility for modelling both the baseline hazard and the covariate effects ( 12 )( 13 ). Another class of flexible parametric models has recently been proposed based on a general hazard structure (14)( 15 ). This general structure contains the proportional hazards and accelerate failure time as particular cases. This model is able to incorporate time-level effects, while avoiding the need for numerical integration, and flexibly modelling the baseline hazard using a flexible parametric distribution (e.g., the Power Generalized Weibull). As mentioned in a book by E. Steyerberg ( 16 ), flexible models might be better for fitting particular patterns in a specific dataset at the increased risk of overfitting. It is therefore not immediately clear whether a more flexible model is to be preferred in comparison of a simpler model. Once a predictive hazard-based regression model has been built or selected ( 17 ), it is necessary to quantify the accuracy of predictions derived from that model ( 16 ). Assessing a model’s predictive ability can be performed by using (i) overall performance measures of model predictive accuracy, (ii) discrimination measures to assess the ability of the model to classify patients in groups with homogeneous risk levels, and (iii) calibration measures, which describe the agreement between the observed and the predicted risk ( 18 )( 19 ). Their combination is important as a model may exhibit good classification performance (discrimination), but it may also have poor performance in terms of predictions related to specific subgroups (calibration), or vice versa. Our main aim is to illustrate the use of prediction performance measures for two types of parametric hazard-based regression models applied on time-to-event data, in addition to the classical semiparametric Cox PH model. A secondary aim is to investigate the trade-off between flexibility and predictive performance. That is, whether the cost of using a more flexible model is justified by a considerable increase in predictive performance. We focus on two types of flexible models: (i) one based on splines and a classical hazard structure (i.e. where the baseline hazard is multiplied by the exponential of a linear predictor, which may include time-dependent effects) ( 12 )( 13 )( 20 ), and (ii) another one based on a general hazard structure (14) ( 15 ) ( 21 ), where the covariates play a role at both the hazard level and the time scale (time-dependent effects). The remainder of the paper is organized as follows. In Section 2, we describe the dataset used in our illustrative example. In Section 3, we describe the hazard regression models to be used in our examples as well as the performance measures used for evaluating their predictive abilities. Section 4 presents the results for the model selection strategy and the prediction measures computed for the different candidate models. We conclude with a discussion of the results and potential research directions. Material: The LeukSurv dataset This dataset is freely available in the spBayesSurv R package and it was first analyzed by Henderson et al. (2002) ( 22 ). It includes 1043 patients diagnosed with acute myeloid leukemia (AML), with the time until death or last known follow-up \(T\) (in years) and the corresponding event indicator \(\delta\) (being 0 for censored patients and 1 for death). We aim to predict the survival after AML diagnosis according to the following covariates: age (continuous) at diagnosis, sex, white blood cell count (WBC) at diagnosis ( \(50*{10}^{9}/L\) ) (continuous) and the Townsend deprivation score (continuous) which is an area-based measure of deprivation for the enumeration district of residence (higher values indicating more deprived areas). Method Hazard regression models In this section, we present the two regression models that we are considering as potential alternative to the Cox PH model. Flexible hazard-based regression model using regression splines This model uses regression spline functions to model the baseline hazard function and (possibly) time-dependent and/or non-linear effect(s) of covariates. Spline functions are flexible mathematical functions defined by D-degree polynomial segments, which are joined at K junction points called knots ( 6 )( 10 )( 23 )( 24 )( 25 ). A simple example with 2 covariates \({(x}_{1},{x}_{2})\) , where the variable \({x}_{2}\) is assumed to be a continuous variable (such as age), helps to illustrate the parametrisation of the model for the hazard $$\lambda \left(t|{x}_{1},{x}_{2}\right)= {\lambda }_{0}\left(t\right)*\text{exp}\left({\beta }_{1}{x}_{1}+ g\left(t\right)*{x}_{2}+j\left({x}_{2}\right)\right),$$ where \({\beta }_{1}\) is the regression coefficient for \({x}_{1}\) ; while \(g\left(t\right)\) , \(j\left({x}_{2}\right)\) and the log baseline-hazard \({\text{l}\text{o}\text{g}(\lambda }_{0}\left(t\right))\) are B-splines functions. The degree of the spline (D) as well as the number and the locations of the (K) knots need to be specified by the user and may differ for \(g\) , \(j\) and \({\lambda }_{0}\) . The use of flexible functions to describe baseline hazard, non-proportional effect of variables and non-linear functional form for continuous variables, allows for modelling smooth and more realistic hazard shapes while ensuring parsimony for the model. One drawback of this model is that we need to determine the number and specify the position of the knots for the spline functions. To have enough flexibility, we choose a cubic B-spline with one knot located at the median of the observed event times for the spline functions of time (i.e., the baseline hazard \({\lambda }_{0}\left(t\right)\) and time-dependent effect \(g\left(t\right)\) ). This knot location ensures to have the same quantity of information on both sides of the knot. For the non-linear functional form of continuous variable, the user may choose another type of spline, as for example a restricted cubic spline with 2 knots located at the 33rd and 66th quantile of the distribution of \({x}_{2}\) . The more knots there are, the greater the flexibility of the model but at a higher risk of overfitting. General hazard structure model with a flexible baseline hazard This general hazard (GH) structure (15) represents a rich class of hazard-based regression models, which is formulated in terms of the hazard function (exemplified with two covariates \({x}_{1}\) and \({x}_{2}\) here): $${\lambda }^{GH}\left(t;{x}_{1},{x}_{2}\right)= {\lambda }_{0}\left(t\text{exp}\left({\beta }_{11}{x}_{1}+{\beta }_{12}{x}_{2}\right)\right)\text{exp}\left({{\beta }_{21}x}_{1}+{\beta }_{22}{x}_{2}\right),$$ where \({\lambda }_{0}\left(t\right)\) is the baseline hazard function, which can be modelled using a pre-specified parametric distribution (see below), { \({\beta }_{11}, {\beta }_{21}\}\) and { \({\beta }_{12}, {\beta }_{22}\}\) are regression coefficients of the variables \({x}_{1}\) and \({x}_{2}\) , respectively. The parameters \({\{\beta }_{11}, {\beta }_{12}\}\) allow for the inclusion of time-level effects of the covariates \({x}_{1}\) and \({x}_{2}\) . The GH model includes as special cases: the proportional hazards model (PH, when \({\beta }_{1j}=0\) ), the accelerated failure time model (AFT, when \({\beta }_{1j}= {\beta }_{2j}\) ), the accelerated hazards model (AH, when \({\beta }_{2j}=0\) ) and the Hybrid Hazard model (HH, when different covariates are included in the argument of the baseline hazard and those in the argument of the exponential function multiplying the baseline hazard). The GH model is identifiable provided that the baseline hazard is not the hazard function associated with the Weibull distribution ( 14 ). To complete the specification of the GH model, we need to select a parametric distribution for the baseline hazard. Here, we will use the Power Generalized Weibull (PGW) distribution instead of Exponentiated Weibull distribution originally proposed ( 15 ), as it is easier to implement while being equally flexible ( 26 ). The PGW distribution is a 3-parameter distribution which is defined according to the following equations for the density, survival, and hazard functions: $$f\left(t;\sigma ,\nu ,\gamma \right)=\frac{\nu }{{\gamma \sigma }^{\nu }}{t}^{\nu -1}{\left[1+{\left(\frac{t}{\sigma }\right)}^{\nu }\right]}^{\left(\frac{1}{\gamma }-1\right)}exp\left\{1-{\left[1+{\left(\frac{t}{\sigma }\right)}^{\nu }\right]}^{\frac{1}{\gamma }}\right\}$$ , $$S\left(t;\sigma ,\nu ,\gamma \right)= exp\left\{1-{\left[1+{\left(\frac{t}{\sigma }\right)}^{\nu }\right]}^{\frac{1}{\gamma }}\right\}$$ , $$\lambda \left(t;\sigma ,\nu ,\gamma \right)=\frac{\nu }{{\gamma \sigma }^{\nu }}{t}^{\nu -1}{\left[1+{\left(\frac{t}{\sigma }\right)}^{\nu }\right]}^{\left(\frac{1}{\gamma }-1\right)}$$ , where \(f\left(t;\sigma ,\nu ,\gamma \right)\) is the probability density function, \(S\left(t;\sigma ,\nu ,\gamma \right)\) is the survival function and \(\lambda \left(t;\sigma ,\nu ,\gamma \right)\) is the hazard function, \(\sigma >0\) is a scale parameter and \(\nu ,\gamma >0\) are shape parameters. The hazard function associated with the PGW distribution can capture several shapes: constant, increasing, decreasing, bathtub and unimodal, which are the shapes usually observed in cancer survival applications ( 15 ). Implementations of the GH model with PGW baseline hazard (among others) can also be found in the R packages GHSurv and HazReg. Model-building strategy To select between different regression models within a given class of model, (i.e., spline-based model or GH model), we use the Akaike Information Criterion (AIC). We start by fitting several pre-specified candidate models, based on clinical knowledge of the disease, defined with or without time-dependent effect for some covariates (specified in the result section, and then we choose the model with the smallest AIC among the set of candidate models. For comparison purposes, we also fit the Cox model assuming proportional hazards for each covariate in the model. Evaluating the predictive abilities of hazard-based regression models for survival data Several performance measures have been developed to compare regression models in terms of their ability to predict individual survival probabilities at a pre-specified time horizon \({t}^{*}\) . In this paper, because we aimed to focus on describing the predictive performance measures, we assume that the predictive models have been built based on a different dataset and they are then validated on the LeukSurv dataset. If the model was built and also validated on the same dataset, specific techniques need to be used such as cross-validation ( 16 ) to account for the optimism of the performance measures The performance measures can be broadly classified into three classes ( 18 )( 2 ): overall performance measures, discrimination measures and calibration measures. Overall performance measures estimate the accuracy of prediction at time \(t\) by calculating the difference between predicted and observed outcomes for all individuals at time \(t\) using a loss function (that is, the “prediction error”). Discrimination measures reflect the model’s ability to separate individuals who experienced the event from those who did not. Calibration measures compare the survival predictions for specific subgroups to their observed survival. We describe these performance measures in some details before we use them in our illustrative example. Overall performance measure: prediction error estimated with the Brier Score The Brier Score at time \(t\) ( 27 ) measures the mean quadratic difference between the predicted survival probability for subject \(i\) at time \(t\) and the observed event status for the same subject \(i\) at time \(t\) . It quantifies the degree to which the predicted values coincide with the observed outcomes. Without censoring it can be estimated at time \(t\) by ( 28 )( 29 ) $$\widehat{BS\left(t\right)}= \frac{1}{N}{\sum _{i=1}^{N}(\mathbb{l}\left\{{T}_{i} > t\right\}-{\widehat{S}}_{i}\left(t\right))}^{2},$$ where \({\widehat{S}}_{i}\left(t\right)\) is the model-based predicted probability of surviving beyond time \(t\) for individual \(i\) , and the indicator \(\mathbb{l}\left\{{T}_{i} > t\right\}\) is equal to 1 when patient \(i\) is known to be alive at time \(t\) and is equal to 0 if patient \(i\) died before or at \(t\) . When the sample contains right-censored observations, the indicator \(\mathbb{l}\left\{{T}_{i} > t\right\}\) cannot always be computed. Indeed, if patient \(i\) is censored before \(t\) , \(\mathbb{l}\left\{{T}_{i} > t\right\}\) is unknown because \({T}_{i}\) is not observed. A proposed solution to handle right censoring is based on weighting each contribution to the Brier Score ( 30 ), assuming that censored patients can be represented by patients with complete information. These weights are defined through the Kaplan-Meier estimates of the conditional survival function of the censoring times, which therefore represents the probability to be uncensored at time \(t\) , \(\widehat{{S}_{C}}\left(t\right)=\widehat{P}(C>t)\) . Therefore, an estimator of the Brier Score that incorporates censoring (assuming random censoring) is given by: $$\widehat{BS\left(t\right)}=\frac{1}{N}\sum _{i=1}^{N}\left[\frac{{\mathbb{I}}_{\left({\stackrel{\sim}{T}}_{i}\le t,{\delta }_{i}=1\right)}}{\widehat{{S}_{C}}\left({\stackrel{\sim}{T}}_{i}\right)}{\left(\mathbb{l}\left\{{\stackrel{\sim}{T}}_{i} > t\right\}-{\widehat{S}}_{i}\left(t\right)\right)}^{2}+\frac{{\mathbb{I}}_{\left({\stackrel{\sim}{T}}_{i}>t\right)}}{\widehat{{S}_{C}}\left(t\right)}{\left(\mathbb{l}\left\{{\stackrel{\sim}{T}}_{i} > t\right\}-{\widehat{S}}_{i}\left(t\right)\right)}^{2}\right],$$ where \({\stackrel{\sim}{T}}_{ }=\text{min}\left(T,C\right)\) with C is the right-censoring time and \(\delta\) is the event indicator (equals to 1 if \({\stackrel{\sim}{T}=T}_{ }\) and 0 if \({\stackrel{\sim}{T}=C}_{ }\) ). The contribution of patients who had the event before time \(t\) is \({ \left(0-{\widehat{S}}_{i}\left(t\right)\right)}^{2}\) , and this contribution is weighted by the inverse probability to be uncensored at time \({\stackrel{\sim}{T}}_{i}\) (first term in the equation above). The contribution of patients who are still at risk at time \(t\) is \({\left(1-{\widehat{S}}_{i}\left(t\right)\right)}^{2}\) , which is again weighted by the inverse probability of being uncensored at time \(t\) (second term in the equation). If the probability to be uncensored beyond \({\stackrel{\sim}{T}}_{i}\) or \(t\) is close to 1, then the weight is slightly bigger than 1. Conversely, if this probability is low, meaning that many patients have been censored before time \({\stackrel{\sim}{T}}_{i}\) or \(t\) , the weight is far bigger than 1, thus upweighting the difference calculated on patients still under observation. Notice that patients who are censored before time \(t\) contribute to none of the 2 terms detailed just above (both indicators \({\mathbb{I}}_{\left({\stackrel{\sim}{T}}_{i}\le t,{\delta }_{i}=1\right)}\) and \({\mathbb{I}}_{\left({\stackrel{\sim}{T}}_{i}\ge t\right)}\) are equal to 0 for them); but censored patients contribute through the estimation of the censoring distribution \(\widehat{{S}_{C}}\) . The closer the Brier Score is to 0, the better the overall model accuracy. Discrimination Intuitively, we can say that models that are able to distinguish between patients who die shortly after diagnosis from those who survive longer are well discriminated. The C-Index The C-Index (a.k.a Concordance Index) ( 31 )( 32 )( 33 ) is a commonly used discrimination measure. In the context of the Cox PH model, the C-Index indicates the probability that a subject \(i\) has an estimated Cox model linear predictor higher than subject \(j\) , given that subject \(i\) died before subject \(j\) and with subjects \(i\) and \(j\) randomly selected ( 34 ). In the Cox PH model, the linear predictor \(\eta\) is the quantity involving the regression coefficients and the covariates in the equation of the regression hazard modelling, \(\lambda \left(t;X\right)={\lambda }_{0}\left(t\right){exp}\left({X}^{T}\beta \right)\) (that is, \(\eta ={X}^{T}\beta\) ), where \(X\) is the vector of covariates. The use of the linear predictor from a Cox PH model uniquely determines the ranking of the risks between two individuals at any time \(t\) since no time-dependent regression coefficient are involved. When one uses a more general model, where one (or more) time-dependent regression coefficients are included, then the ranking is not preserved over time. Then, one can use the predicted risk ( i.e. \(\widehat{Risk}\left({t}^{*},i\right)= 1-\widehat{S}\left({t}^{*};{X}_{i}\right)\) ) instead of the linear predictor to calculate the C-index, $$C= \text{Prob}(\widehat{Risk}\left({t}^{*},i\right)>\widehat{Risk}\left({t}^{*},j\right)\mid individual i had the event before j)$$ , where \(\widehat{Risk}\left({t}^{*},i\right)\) is the estimated risk at time horizon \({t}^{*}\) for individual \(i\) derived from the prognosis model under study. For a Cox PH model, the ordering of the predicted risks \(\widehat{Risk}\left({t}^{*},i\right), i=1,\dots ,N\) will remain the same for any time horizon \({t}^{*}\) , while the ordering might change using a model that includes time-dependent regression coefficient(s). Calculating the C-index is equivalent to calculate the proportion of concordant pairs among all usable pairs ( 34 ). One pair is concordant when the predicted risk of an event is lower for subject \(j\) who will have the event at later point compared to subject \(i\) . A concordance probability of 1 means that the model has a perfect discrimination, while a value of 0.5 indicates that the model provides no more information than random ordering. In the absence of censoring, it is easier to calculate the C-index as all the required quantities can be calculated straightforwardly. However, the presence of censoring is more the rule than the exception in most real-life applications in survival analysis, which complicates the estimation of the C-index. Indeed, it is impossible to order the event times of interest (e.g., death) for pairs of patients if one event of a pair is censored. These pairs of observations are discarded, as they are non-informative for the estimation of the C-index. Therefore, the C-index is affected by the proportion of censored observations. Moreover, it has been shown that the C-index is not proper when interest lies in evaluating the risk of an event after \({t}^{*}\) -years ( 35 ). The time-dependent Area Under the Receiver Operating Characteristic Curve Another tool for assessing discrimination is the time-dependent Area Under the Receiver Operating Characteristic (ROC) Curve, \(AUC\left(t\right)\) ( 36 ). A ROC curve at a specific time point \(t\) , \(ROC\left(t\right)\) , displays the “true positive rate” (sensitivity) on the y-axis according to the “false positive rate” (i.e., 1-specificity) on the x-axis for successive threshold values. Each threshold value is used in turn to classify the patients into two groups at time \(t\) : those predicted to have experienced the event of interest at \(t\) among those considered as cases, and those predicted to not experience the event at \(t\) among controls, based on their marker \(Q\) . The Area Under the ROC Curve, \(AUC\left(t\right)\) represents the probability that a marker or a model-based quantity, denoted \({Q}_{i}\) , of a randomly selected subject i among cases is greater than \({Q}_{j}\) of a randomly selected subject j among controls (by convention, we assume that a greater \(Q\) leads to a higher risk). The closer the \(AUC\left(t\right)\) is to 1, the better the discrimination. The way we define patients as cases or controls is important for clear definitions of sensitivity and specificity ( 37 ). We propose to define cases at time \(t\) as patients who have experienced the event of interest prior to time \(t\) , i.e. , \(D\left(t\right)=1\) if \(T\le t\) & \(\delta =1\) , and we define controls at time \(t\) as patients who have not yet had the event at time \(t\) , i.e. , \(D\left(t\right)=0\) and \(T>t\) . From the terminology proposed by Heagerty and Zheng ( 38 ), this definition corresponds to the cumulative case/dynamic control \(AUC\left(t\right)\) , and it will be the focus of the remaining part. Alternatively, one could use incident case and static control or incidence case/dynamic control, but this AUC seems to be inaccurate to evaluate model performance. More details on these definitions can be found in ( 37 ) . If we assume a sample without censoring, the steps to estimate the \(AUC\left(t\right)\) will be as follows. We start with a threshold value of \({c}_{1}\) . Among those who have had the event prior to time \(t\) , we classify patients in the group of “patients who will have the event prior to time \(t\) as predicted from the prognosis model” if \(Q>{c}_{1}\) , and this will be used to estimate the sensitivity, \(sen\left({c}_{1},t\right)= P\left\{Q>{c}_{1} \right| D\left(t\right)=1\}\) . Among those who have not had the event at time \(t\) , we classify patients in the group of “patients who will not have the event at time \(t\) as predicted from the prognosis model” if \(Q\le {c}_{1}\) , and this will be used to estimate the specificity, \(spe\left({c}_{1},t\right)= P\left\{Q\le {c}_{1} \right| D\left(t\right)=0\}\) . We can then repeat these steps with different thresholds to obtain a ROC curve at time \(t\) and calculate \(AUC\left(t\right)\) . We can then obtain the ROC curve for any time \(t\) , \(ROC\left(t\right)\) and so the \(AUC\left(t\right)\) . In the definitions above, the quantity \(Q\) can be a model-based quantity, such as the predicted value of the linear predictor \(X\widehat{\beta }\) derived from the prognosis model. It may also be a time-varying quantity, such as \(Q\left(t\right)=X\widehat{\beta \left(t\right)}\) or \(Q\left(t\right)= 1-\widehat{S}\left(t;X\right)\) : that will prove useful when a general model with time-dependent regressions coefficient(s) has been fitted. We use the estimator of cumulative dynamic AUC as detailed in \(\left(37\right)\) , $${\widehat{AUC}}^{C,D}\left(\text{t}\right)= \frac{\sum _{i=1}^{n}\sum _{j=1}^{n}\mathbb{I}\left( {T}_{i}\le t\right)\mathbb{I}\left( {T}_{j}>t\right)\mathbb{I}\left({Q}_{i}\left(t\right)>{Q}_{j}\left(t\right)\right)* \frac{{\delta }_{i}}{\widehat{{S}_{C}}\left({T}_{i}\right)\widehat{{S}_{C}}\left(t\right)}}{{n}^{2}\widehat{S}\left(t\right)\left[1-\widehat{S}\left(t\right)\right]},$$ where \(\widehat{{S}_{C}}\) is the Kaplan-Meier estimator of the survival function of the censoring time C (thus allowing to deal with censored observations through Inverse Probability of Censoring Weights ( 40 )), \(\widehat{S}\) is the Kaplan-Meier estimator of \(\mathbb{P}(T>t)\) , \(n\) is the number of patients included in the model. Calibration Calibration refers to the model’s ability to match observed and model-based predicted risk, that is, the probability of death. In other words, calibration allows one to assess how closely the predicted risks agree with the observed ones ( 41 ). A prognosis model is considered well calibrated when the number of observed deaths corresponds to the number of deaths predicted by the model. A prognosis model with good calibration is essential to predict a reliable risk. To assess a model’s calibration ( 42 ), we can do a calibration plot, that is we divide the predicted risks in, say, 5 or 10 groups using percentiles of the predicted probabilities of death and compare, for each of these groups, the average of the predicted risk to the observed risk. The different calibration measures in the context of the Cox model can be found in McLernon et al ( 43 ). One may assess the calibration of the model by plotting predicted vs. observed risks: good calibration is achieved when the points lay close to the first diagonal. A complementary way to assess calibration for a prognosis model is to compare Kaplan-Meier survival estimates to model-based prediction of survival curves among pre-defined subgroups as defined by covariate values (for example, dividing the sample into 5 age-groups). Model-based prediction for a given subgroup is obtained by averaging the model-based survival predictions of each individual from that subgroup. Implementation All the implementation has been carried out using the R programming language ( 44 ).The computation of the C-index and the Brier score were already implemented in the package dynpred ( 45 ) for the Cox model, and we adapted these functions for the Bspline-based model and the GH model. We also implemented the calculation of AUC function for these models. These functions are provided in a Github repository “ https://github.com/margueritefournier/Perf_measures_survival_models ”. Results We have used the following covariates in the LeukSurv data set: age at diagnosis, sex, white blood cell-count at diagnosis with \(1 unit=50 \times {10}^{9}/L\) , and Townsend score, which is a measure of deprivation for the enumeration district (small geographical area with mean population of less than a thousand) area of residence. All covariates are continuous except for the variable sex. The maximum follow-up is 13.6 year. There were 1043 patients and 15.7% of censored survival times (Table 1 ). Table 1 Patient’s characteristics at baseline – Leukemia data Overall (N = 1043) Age at diagnosis Mean (SD) 60.7 (18.3) Median [Min, Max] 65.0 [14.0, 92.0] Sex Female 496 (47.6%) Male 547 (52.4%) White blood cell count Mean (SD) 38.6 (72.8) Median [Min, Max] 7.90 [0, 500] Townsend score Mean (SD) 0.340 (3.64) Median [Min, Max] -0.370 [-6.09, 9.55] Survival time (years) Mean (SD) 1.46 (2.46) Median [Min, Max] 0.507 [0.00274, 13.6] Survival status Censored 164 (15.7%) Death 879 (84.3%) We fitted three types of regression models: a Cox model, a B-Spline-based model, and a GH model. For both Bspline-based and GH models, we investigated time-dependent effects for age. We selected the best model between a model with no time-dependent effect of age and a model with time-dependent effect of age using AIC (Akaike Information Criterion). For both Bspline-based and GH models, the selected candidate models included a time-dependent effect for age. For each model type, we computed the predictive measures of overall performance (Brier Score), discrimination ( \(AUC\left(t\right)\) ) and calibration. Figure 1 shows the prediction error curves for the three candidate models. The three models were similar in terms of overall prediction error as evaluated with the time-varying Brier Score. Figure 2 presents the cumulative-dynamic AUC according to time since diagnosis for the three candidate models, where we can see that all models have similar performance in terms of this discrimination measure. Figure 3 shows the calibration-in-the-large at three different times after diagnosis, 6 months, 3 years and 8 years, comparing the observed risks and the predicted risks of 10 groups defined using deciles of predicted risks. The Bspline-based model seemed to have a better calibration with a regression slope of 1.02, as compared to Cox (slope: 1.22) and GH (slope: 1.08) models at 6 months and at 3 years (0.91 vs 0.72 and 0.71, respectively). At 8 years, the GH models has a better regression slope, but all models have a poor calibration (around 0.5). Figure 4 compares the survival as predicted from each model to the non-parametric survival estimated with Kaplan Meier (the “observed” data). All curves were plotted by categories of the variables included in the model. Overall, all models matched well the observed data. Discussion This article aims to explain the definition and the practical use of different performance measures to assess the predictive ability of a prognosis model, with special focus on overall performance, discrimination, and calibration measures. Many tutorials already exist, with focus on binary data, but much less attention has been paid to the context of time-to-event outcomes, especially when regression models other than the Cox PH model (potentially with time-dependent and/or non-linear effect(s)) are of interest. We hope that this tutorial, along with the R-code provided, will help researchers applying these measures when developing a prognosis model. Regarding the overall measures, the Brier score is the most widely used, with its weighted version ( 30 ) to handle right censoring. Discrimination allows one to quantify the ability of a model to correctly classify patients in a risk group. We have favoured the description of the cumulative dynamic AUC because it is the most widely adopted measure for hazard-based regression models ( 38 ), while it has been shown that the C-index is not a proper scoring rule for survival models ( 35 ). Finally, calibration measures help assessing the adequacy between the predicted and observed risk for a given group of individuals. In the line, we have focused on the presentation and use of the calibration plot. We have also presented a graphical comparison between survival curves as estimated non-parametrically in each subgroup vs. the average of the predicted survival curves for all individuals in the same subgroup (while accounting for their observed covariates). A good overlap between the model-based curves and the survival curves estimated non-parametrically suggests a good calibration of the predictive model. In our illustrative example, the predictive performances of the 3 different models were not substantially different, thus suggesting that an increase in flexibility on this dataset does not improve predictions, except for the calibration plot at 3 years for the B-spline model. However, such interpretation should be done with caution because we have built and validated the model using the same dataset. More work should investigate the performances of the 3 models using simulations studies with different scenarios. We have focused on explaining the use and implementation of three types of predictive performance measures. However, predictive modelling also aims at helping clinicians inform and improve their decision-making process. Once the predictive performance of a hazard-based regression model has been assessed, its usefulness to answer the question “would the use of the predictive model improve clinical decision making?” should be evaluated ( 43 , 46 ). This represents an active research area, beyond the scope of this tutorial, where the concepts of “Decision Curve Analysis” and “Net Benefit” have been recently developed ( 43 ). Abbreviations AIC Akaike Information Criterion AH Accelerated Hazards model AFT accelerated failure time model AML Acute Myeloid Leukemia AUC Area Under the ROC BS B-spline GH General Hazard HH Hybrid Hazard model HR Hazard Ratio PGW Power Generalized Weibull PH Proportional Hazards ROC Receiver Operating Characteristic TDS Townsend Deprivation Score WBC White Blood Cell Declarations Ethics approval and consent to participate Not applicable. Consent for publication Not applicable. Competing Interests Authors declare no competing interests. Funding Researchers from the Inequalities in Cancer Outcome Network is funded by Cancer Research UK programme (Grant No. EPNCZS34). Acknowledgements Authors thank the biometry department of the LYSARC for useful discussions during several meetings. Availability of data and materials The dataset used to run the example is freely available in the spBayesSurv R package. Authors' contributions MF and AB developed the concept, designed the first draft of the article and the computing code. MF, FJR, LC, CM, AB interpreted and reviewed the code and the data, drafted and revised the article. MF, FJR, LC, CM, AB read and approved the final version of the article. References Henderson R, Jones M, Stare J. Accuracy of point predictions in survival analysis. Stat Med. 2001;20(20):3083–96. Alba AC, Agoritsas T, Walsh M, Hanna S, Iorio A, Devereaux PJ, et al. Discrimination and Calibration of Clinical Prediction Models: Users’ Guides to the Medical Literature. JAMA. 2017;318(14):1377–84. Moons KGM, Royston P, Vergouwe Y, Grobbee DE, Altman DG. Prognosis and prognostic research: what, why, and how? BMJ. 2009;338:b375. Cox DR. Regression Models and Life-Tables. J Roy Stat Soc. 1972;34(2):187–220. Abrahamowicz M, Mackenzie T, Esdaile JM. Time-Dependent Hazard Ratio: Modeling and Hypothesis Testing with Application in Lupus Nephritis. J Am Stat Assoc. 1996;91(436):1432–9. Sleeper LA, Harrington DP. Regression Splines in the Cox Model with Application to Covariate Effects in Liver Disease. J Am Stat Assoc. 1990;85(412):941–9. Royston P, Parmar MKB. Flexible parametric proportional-hazards and proportional-odds models for censored survival data, with application to prognostic modelling and estimation of treatment effects. Stat Med. 2002;21(15):2175–97. Kooperberg C, Stone CJ, Truong YK. Hazard Regression. J Am Stat Assoc. 1995;90(429):78–94. Hagar Y, Dignam JJ, Dukic V. Flexible modeling of the hazard rate and treatment effects in long-term survival studies. Stat Methods Med Res. 2017;26(5):2455–80. Perperoglou A, Sauerbrei W, Abrahamowicz M, Schmid M. A review of spline function procedures in R. BMC Med Res Methodol. 2019;19(1):46. Giorgi R, Abrahamowicz M, Quantin C, Bolard P, Esteve J, Gouvernet J, et al. A relative survival regression model using B-spline functions to model non-proportional hazards. Stat Med. 2003;22(17):2767–84. Remontet L, Bossard N, Belot A, Estève J. An overall strategy based on regression models to estimate relative survival and model the effects of prognostic factors in cancer survival studies. Stat Med. 2007;26(10):2214–28. Charvat H, Remontet L, Bossard N, Roche L, Dejardin O, Rachet B, et al. A multilevel excess hazard model to estimate net survival on hierarchical data allowing for non-linear and non-proportional effects of covariates. Stat Med. 2016;35(18):3066–84. Chen YQ, Jewell NP. On a general class of semiparametric hazards regression models. Biometrika. 2001;88(3):687–702. Rubio FJ, Remontet L, Jewell NP, Belot A. On a general structure for hazard-based regression models: An application to population-based cancer research. Stat Methods Med Res. 2019;28(8):2404–17. Steyerberg E. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating [Internet]. New York: Springer-Verlag; 2009 [cited 2020 Jan 19]. (Statistics for Biology and Health). Available from: https://www.springer.com/gp/book/9780387772431 . Royston P, Moons KGM, Altman DG, Vergouwe Y. Prognosis and prognostic research: Developing a prognostic model. BMJ. 2009;338:b604. Tournoud M, Larue A, Cazalis MA, Venet F, Pachot A, Monneret G, et al. A strategy to build and validate a prognostic biomarker model based on RT-qPCR gene expression and clinical covariates. BMC Bioinformatics. 2015;16(1):1–15. Steyerberg EW, Vergouwe Y. Towards better clinical prediction models: seven steps for development and an ABCD for validation. Eur Heart J. 2014;35(29):1925–31. Remontet L, Bossard N, Iwaz J, Estève J, Belot A. Framework and optimisation procedure for flexible parametric survival models. Stat Med. 2015;34(25):3376–7. Rubio FJ, Rachet B, Giorgi R, Maringe C, Belot A. On models for the estimation of the excess mortality hazard in case of insufficiently stratified life tables. Biostatistics. 2021;22(1):51–67. Henderson R, Shimakura S, Gorst D. Modeling Spatial Variation in Leukemia Survival Data. 2002. Durrleman S, Simon R. Flexible regression models with cubic splines. Stat Med. 1989;8(5):551–61. Hess KR. Assessing time-by-covariate interactions in proportional hazards regression models using cubic spline functions. Stat Med. 1994;13(10):1045–62. Gauthier J, Wu QV, Gooley TA. Cubic splines to model relationships between continuous variables and outcomes: a guide for clinicians. Bone Marrow Transplant. 2020;55(4):675–80. Jones MC, Noufaily A. Log-location-scale-log-concave distributions for survival and reliability analysis. Electron J Stat. 2015;9(2):2732–50. Graf E, Schmoor C, Sauerbrei W, Schumacher M. Assessment and comparison of prognostic classification schemes for survival data. Stat Med. 1999;18(17–18):2529–45. Pfeiffer RM, Gail MH, Gail MH. Absolute Risk: Methods and Applications in Clinical Management and Public Health [Internet]. Chapman and Hall/CRC; 2017 [cited 2020 Jan 13]. Available from: https://www-taylorfrancis-com.docelec.univ-lyon1.fr/books/9781315117539 . Tsouprou S, Putter H, Fiocco M. Measures of discrimination and predictive accuracy for interval censored survival data [PhD Thesis]. Master’s thesis, Leiden University; 2015. Schumacher M, Graf E, Gerds T. How to assess prognostic models for survival data: a case study in oncology. Methods Inf Med. 2003;42(5):564–71. Harrell FE, Califf RM, Pryor DB, Lee KL, Rosati RA. Evaluating the yield of medical tests. JAMA. 1982;247(18):2543–6. Royston P, Altman DG. Visualizing and assessing discrimination in the logistic regression model. Stat Med. 2010;29(24):2508–20. Harrell FE, Lee KL, Mark DB. Multivariable Prognostic Models: Issues in Developing Models, Evaluating Assumptions and Adequacy, and Measuring and Reducing Errors. Stat Med. 1996;15(4):361–87. Pencina MJ, D’Agostino RB. Overall C as a measure of discrimination in survival analysis: model specific population value and confidence interval estimation. Stat Med. 2004;23(13):2109–23. Blanche P, Kattan MW, Gerds TA. The c-index is not proper for the evaluation of $ t $ -year predicted risks. Biostatistics. 2019;20(2):347–57. Heagerty PJ, Lumley T, Pepe MS. Time-dependent ROC curves for censored survival data and a diagnostic marker. Biometrics. 2000;56(2):337–44. Blanche P, Latouche A, Viallon V. Time-dependent AUC with right-censored data: a survey study. arXiv:12106805 [stat] [Internet]. 2012 Oct 25 [cited 2022 Feb 7]; Available from: http://arxiv.org/abs/1210.6805 . Heagerty. Survival Model Predictive Accuracy and ROC Curves [Internet]. 2005 [cited 2021 Sep 7]. Available from: https://onlinelibrary.wiley.com/doi/ 10.1111/j.0006-341X.2005.030814.x . Hung H, Chiang CT. Estimation methods for time-dependent AUC models with survival data. Can J Stat. 2010;38(1):8–26. Satten GA, Datta S. The Kaplan-Meier Estimator as an Inverse-Probability-of-Censoring Weighted Average. Am Stat. 2001;55(3):207–10. Van Calster B, McLernon DJ, van Smeden M, Wynants L, Steyerberg EW, Bossuyt P, et al. Calibration: the Achilles heel of predictive analytics. BMC Med. 2019;17(1):230. Steyerberg EW, Calster BV, Pencina MJ. Performance Measures for Prediction Models and Markers: Evaluation of Predictions and Classifications. Rev Esp Cardiol. 2011;64(9):788–94. McLernon DJ, Giardiello D, Calster BV, Wynants L, van Geloven N, van Smeden M et al. Assessing performance and clinical usefulness in prediction models with survival outcomes: practical guidance for Cox proportional hazards models [Internet]. medRxiv; 2022 [cited 2022 May 30]. p. 2022.03.17.22272411. Available from: https://www.medrxiv.org/content/ 10.1101/2022.03.17.22272411v1 . Ihaka R, Gentleman R. R: A Language for Data Analysis and Graphics. J Comput Graphical Stat. 1996;5(3):299–314. Van Houwelingen H, Putter H. Dynamic prediction in clinical survival analysis. CRC Press; 2011. Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making. 2006;26(6):565–74. Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-3866618","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":267644717,"identity":"5a6b5b68-fad1-470d-b3c6-2b381223b4dd","order_by":0,"name":"Marguerite Fournier","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABK0lEQVRIie3QMUvEMBQH8DwC6ZJzjiD0EwgWoQ4i/SoJQt1EKBwd5Eg5SBd1rnDgV+lROJfeud50VApddIibyCGmdx2EVrlRsP/h5fHglxeCUJ8+fzi0LrouBCEoil2IMZA0BB/xHddsSa3Yb+QwnpcvNFwdeJYE+aZWl3vWbRUKNbJP4hTK8KNF3PzCPaV5QClNIbpXQUDoo7sUKnMmOcfO/K5NUp8cDxSnlHGrNI1QzCeGpJAwRPajmzZ5qgz5NMQuYLwhdkWuzMO8H8nSx+VA1ltQQxghSCgsNkS+d5CKwGRmSC6iKFkYQn3M+CI7T6gYO5HseJiP9es196w4m0o95OIhnoHWw9FZYmXTZ7nu+mfCtid8uxBIMwHVRbDuGK5bTZ8+ffr833wBbz9o0FTrE50AAAAASUVORK5CYII=","orcid":"","institution":"Lymphoma Study Association Clinical Research (LYSARC), Hopital Lyon Sud","correspondingAuthor":true,"prefix":"","firstName":"Marguerite","middleName":"","lastName":"Fournier","suffix":""},{"id":267644718,"identity":"8a3a3b63-e02f-4f82-9117-3a497301222d","order_by":1,"name":"Francisco Javier Rubio","email":"","orcid":"","institution":"University College London","correspondingAuthor":false,"prefix":"","firstName":"Francisco","middleName":"Javier","lastName":"Rubio","suffix":""},{"id":267644719,"identity":"20f1c53f-0b69-4796-8f0c-378ed30dbae7","order_by":2,"name":"Loïc Chartier","email":"","orcid":"","institution":"Lymphoma Study Association Clinical Research (LYSARC), Hopital Lyon Sud","correspondingAuthor":false,"prefix":"","firstName":"Loïc","middleName":"","lastName":"Chartier","suffix":""},{"id":267644720,"identity":"9cd911a9-e14c-48a7-af6b-54a809da1b0a","order_by":3,"name":"Camille Maringe","email":"","orcid":"","institution":"London School of Hygiene \u0026 Tropical Medicine","correspondingAuthor":false,"prefix":"","firstName":"Camille","middleName":"","lastName":"Maringe","suffix":""},{"id":267644721,"identity":"29ea273d-e047-402d-bfbf-0c91862400c7","order_by":4,"name":"Aurélien Belot","email":"","orcid":"","institution":"Lymphoma Study Association Clinical Research (LYSARC), Hopital Lyon Sud","correspondingAuthor":false,"prefix":"","firstName":"Aurélien","middleName":"","lastName":"Belot","suffix":""}],"badges":[],"createdAt":"2024-01-15 13:29:13","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-3866618/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-3866618/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":49825583,"identity":"a1c72a5f-d872-4225-830c-ad5be731fa6a","added_by":"auto","created_at":"2024-01-18 15:44:44","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":10809,"visible":true,"origin":"","legend":"\u003cp\u003ePrediction error curves for each model as estimated with the Brier score\u003c/p\u003e","description":"","filename":"Onlinefloatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-3866618/v1/09df76983769c1dd113c763f.png"},{"id":49826572,"identity":"560e3b8c-e14a-48e3-9271-7d81b10cad70","added_by":"auto","created_at":"2024-01-18 15:52:44","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":10976,"visible":true,"origin":"","legend":"\u003cp\u003eCumulative Dynamic AUC(t) estimated for the three models (BS model = Bspline-based model, and GH Model = General Hazard structure model)\u003c/p\u003e","description":"","filename":"Onlinefloatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-3866618/v1/e985ac4634f5a4a9bcc6f526.png"},{"id":49825586,"identity":"86c19644-c9b1-4a3e-b4eb-f5d5128b6de7","added_by":"auto","created_at":"2024-01-18 15:44:44","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":44155,"visible":true,"origin":"","legend":"\u003cp\u003eCalibration for each model (in column) and at 6 months (1\u003csup\u003est\u003c/sup\u003e row of graphs), 3 years (2\u003csup\u003end\u003c/sup\u003e row) and 8 years (3\u003csup\u003erd\u003c/sup\u003e row) after diagnosis: the dots represent the estimates for each subgroup as predicted by the model investigated, and the grey line represents the first diagonal (y=x) on each plot.\u003c/p\u003e","description":"","filename":"Onlinefloatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-3866618/v1/8590963b4044ed119b40079d.png"},{"id":49826573,"identity":"6723dc18-a84c-432e-94b7-5050d40aa254","added_by":"auto","created_at":"2024-01-18 15:52:44","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":18966,"visible":true,"origin":"","legend":"\u003cp\u003eCalibration by comparing model-based survival prediction with Kaplan-Meier estimates in different subgroups (WBC=White Blood Cell count at diagnosis, TDS=Townsend Deprivation Score)\u003c/p\u003e","description":"","filename":"Onlinefloatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-3866618/v1/15c2ae30c9883c5077932ded.png"},{"id":56186691,"identity":"745509e8-209a-40dc-a084-c7bb3aadda5b","added_by":"auto","created_at":"2024-05-09 15:38:01","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":976708,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3866618/v1/add8d789-c597-4ed4-9c22-85bf329372aa.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Assessing predictive abilities of hazard-based regression models for survival data: a tutorial for prognosis modelling","fulltext":[{"header":"Introduction","content":"\u003cp\u003ePredicting the occurrence of a time-to-event of interest since cancer diagnosis is an important part of medical research in oncology (\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e) (\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e) (\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e). Predictions are typically based on survival regression models formulated in terms of the hazard function. The main goal of these models is to equip physicians and their patients with accurate tools to develop an optimal care strategy based on the available information from patients (such as age, sex, cancer stage, co-morbidities, frailty, et cetera). The semi-parametric Cox model (\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e) is the most popular model in time-to-event data analysis. The Cox model has many advantages, such as being easily interpretable and relying on a semi-parametric structure as the baseline hazard is not specified (and therefore not forced to have a particular shape). However, to predict survival probabilities from the Cox model, the baseline hazard function needs to be estimated. Moreover, the classical Cox model relies on the proportional hazards (PH) assumption, where the hazard ratio (HR) associated with a covariate is assumed to be constant over time. This assumption may be reasonable only for a short follow-up period after diagnosis. Consequently, the Cox model may be unable to describe more complex scenarios where the proportionality of hazards assumption is not reasonable. Several regression models have been proposed to relax the PH assumption and thus allow for non-proportional hazards. For instance, alternative regression models have been proposed using flexible functions for modelling the baseline hazard and/or time-dependent regression coefficients for covariates (\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e)(\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e)(\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e)(\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e)(\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e). One possible choice for such flexible functions are spline functions (\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e), in particular B-splines (\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e). Models based on B-splines allow great flexibility for modelling both the baseline hazard and the covariate effects (\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e)(\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e). Another class of flexible parametric models has recently been proposed based on a general hazard structure (14)(\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e). This general structure contains the proportional hazards and accelerate failure time as particular cases. This model is able to incorporate time-level effects, while avoiding the need for numerical integration, and flexibly modelling the baseline hazard using a flexible parametric distribution (e.g., the Power Generalized Weibull). As mentioned in a book by E. Steyerberg (\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e), flexible models might be better for fitting particular patterns in a specific dataset at the increased risk of overfitting. It is therefore not immediately clear whether a more flexible model is to be preferred in comparison of a simpler model.\u003c/p\u003e \u003cp\u003eOnce a predictive hazard-based regression model has been built or selected (\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e), it is necessary to quantify the accuracy of predictions derived from that model (\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e). Assessing a model\u0026rsquo;s predictive ability can be performed by using (i) overall performance measures of model predictive accuracy, (ii) discrimination measures to assess the ability of the model to classify patients in groups with homogeneous risk levels, and (iii) calibration measures, which describe the agreement between the observed and the predicted risk (\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e)(\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e). Their combination is important as a model may exhibit good classification performance (discrimination), but it may also have poor performance in terms of predictions related to specific subgroups (calibration), or vice versa.\u003c/p\u003e \u003cp\u003eOur main aim is to illustrate the use of prediction performance measures for two types of parametric hazard-based regression models applied on time-to-event data, in addition to the classical semiparametric Cox PH model. A secondary aim is to investigate the trade-off between flexibility and predictive performance. That is, whether the cost of using a more flexible model is justified by a considerable increase in predictive performance. We focus on two types of flexible models: (i) one based on splines and a classical hazard structure (i.e. where the baseline hazard is multiplied by the exponential of a linear predictor, which may include time-dependent effects) (\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e)(\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e)(\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e), and (ii) another one based on a general hazard structure (14) (\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e) (\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e), where the covariates play a role at both the hazard level and the time scale (time-dependent effects).\u003c/p\u003e \u003cp\u003eThe remainder of the paper is organized as follows. In Section 2, we describe the dataset used in our illustrative example. In Section 3, we describe the hazard regression models to be used in our examples as well as the performance measures used for evaluating their predictive abilities. Section 4 presents the results for the model selection strategy and the prediction measures computed for the different candidate models. We conclude with a discussion of the results and potential research directions.\u003c/p\u003e"},{"header":"Material: The LeukSurv dataset","content":"\u003cp\u003eThis dataset is freely available in the \u003cem\u003espBayesSurv\u003c/em\u003e R package and it was first analyzed by Henderson et al. (2002) (\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e). It includes 1043 patients diagnosed with acute myeloid leukemia (AML), with the time until death or last known follow-up \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(T\\)\u003c/span\u003e\u003c/span\u003e (in years) and the corresponding event indicator \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\delta\\)\u003c/span\u003e\u003c/span\u003e (being 0 for censored patients and 1 for death). We aim to predict the survival after AML diagnosis according to the following covariates: age (continuous) at diagnosis, sex, white blood cell count (WBC) at diagnosis (\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(50*{10}^{9}/L\\)\u003c/span\u003e\u003c/span\u003e) (continuous) and the Townsend deprivation score (continuous) which is an area-based measure of deprivation for the enumeration district of residence (higher values indicating more deprived areas).\u003c/p\u003e\n\u003ch3\u003eMethod\u003c/h3\u003e\n\u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003eHazard regression models\u003c/h2\u003e \u003cp\u003eIn this section, we present the two regression models that we are considering as potential alternative to the Cox PH model.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003eFlexible hazard-based regression model using regression splines\u003c/h2\u003e \u003cp\u003eThis model uses regression spline functions to model the baseline hazard function and (possibly) time-dependent and/or non-linear effect(s) of covariates. Spline functions are flexible mathematical functions defined by D-degree polynomial segments, which are joined at K junction points called knots (\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e)(\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e)(\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e)(\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e)(\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e). A simple example with 2 covariates \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({(x}_{1},{x}_{2})\\)\u003c/span\u003e\u003c/span\u003e, where the variable \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({x}_{2}\\)\u003c/span\u003e\u003c/span\u003e is assumed to be a continuous variable (such as age), helps to illustrate the parametrisation of the model for the hazard\u003cdiv id=\"Equa\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equa\" name=\"EquationSource\"\u003e\n$$\\lambda \\left(t|{x}_{1},{x}_{2}\\right)= {\\lambda }_{0}\\left(t\\right)*\\text{exp}\\left({\\beta }_{1}{x}_{1}+ g\\left(t\\right)*{x}_{2}+j\\left({x}_{2}\\right)\\right),$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\beta }_{1}\\)\u003c/span\u003e\u003c/span\u003eis the regression coefficient for \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({x}_{1}\\)\u003c/span\u003e\u003c/span\u003e; while \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(g\\left(t\\right)\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(j\\left({x}_{2}\\right)\\)\u003c/span\u003e\u003c/span\u003e and the log baseline-hazard \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\text{l}\\text{o}\\text{g}(\\lambda }_{0}\\left(t\\right))\\)\u003c/span\u003e\u003c/span\u003eare B-splines functions. The degree of the spline (D) as well as the number and the locations of the (K) knots need to be specified by the user and may differ for \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(g\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(j\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\lambda }_{0}\\)\u003c/span\u003e\u003c/span\u003e.\u003c/p\u003e \u003cp\u003eThe use of flexible functions to describe baseline hazard, non-proportional effect of variables and non-linear functional form for continuous variables, allows for modelling smooth and more realistic hazard shapes while ensuring parsimony for the model. One drawback of this model is that we need to determine the number and specify the position of the knots for the spline functions. To have enough flexibility, we choose a cubic B-spline with one knot located at the median of the observed event times for the spline functions of time (i.e., the baseline hazard \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\lambda }_{0}\\left(t\\right)\\)\u003c/span\u003e\u003c/span\u003e and time-dependent effect \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(g\\left(t\\right)\\)\u003c/span\u003e\u003c/span\u003e). This knot location ensures to have the same quantity of information on both sides of the knot. For the non-linear functional form of continuous variable, the user may choose another type of spline, as for example a restricted cubic spline with 2 knots located at the 33rd and 66th quantile of the distribution of \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({x}_{2}\\)\u003c/span\u003e\u003c/span\u003e. The more knots there are, the greater the flexibility of the model but at a higher risk of overfitting.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003eGeneral hazard structure model with a flexible baseline hazard\u003c/h2\u003e \u003cp\u003eThis general hazard (GH) structure (15) represents a rich class of hazard-based regression models, which is formulated in terms of the hazard function (exemplified with two covariates \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({x}_{1}\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({x}_{2}\\)\u003c/span\u003e\u003c/span\u003e here):\u003cdiv id=\"Equb\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equb\" name=\"EquationSource\"\u003e\n$${\\lambda }^{GH}\\left(t;{x}_{1},{x}_{2}\\right)= {\\lambda }_{0}\\left(t\\text{exp}\\left({\\beta }_{11}{x}_{1}+{\\beta }_{12}{x}_{2}\\right)\\right)\\text{exp}\\left({{\\beta }_{21}x}_{1}+{\\beta }_{22}{x}_{2}\\right),$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\lambda }_{0}\\left(t\\right)\\)\u003c/span\u003e\u003c/span\u003e is the baseline hazard function, which can be modelled using a pre-specified parametric distribution (see below), {\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\beta }_{11}, {\\beta }_{21}\\}\\)\u003c/span\u003e\u003c/span\u003e and {\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\beta }_{12}, {\\beta }_{22}\\}\\)\u003c/span\u003e\u003c/span\u003e are regression coefficients of the variables \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({x}_{1}\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({x}_{2}\\)\u003c/span\u003e\u003c/span\u003e, respectively. The parameters \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\{\\beta }_{11}, {\\beta }_{12}\\}\\)\u003c/span\u003e\u003c/span\u003eallow for the inclusion of time-level effects of the covariates \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({x}_{1}\\)\u003c/span\u003e\u003c/span\u003eand \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({x}_{2}\\)\u003c/span\u003e\u003c/span\u003e.\u003c/p\u003e \u003cp\u003eThe GH model includes as special cases: the proportional hazards model (PH, when \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\beta }_{1j}=0\\)\u003c/span\u003e\u003c/span\u003e), the accelerated failure time model (AFT, when \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\beta }_{1j}= {\\beta }_{2j}\\)\u003c/span\u003e\u003c/span\u003e), the accelerated hazards model (AH, when \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\beta }_{2j}=0\\)\u003c/span\u003e\u003c/span\u003e) and the Hybrid Hazard model (HH, when different covariates are included in the argument of the baseline hazard and those in the argument of the exponential function multiplying the baseline hazard). The GH model is identifiable provided that the baseline hazard is not the hazard function associated with the Weibull distribution (\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eTo complete the specification of the GH model, we need to select a parametric distribution for the baseline hazard. Here, we will use the Power Generalized Weibull (PGW) distribution instead of Exponentiated Weibull distribution originally proposed (\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e), as it is easier to implement while being equally flexible (\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e). The PGW distribution is a 3-parameter distribution which is defined according to the following equations for the density, survival, and hazard functions:\u003cdiv id=\"Equc\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equc\" name=\"EquationSource\"\u003e\n$$f\\left(t;\\sigma ,\\nu ,\\gamma \\right)=\\frac{\\nu }{{\\gamma \\sigma }^{\\nu }}{t}^{\\nu -1}{\\left[1+{\\left(\\frac{t}{\\sigma }\\right)}^{\\nu }\\right]}^{\\left(\\frac{1}{\\gamma }-1\\right)}exp\\left\\{1-{\\left[1+{\\left(\\frac{t}{\\sigma }\\right)}^{\\nu }\\right]}^{\\frac{1}{\\gamma }}\\right\\}$$\u003c/div\u003e\u003c/div\u003e,\u003cdiv id=\"Equd\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equd\" name=\"EquationSource\"\u003e\n$$S\\left(t;\\sigma ,\\nu ,\\gamma \\right)= exp\\left\\{1-{\\left[1+{\\left(\\frac{t}{\\sigma }\\right)}^{\\nu }\\right]}^{\\frac{1}{\\gamma }}\\right\\}$$\u003c/div\u003e\u003c/div\u003e,\u003cdiv id=\"Eque\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Eque\" name=\"EquationSource\"\u003e\n$$\\lambda \\left(t;\\sigma ,\\nu ,\\gamma \\right)=\\frac{\\nu }{{\\gamma \\sigma }^{\\nu }}{t}^{\\nu -1}{\\left[1+{\\left(\\frac{t}{\\sigma }\\right)}^{\\nu }\\right]}^{\\left(\\frac{1}{\\gamma }-1\\right)}$$\u003c/div\u003e\u003c/div\u003e ,\u003c/p\u003e \u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(f\\left(t;\\sigma ,\\nu ,\\gamma \\right)\\)\u003c/span\u003e\u003c/span\u003e is the probability density function, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(S\\left(t;\\sigma ,\\nu ,\\gamma \\right)\\)\u003c/span\u003e\u003c/span\u003e is the survival function and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\lambda \\left(t;\\sigma ,\\nu ,\\gamma \\right)\\)\u003c/span\u003e\u003c/span\u003e is the hazard function, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\sigma \u0026gt;0\\)\u003c/span\u003e\u003c/span\u003e is a scale parameter and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\nu ,\\gamma \u0026gt;0\\)\u003c/span\u003e\u003c/span\u003eare shape parameters.\u003c/p\u003e \u003cp\u003eThe hazard function associated with the PGW distribution can capture several shapes: constant, increasing, decreasing, bathtub and unimodal, which are the shapes usually observed in cancer survival applications (\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e). Implementations of the GH model with PGW baseline hazard (among others) can also be found in the R packages GHSurv and HazReg.\u003c/p\u003e \u003c/div\u003e\n\u003ch3\u003eModel-building strategy\u003c/h3\u003e\n\u003cp\u003eTo select between different regression models within a given class of model, (i.e., spline-based model or GH model), we use the Akaike Information Criterion (AIC). We start by fitting several pre-specified candidate models, based on clinical knowledge of the disease, defined with or without time-dependent effect for some covariates (specified in the result section, and then we choose the model with the smallest AIC among the set of candidate models. For comparison purposes, we also fit the Cox model assuming proportional hazards for each covariate in the model.\u003c/p\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003eEvaluating the predictive abilities of hazard-based regression models for survival data\u003c/h2\u003e \u003cp\u003eSeveral performance measures have been developed to compare regression models in terms of their ability to predict individual survival probabilities at a pre-specified time horizon \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({t}^{*}\\)\u003c/span\u003e\u003c/span\u003e. In this paper, because we aimed to focus on describing the predictive performance measures, we assume that the predictive models have been built based on a different dataset and they are then validated on the LeukSurv dataset. If the model was built and also validated on the same dataset, specific techniques need to be used such as cross-validation (\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e) to account for the optimism of the performance measures\u003c/p\u003e \u003cp\u003eThe performance measures can be broadly classified into three classes (\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e)(\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e): overall performance measures, discrimination measures and calibration measures. Overall performance measures estimate the accuracy of prediction at time \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003eby calculating the difference between predicted and observed outcomes for all individuals at time \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003eusing a loss function (that is, the \u0026ldquo;prediction error\u0026rdquo;). Discrimination measures reflect the model\u0026rsquo;s ability to separate individuals who experienced the event from those who did not. Calibration measures compare the survival predictions for specific subgroups to their observed survival. We describe these performance measures in some details before we use them in our illustrative example.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec9\" class=\"Section2\"\u003e \u003ch2\u003eOverall performance measure: prediction error estimated with the Brier Score\u003c/h2\u003e \u003cp\u003eThe Brier Score at time \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e (\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e) measures the mean quadratic difference between the predicted survival probability for subject \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(i\\)\u003c/span\u003e\u003c/span\u003e at time \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e and the observed event status for the same subject \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(i\\)\u003c/span\u003e\u003c/span\u003e at time \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e. It quantifies the degree to which the predicted values coincide with the observed outcomes. Without censoring it can be estimated at time \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e by (\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e)(\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e)\u003cdiv id=\"Equf\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equf\" name=\"EquationSource\"\u003e\n$$\\widehat{BS\\left(t\\right)}= \\frac{1}{N}{\\sum _{i=1}^{N}(\\mathbb{l}\\left\\{{T}_{i} \u0026gt; t\\right\\}-{\\widehat{S}}_{i}\\left(t\\right))}^{2},$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\widehat{S}}_{i}\\left(t\\right)\\)\u003c/span\u003e\u003c/span\u003eis the model-based predicted probability of surviving beyond time \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e for individual \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(i\\)\u003c/span\u003e\u003c/span\u003e, and the indicator \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\mathbb{l}\\left\\{{T}_{i} \u0026gt; t\\right\\}\\)\u003c/span\u003e\u003c/span\u003eis equal to 1 when patient \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(i\\)\u003c/span\u003e\u003c/span\u003e is known to be alive at time \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e and is equal to 0 if patient \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(i\\)\u003c/span\u003e\u003c/span\u003e died before or at \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e.\u003c/p\u003e \u003cp\u003eWhen the sample contains right-censored observations, the indicator \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\mathbb{l}\\left\\{{T}_{i} \u0026gt; t\\right\\}\\)\u003c/span\u003e\u003c/span\u003e cannot always be computed. Indeed, if patient \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(i\\)\u003c/span\u003e\u003c/span\u003e is censored before\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\mathbb{l}\\left\\{{T}_{i} \u0026gt; t\\right\\}\\)\u003c/span\u003e\u003c/span\u003e is unknown because \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({T}_{i}\\)\u003c/span\u003e\u003c/span\u003eis not observed. A proposed solution to handle right censoring is based on weighting each contribution to the Brier Score (\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e), assuming that censored patients can be represented by patients with complete information. These weights are defined through the Kaplan-Meier estimates of the conditional survival function of the censoring times, which therefore represents the probability to be uncensored at time \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\widehat{{S}_{C}}\\left(t\\right)=\\widehat{P}(C\u0026gt;t)\\)\u003c/span\u003e\u003c/span\u003e. Therefore, an estimator of the Brier Score that incorporates censoring (assuming random censoring) is given by:\u003cdiv id=\"Equg\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equg\" name=\"EquationSource\"\u003e\n$$\\widehat{BS\\left(t\\right)}=\\frac{1}{N}\\sum _{i=1}^{N}\\left[\\frac{{\\mathbb{I}}_{\\left({\\stackrel{\\sim}{T}}_{i}\\le t,{\\delta }_{i}=1\\right)}}{\\widehat{{S}_{C}}\\left({\\stackrel{\\sim}{T}}_{i}\\right)}{\\left(\\mathbb{l}\\left\\{{\\stackrel{\\sim}{T}}_{i} \u0026gt; t\\right\\}-{\\widehat{S}}_{i}\\left(t\\right)\\right)}^{2}+\\frac{{\\mathbb{I}}_{\\left({\\stackrel{\\sim}{T}}_{i}\u0026gt;t\\right)}}{\\widehat{{S}_{C}}\\left(t\\right)}{\\left(\\mathbb{l}\\left\\{{\\stackrel{\\sim}{T}}_{i} \u0026gt; t\\right\\}-{\\widehat{S}}_{i}\\left(t\\right)\\right)}^{2}\\right],$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\stackrel{\\sim}{T}}_{ }=\\text{min}\\left(T,C\\right)\\)\u003c/span\u003e\u003c/span\u003e with C is the right-censoring time and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\delta\\)\u003c/span\u003e\u003c/span\u003e is the event indicator (equals to 1 if \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\stackrel{\\sim}{T}=T}_{ }\\)\u003c/span\u003e\u003c/span\u003e and 0 if \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\stackrel{\\sim}{T}=C}_{ }\\)\u003c/span\u003e\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eThe contribution of patients who had the event before time \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e is\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({ \\left(0-{\\widehat{S}}_{i}\\left(t\\right)\\right)}^{2}\\)\u003c/span\u003e\u003c/span\u003e, and this contribution is weighted by the inverse probability to be uncensored at time\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\stackrel{\\sim}{T}}_{i}\\)\u003c/span\u003e\u003c/span\u003e (first term in the equation above). The contribution of patients who are still at risk at time \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e is\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\left(1-{\\widehat{S}}_{i}\\left(t\\right)\\right)}^{2}\\)\u003c/span\u003e\u003c/span\u003e, which is again weighted by the inverse probability of being uncensored at time \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e (second term in the equation).\u003c/p\u003e \u003cp\u003eIf the probability to be uncensored beyond \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\stackrel{\\sim}{T}}_{i}\\)\u003c/span\u003e\u003c/span\u003e or \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e is close to 1, then the weight is slightly bigger than 1. Conversely, if this probability is low, meaning that many patients have been censored before time \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\stackrel{\\sim}{T}}_{i}\\)\u003c/span\u003e\u003c/span\u003e or \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e, the weight is far bigger than 1, thus upweighting the difference calculated on patients still under observation. Notice that patients who are censored before time \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e contribute to none of the 2 terms detailed just above (both indicators \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\mathbb{I}}_{\\left({\\stackrel{\\sim}{T}}_{i}\\le t,{\\delta }_{i}=1\\right)}\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\mathbb{I}}_{\\left({\\stackrel{\\sim}{T}}_{i}\\ge t\\right)}\\)\u003c/span\u003e\u003c/span\u003e are equal to 0 for them); but censored patients contribute through the estimation of the censoring distribution \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\widehat{{S}_{C}}\\)\u003c/span\u003e\u003c/span\u003e. The closer the Brier Score is to 0, the better the overall model accuracy.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec10\" class=\"Section2\"\u003e \u003ch2\u003eDiscrimination\u003c/h2\u003e \u003cp\u003eIntuitively, we can say that models that are able to distinguish between patients who die shortly after diagnosis from those who survive longer are well discriminated.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003eThe C-Index\u003c/h2\u003e \u003cp\u003eThe C-Index (a.k.a Concordance Index) (\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e)(\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e)(\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e) is a commonly used discrimination measure. In the context of the Cox PH model, the C-Index indicates the probability that a subject \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(i\\)\u003c/span\u003e\u003c/span\u003e has an estimated Cox model linear predictor higher than subject \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(j\\)\u003c/span\u003e\u003c/span\u003e, given that subject \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(i\\)\u003c/span\u003e\u003c/span\u003e died before subject \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(j\\)\u003c/span\u003e\u003c/span\u003e and with subjects \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(i\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(j\\)\u003c/span\u003e\u003c/span\u003e randomly selected (\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e). In the Cox PH model, the linear predictor \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\eta\\)\u003c/span\u003e\u003c/span\u003e is the quantity involving the regression coefficients and the covariates in the equation of the regression hazard modelling, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\lambda \\left(t;X\\right)={\\lambda }_{0}\\left(t\\right){exp}\\left({X}^{T}\\beta \\right)\\)\u003c/span\u003e\u003c/span\u003e (that is, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\eta ={X}^{T}\\beta\\)\u003c/span\u003e\u003c/span\u003e), where \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(X\\)\u003c/span\u003e\u003c/span\u003e is the vector of covariates.\u003c/p\u003e \u003cp\u003eThe use of the linear predictor from a Cox PH model uniquely determines the ranking of the risks between two individuals at any time \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e since no time-dependent regression coefficient are involved. When one uses a more general model, where one (or more) time-dependent regression coefficients are included, then the ranking is not preserved over time. Then, one can use the predicted risk (\u003cem\u003ei.e.\u003c/em\u003e \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\widehat{Risk}\\left({t}^{*},i\\right)= 1-\\widehat{S}\\left({t}^{*};{X}_{i}\\right)\\)\u003c/span\u003e\u003c/span\u003e) instead of the linear predictor to calculate the C-index,\u003cdiv id=\"Equh\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equh\" name=\"EquationSource\"\u003e\n$$C= \\text{Prob}(\\widehat{Risk}\\left({t}^{*},i\\right)\u0026gt;\\widehat{Risk}\\left({t}^{*},j\\right)\\mid individual i had the event before j)$$\u003c/div\u003e\u003c/div\u003e,\u003c/p\u003e \u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\widehat{Risk}\\left({t}^{*},i\\right)\\)\u003c/span\u003e\u003c/span\u003e is the estimated risk at time horizon \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({t}^{*}\\)\u003c/span\u003e\u003c/span\u003e for individual \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(i\\)\u003c/span\u003e\u003c/span\u003e derived from the prognosis model under study.\u003c/p\u003e \u003cp\u003eFor a Cox PH model, the ordering of the predicted risks \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\widehat{Risk}\\left({t}^{*},i\\right), i=1,\\dots ,N\\)\u003c/span\u003e\u003c/span\u003e will remain the same for any time horizon \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({t}^{*}\\)\u003c/span\u003e\u003c/span\u003e, while the ordering might change using a model that includes time-dependent regression coefficient(s).\u003c/p\u003e \u003cp\u003eCalculating the C-index is equivalent to calculate the proportion of concordant pairs among all usable pairs (\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e). One pair is concordant when the predicted risk of an event is lower for subject \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(j\\)\u003c/span\u003e\u003c/span\u003e who will have the event at later point compared to subject \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(i\\)\u003c/span\u003e\u003c/span\u003e. A concordance probability of 1 means that the model has a perfect discrimination, while a value of 0.5 indicates that the model provides no more information than random ordering.\u003c/p\u003e \u003cp\u003eIn the absence of censoring, it is easier to calculate the C-index as all the required quantities can be calculated straightforwardly. However, the presence of censoring is more the rule than the exception in most real-life applications in survival analysis, which complicates the estimation of the C-index. Indeed, it is impossible to order the event times of interest (e.g., death) for pairs of patients if one event of a pair is censored. These pairs of observations are discarded, as they are non-informative for the estimation of the C-index. Therefore, the C-index is affected by the proportion of censored observations. Moreover, it has been shown that the C-index is not proper when interest lies in evaluating the risk of an event after \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({t}^{*}\\)\u003c/span\u003e\u003c/span\u003e-years (\u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e).\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003eThe time-dependent Area Under the Receiver Operating Characteristic Curve\u003c/h2\u003e \u003cp\u003eAnother tool for assessing discrimination is the time-dependent Area Under the Receiver Operating Characteristic (ROC) Curve, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(AUC\\left(t\\right)\\)\u003c/span\u003e\u003c/span\u003e (\u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eA ROC curve at a specific time point \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(ROC\\left(t\\right)\\)\u003c/span\u003e\u003c/span\u003e, displays the \u0026ldquo;true positive rate\u0026rdquo; (sensitivity) on the y-axis according to the \u0026ldquo;false positive rate\u0026rdquo; (i.e., 1-specificity) on the x-axis for successive threshold values. Each threshold value is used in turn to classify the patients into two groups at time \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e: those predicted to have experienced the event of interest at \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e among those considered as cases, and those predicted to not experience the event at \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e among controls, based on their marker \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(Q\\)\u003c/span\u003e\u003c/span\u003e. The Area Under the ROC Curve, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(AUC\\left(t\\right)\\)\u003c/span\u003e\u003c/span\u003e represents the probability that a marker or a model-based quantity, denoted \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({Q}_{i}\\)\u003c/span\u003e\u003c/span\u003e, of a randomly selected subject \u003cem\u003ei\u003c/em\u003e among cases is greater than \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({Q}_{j}\\)\u003c/span\u003e\u003c/span\u003e of a randomly selected subject \u003cem\u003ej\u003c/em\u003e among controls (by convention, we assume that a greater \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(Q\\)\u003c/span\u003e\u003c/span\u003e leads to a higher risk). The closer the \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(AUC\\left(t\\right)\\)\u003c/span\u003e\u003c/span\u003e is to 1, the better the discrimination.\u003c/p\u003e \u003cp\u003eThe way we define patients as cases or controls is important for clear definitions of sensitivity and specificity (\u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e). We propose to define cases at time \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e as patients who have experienced the event of interest prior to time \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e, \u003cem\u003ei.e.\u003c/em\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(D\\left(t\\right)=1\\)\u003c/span\u003e\u003c/span\u003e if \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(T\\le t\\)\u003c/span\u003e\u003c/span\u003e \u0026amp; \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\delta =1\\)\u003c/span\u003e\u003c/span\u003e, and we define controls at time \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e as patients who have not yet had the event at time \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e, \u003cem\u003ei.e.\u003c/em\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(D\\left(t\\right)=0\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(T\u0026gt;t\\)\u003c/span\u003e\u003c/span\u003e. From the terminology proposed by Heagerty and Zheng (\u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e38\u003c/span\u003e), this definition corresponds to the cumulative case/dynamic control \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(AUC\\left(t\\right)\\)\u003c/span\u003e\u003c/span\u003e, and it will be the focus of the remaining part. Alternatively, one could use incident case and static control or incidence case/dynamic control, but this AUC seems to be inaccurate to evaluate model performance. More details on these definitions can be found in (\u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e) .\u003c/p\u003e \u003cp\u003eIf we assume a sample without censoring, the steps to estimate the \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(AUC\\left(t\\right)\\)\u003c/span\u003e\u003c/span\u003e will be as follows. We start with a threshold value of \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({c}_{1}\\)\u003c/span\u003e\u003c/span\u003e. Among those who have had the event prior to time \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e, we classify patients in the group of \u0026ldquo;patients who will have the event prior to time \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e as predicted from the prognosis model\u0026rdquo; if \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(Q\u0026gt;{c}_{1}\\)\u003c/span\u003e\u003c/span\u003e, and this will be used to estimate the sensitivity, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(sen\\left({c}_{1},t\\right)= P\\left\\{Q\u0026gt;{c}_{1} \\right| D\\left(t\\right)=1\\}\\)\u003c/span\u003e\u003c/span\u003e. Among those who have not had the event at time \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e, we classify patients in the group of \u0026ldquo;patients who will not have the event at time \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e as predicted from the prognosis model\u0026rdquo; if \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(Q\\le {c}_{1}\\)\u003c/span\u003e\u003c/span\u003e, and this will be used to estimate the specificity, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(spe\\left({c}_{1},t\\right)= P\\left\\{Q\\le {c}_{1} \\right| D\\left(t\\right)=0\\}\\)\u003c/span\u003e\u003c/span\u003e. We can then repeat these steps with different thresholds to obtain a ROC curve at time \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e and calculate \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(AUC\\left(t\\right)\\)\u003c/span\u003e\u003c/span\u003e. We can then obtain the ROC curve for any time \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(t\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(ROC\\left(t\\right)\\)\u003c/span\u003e\u003c/span\u003e and so the \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(AUC\\left(t\\right)\\)\u003c/span\u003e\u003c/span\u003e.\u003c/p\u003e \u003cp\u003eIn the definitions above, the quantity \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(Q\\)\u003c/span\u003e\u003c/span\u003e can be a model-based quantity, such as the predicted value of the linear predictor \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(X\\widehat{\\beta }\\)\u003c/span\u003e\u003c/span\u003e derived from the prognosis model. It may also be a time-varying quantity, such as \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(Q\\left(t\\right)=X\\widehat{\\beta \\left(t\\right)}\\)\u003c/span\u003e\u003c/span\u003e or \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(Q\\left(t\\right)= 1-\\widehat{S}\\left(t;X\\right)\\)\u003c/span\u003e\u003c/span\u003e: that will prove useful when a general model with time-dependent regressions coefficient(s) has been fitted.\u003c/p\u003e \u003cp\u003eWe use the estimator of cumulative dynamic AUC as detailed in \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\left(37\\right)\\)\u003c/span\u003e\u003c/span\u003e,\u003cdiv id=\"Equi\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equi\" name=\"EquationSource\"\u003e\n$${\\widehat{AUC}}^{C,D}\\left(\\text{t}\\right)= \\frac{\\sum _{i=1}^{n}\\sum _{j=1}^{n}\\mathbb{I}\\left( {T}_{i}\\le t\\right)\\mathbb{I}\\left( {T}_{j}\u0026gt;t\\right)\\mathbb{I}\\left({Q}_{i}\\left(t\\right)\u0026gt;{Q}_{j}\\left(t\\right)\\right)* \\frac{{\\delta }_{i}}{\\widehat{{S}_{C}}\\left({T}_{i}\\right)\\widehat{{S}_{C}}\\left(t\\right)}}{{n}^{2}\\widehat{S}\\left(t\\right)\\left[1-\\widehat{S}\\left(t\\right)\\right]},$$\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\widehat{{S}_{C}}\\)\u003c/span\u003e\u003c/span\u003e is the Kaplan-Meier estimator of the survival function of the censoring time C (thus allowing to deal with censored observations through Inverse Probability of Censoring Weights (\u003cspan citationid=\"CR40\" class=\"CitationRef\"\u003e40\u003c/span\u003e)), \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\widehat{S}\\)\u003c/span\u003e\u003c/span\u003e is the Kaplan-Meier estimator of \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\mathbb{P}(T\u0026gt;t)\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(n\\)\u003c/span\u003e\u003c/span\u003e is the number of patients included in the model.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003eCalibration\u003c/h2\u003e \u003cp\u003eCalibration refers to the model\u0026rsquo;s ability to match observed and model-based predicted risk, that is, the probability of death. In other words, calibration allows one to assess how closely the predicted risks agree with the observed ones (\u003cspan citationid=\"CR41\" class=\"CitationRef\"\u003e41\u003c/span\u003e). A prognosis model is considered well calibrated when the number of observed deaths corresponds to the number of deaths predicted by the model. A prognosis model with good calibration is essential to predict a reliable risk. To assess a model\u0026rsquo;s calibration (\u003cspan citationid=\"CR42\" class=\"CitationRef\"\u003e42\u003c/span\u003e), we can do a calibration plot, that is we divide the predicted risks in, say, 5 or 10 groups using percentiles of the predicted probabilities of death and compare, for each of these groups, the average of the predicted risk to the observed risk. The different calibration measures in the context of the Cox model can be found in McLernon et al (\u003cspan citationid=\"CR43\" class=\"CitationRef\"\u003e43\u003c/span\u003e). One may assess the calibration of the model by plotting predicted vs. observed risks: good calibration is achieved when the points lay close to the first diagonal.\u003c/p\u003e \u003cp\u003eA complementary way to assess calibration for a prognosis model is to compare Kaplan-Meier survival estimates to model-based prediction of survival curves among pre-defined subgroups as defined by covariate values (for example, dividing the sample into 5 age-groups). Model-based prediction for a given subgroup is obtained by averaging the model-based survival predictions of each individual from that subgroup.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec14\" class=\"Section2\"\u003e \u003ch2\u003eImplementation\u003c/h2\u003e \u003cp\u003eAll the implementation has been carried out using the R programming language (\u003cspan citationid=\"CR44\" class=\"CitationRef\"\u003e44\u003c/span\u003e).The computation of the C-index and the Brier score were already implemented in the package \u003cem\u003edynpred\u003c/em\u003e (\u003cspan citationid=\"CR45\" class=\"CitationRef\"\u003e45\u003c/span\u003e) for the Cox model, and we adapted these functions for the Bspline-based model and the GH model. We also implemented the calculation of AUC function for these models. These functions are provided in a Github repository \u0026ldquo;\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://github.com/margueritefournier/Perf_measures_survival_models\u003c/span\u003e\u003cspan address=\"https://github.com/margueritefournier/Perf_measures_survival_models\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u0026rdquo;.\u003c/p\u003e \u003c/div\u003e"},{"header":"Results","content":"\u003cp\u003eWe have used the following covariates in the LeukSurv data set: age at diagnosis, sex, white blood cell-count at diagnosis with \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(1 unit=50 \\times {10}^{9}/L\\)\u003c/span\u003e\u003c/span\u003e, and Townsend score, which is a measure of deprivation for the enumeration district (small geographical area with mean population of less than a thousand) area of residence. All covariates are continuous except for the variable sex. The maximum follow-up is 13.6 year. There were 1043 patients and 15.7% of censored survival times (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003ePatient\u0026rsquo;s characteristics at baseline \u0026ndash; Leukemia data\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"2\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eOverall\u003c/p\u003e \u003cp\u003e(N\u0026thinsp;=\u0026thinsp;1043)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAge at diagnosis\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMean (SD)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e60.7 (18.3)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMedian [Min, Max]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e65.0 [14.0, 92.0]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eSex\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFemale\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e496 (47.6%)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMale\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e547 (52.4%)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eWhite blood cell count\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMean (SD)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e38.6 (72.8)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMedian [Min, Max]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e7.90 [0, 500]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eTownsend score\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMean (SD)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.340 (3.64)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMedian [Min, Max]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e-0.370 [-6.09, 9.55]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eSurvival time (years)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMean (SD)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1.46 (2.46)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMedian [Min, Max]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0.507 [0.00274, 13.6]\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eSurvival status\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCensored\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e164 (15.7%)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDeath\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e879 (84.3%)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eWe fitted three types of regression models: a Cox model, a B-Spline-based model, and a GH model. For both Bspline-based and GH models, we investigated time-dependent effects for age. We selected the best model between a model with no time-dependent effect of age and a model with time-dependent effect of age using AIC (Akaike Information Criterion). For both Bspline-based and GH models, the selected candidate models included a time-dependent effect for age.\u003c/p\u003e \u003cp\u003eFor each model type, we computed the predictive measures of overall performance (Brier Score), discrimination (\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(AUC\\left(t\\right)\\)\u003c/span\u003e\u003c/span\u003e) and calibration.\u003c/p\u003e \u003cp\u003eFigure \u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e shows the prediction error curves for the three candidate models. The three models were similar in terms of overall prediction error as evaluated with the time-varying Brier Score.\u003c/p\u003e \u003cp\u003eFigure \u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e presents the cumulative-dynamic AUC according to time since diagnosis for the three candidate models, where we can see that all models have similar performance in terms of this discrimination measure.\u003c/p\u003e\u003cp\u003eFigure \u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e shows the calibration-in-the-large at three different times after diagnosis, 6 months, 3 years and 8 years, comparing the observed risks and the predicted risks of 10 groups defined using deciles of predicted risks. The Bspline-based model seemed to have a better calibration with a regression slope of 1.02, as compared to Cox (slope: 1.22) and GH (slope: 1.08) models at 6 months and at 3 years (0.91 vs 0.72 and 0.71, respectively). At 8 years, the GH models has a better regression slope, but all models have a poor calibration (around 0.5). Figure\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e compares the survival as predicted from each model to the non-parametric survival estimated with Kaplan Meier (the \u0026ldquo;observed\u0026rdquo; data). All curves were plotted by categories of the variables included in the model. Overall, all models matched well the observed data.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eThis article aims to explain the definition and the practical use of different performance measures to assess the predictive ability of a prognosis model, with special focus on overall performance, discrimination, and calibration measures. Many tutorials already exist, with focus on binary data, but much less attention has been paid to the context of time-to-event outcomes, especially when regression models other than the Cox PH model (potentially with time-dependent and/or non-linear effect(s)) are of interest. We hope that this tutorial, along with the R-code provided, will help researchers applying these measures when developing a prognosis model.\u003c/p\u003e \u003cp\u003eRegarding the overall measures, the Brier score is the most widely used, with its weighted version (\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e) to handle right censoring. Discrimination allows one to quantify the ability of a model to correctly classify patients in a risk group. We have favoured the description of the cumulative dynamic AUC because it is the most widely adopted measure for hazard-based regression models (\u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e38\u003c/span\u003e), while it has been shown that the C-index is not a proper scoring rule for survival models (\u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e). Finally, calibration measures help assessing the adequacy between the predicted and observed risk for a given group of individuals. In the line, we have focused on the presentation and use of the calibration plot. We have also presented a graphical comparison between survival curves as estimated non-parametrically in each subgroup \u003cem\u003evs.\u003c/em\u003e the average of the predicted survival curves for all individuals in the same subgroup (while accounting for their observed covariates). A good overlap between the model-based curves and the survival curves estimated non-parametrically suggests a good calibration of the predictive model.\u003c/p\u003e \u003cp\u003eIn our illustrative example, the predictive performances of the 3 different models were not substantially different, thus suggesting that an increase in flexibility on this dataset does not improve predictions, except for the calibration plot at 3 years for the B-spline model. However, such interpretation should be done with caution because we have built and validated the model using the same dataset. More work should investigate the performances of the 3 models using simulations studies with different scenarios.\u003c/p\u003e \u003cp\u003eWe have focused on explaining the use and implementation of three types of predictive performance measures. However, predictive modelling also aims at helping clinicians inform and improve their decision-making process. Once the predictive performance of a hazard-based regression model has been assessed, its usefulness to answer the question \u0026ldquo;would the use of the predictive model improve clinical decision making?\u0026rdquo; should be evaluated (\u003cspan citationid=\"CR43\" class=\"CitationRef\"\u003e43\u003c/span\u003e, \u003cspan citationid=\"CR46\" class=\"CitationRef\"\u003e46\u003c/span\u003e). This represents an active research area, beyond the scope of this tutorial, where the concepts of \u0026ldquo;Decision Curve Analysis\u0026rdquo; and \u0026ldquo;Net Benefit\u0026rdquo; have been recently developed (\u003cspan citationid=\"CR43\" class=\"CitationRef\"\u003e43\u003c/span\u003e).\u003c/p\u003e"},{"header":"Abbreviations","content":"\u003cdiv class=\"DefinitionList\"\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eAIC\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eAkaike Information Criterion\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eAH\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eAccelerated Hazards model\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eAFT\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eaccelerated failure time model\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eAML\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eAcute Myeloid Leukemia\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eAUC\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eArea Under the ROC\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eBS\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eB-spline\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eGH\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eGeneral Hazard\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eHH\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eHybrid Hazard model\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eHR\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eHazard Ratio\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003ePGW\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003ePower Generalized Weibull\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003ePH\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eProportional Hazards\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eROC\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eReceiver Operating Characteristic\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eTDS\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eTownsend Deprivation Score\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eWBC\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eWhite Blood Cell\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003c/div\u003e"},{"header":"Declarations","content":"\u003cp\u003eEthics approval and consent to participate\u003c/p\u003e\n\u003cp\u003eNot applicable.\u003c/p\u003e\n\u003cp\u003eConsent for publication\u003c/p\u003e\n\u003cp\u003eNot applicable.\u003c/p\u003e\n\u003cp\u003eCompeting Interests\u003c/p\u003e\n\u003cp\u003eAuthors declare no competing interests.\u003c/p\u003e\n\u003cp\u003eFunding\u003c/p\u003e\n\u003cp\u003eResearchers from the Inequalities in Cancer Outcome Network is funded by Cancer Research UK programme (Grant No. EPNCZS34).\u003c/p\u003e\n\u003cp\u003eAcknowledgements\u003c/p\u003e\n\u003cp\u003eAuthors thank the biometry department of the LYSARC for useful discussions during several meetings.\u003c/p\u003e\n\u003cp\u003eAvailability of data and materials\u003c/p\u003e\n\u003cp\u003eThe dataset used to run the example is freely available in the spBayesSurv R package.\u003c/p\u003e\n\u003cp\u003eAuthors\u0026apos; contributions\u003c/p\u003e\n\u003cp\u003eMF and AB developed the concept, designed the first draft of the article and the computing code. MF, FJR, LC, CM, AB interpreted and reviewed the code and the data, drafted and revised the article. MF, FJR, LC, CM, AB read and approved the final version of the article.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eHenderson R, Jones M, Stare J. Accuracy of point predictions in survival analysis. Stat Med. 2001;20(20):3083\u0026ndash;96.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAlba AC, Agoritsas T, Walsh M, Hanna S, Iorio A, Devereaux PJ, et al. Discrimination and Calibration of Clinical Prediction Models: Users\u0026rsquo; Guides to the Medical Literature. JAMA. 2017;318(14):1377\u0026ndash;84.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMoons KGM, Royston P, Vergouwe Y, Grobbee DE, Altman DG. Prognosis and prognostic research: what, why, and how? BMJ. 2009;338:b375.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCox DR. Regression Models and Life-Tables. J Roy Stat Soc. 1972;34(2):187\u0026ndash;220.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAbrahamowicz M, Mackenzie T, Esdaile JM. Time-Dependent Hazard Ratio: Modeling and Hypothesis Testing with Application in Lupus Nephritis. J Am Stat Assoc. 1996;91(436):1432\u0026ndash;9.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSleeper LA, Harrington DP. Regression Splines in the Cox Model with Application to Covariate Effects in Liver Disease. J Am Stat Assoc. 1990;85(412):941\u0026ndash;9.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRoyston P, Parmar MKB. Flexible parametric proportional-hazards and proportional-odds models for censored survival data, with application to prognostic modelling and estimation of treatment effects. Stat Med. 2002;21(15):2175\u0026ndash;97.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKooperberg C, Stone CJ, Truong YK. Hazard Regression. J Am Stat Assoc. 1995;90(429):78\u0026ndash;94.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHagar Y, Dignam JJ, Dukic V. Flexible modeling of the hazard rate and treatment effects in long-term survival studies. Stat Methods Med Res. 2017;26(5):2455\u0026ndash;80.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePerperoglou A, Sauerbrei W, Abrahamowicz M, Schmid M. A review of spline function procedures in R. BMC Med Res Methodol. 2019;19(1):46.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGiorgi R, Abrahamowicz M, Quantin C, Bolard P, Esteve J, Gouvernet J, et al. A relative survival regression model using B-spline functions to model non-proportional hazards. Stat Med. 2003;22(17):2767\u0026ndash;84.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRemontet L, Bossard N, Belot A, Est\u0026egrave;ve J. An overall strategy based on regression models to estimate relative survival and model the effects of prognostic factors in cancer survival studies. Stat Med. 2007;26(10):2214\u0026ndash;28.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCharvat H, Remontet L, Bossard N, Roche L, Dejardin O, Rachet B, et al. A multilevel excess hazard model to estimate net survival on hierarchical data allowing for non-linear and non-proportional effects of covariates. Stat Med. 2016;35(18):3066\u0026ndash;84.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen YQ, Jewell NP. On a general class of semiparametric hazards regression models. Biometrika. 2001;88(3):687\u0026ndash;702.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRubio FJ, Remontet L, Jewell NP, Belot A. On a general structure for hazard-based regression models: An application to population-based cancer research. Stat Methods Med Res. 2019;28(8):2404\u0026ndash;17.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSteyerberg E. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating [Internet]. New York: Springer-Verlag; 2009 [cited 2020 Jan 19]. (Statistics for Biology and Health). Available from: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.springer.com/gp/book/9780387772431\u003c/span\u003e\u003cspan address=\"https://www.springer.com/gp/book/9780387772431\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRoyston P, Moons KGM, Altman DG, Vergouwe Y. Prognosis and prognostic research: Developing a prognostic model. BMJ. 2009;338:b604.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTournoud M, Larue A, Cazalis MA, Venet F, Pachot A, Monneret G, et al. A strategy to build and validate a prognostic biomarker model based on RT-qPCR gene expression and clinical covariates. BMC Bioinformatics. 2015;16(1):1\u0026ndash;15.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSteyerberg EW, Vergouwe Y. Towards better clinical prediction models: seven steps for development and an ABCD for validation. Eur Heart J. 2014;35(29):1925\u0026ndash;31.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRemontet L, Bossard N, Iwaz J, Est\u0026egrave;ve J, Belot A. Framework and optimisation procedure for flexible parametric survival models. Stat Med. 2015;34(25):3376\u0026ndash;7.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRubio FJ, Rachet B, Giorgi R, Maringe C, Belot A. On models for the estimation of the excess mortality hazard in case of insufficiently stratified life tables. Biostatistics. 2021;22(1):51\u0026ndash;67.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHenderson R, Shimakura S, Gorst D. Modeling Spatial Variation in Leukemia Survival Data. 2002.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDurrleman S, Simon R. Flexible regression models with cubic splines. Stat Med. 1989;8(5):551\u0026ndash;61.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHess KR. Assessing time-by-covariate interactions in proportional hazards regression models using cubic spline functions. Stat Med. 1994;13(10):1045\u0026ndash;62.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGauthier J, Wu QV, Gooley TA. Cubic splines to model relationships between continuous variables and outcomes: a guide for clinicians. Bone Marrow Transplant. 2020;55(4):675\u0026ndash;80.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJones MC, Noufaily A. Log-location-scale-log-concave distributions for survival and reliability analysis. Electron J Stat. 2015;9(2):2732\u0026ndash;50.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGraf E, Schmoor C, Sauerbrei W, Schumacher M. Assessment and comparison of prognostic classification schemes for survival data. Stat Med. 1999;18(17\u0026ndash;18):2529\u0026ndash;45.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePfeiffer RM, Gail MH, Gail MH. Absolute Risk: Methods and Applications in Clinical Management and Public Health [Internet]. Chapman and Hall/CRC; 2017 [cited 2020 Jan 13]. Available from: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www-taylorfrancis-com.docelec.univ-lyon1.fr/books/9781315117539\u003c/span\u003e\u003cspan address=\"https://www-taylorfrancis-com.docelec.univ-lyon1.fr/books/9781315117539\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTsouprou S, Putter H, Fiocco M. Measures of discrimination and predictive accuracy for interval censored survival data [PhD Thesis]. Master\u0026rsquo;s thesis, Leiden University; 2015.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSchumacher M, Graf E, Gerds T. How to assess prognostic models for survival data: a case study in oncology. Methods Inf Med. 2003;42(5):564\u0026ndash;71.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHarrell FE, Califf RM, Pryor DB, Lee KL, Rosati RA. Evaluating the yield of medical tests. JAMA. 1982;247(18):2543\u0026ndash;6.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRoyston P, Altman DG. Visualizing and assessing discrimination in the logistic regression model. Stat Med. 2010;29(24):2508\u0026ndash;20.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHarrell FE, Lee KL, Mark DB. Multivariable Prognostic Models: Issues in Developing Models, Evaluating Assumptions and Adequacy, and Measuring and Reducing Errors. Stat Med. 1996;15(4):361\u0026ndash;87.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePencina MJ, D\u0026rsquo;Agostino RB. Overall C as a measure of discrimination in survival analysis: model specific population value and confidence interval estimation. Stat Med. 2004;23(13):2109\u0026ndash;23.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBlanche P, Kattan MW, Gerds TA. The c-index is not proper for the evaluation of \u003cspan\u003e$\u003c/span\u003et\u003cspan\u003e$\u003c/span\u003e-year predicted risks. Biostatistics. 2019;20(2):347\u0026ndash;57.\u003c/span\u003e \u003c/li\u003e \u003cli\u003e\u003cspan\u003eHeagerty PJ, Lumley T, Pepe MS. Time-dependent ROC curves for censored survival data and a diagnostic marker. Biometrics. 2000;56(2):337\u0026ndash;44.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBlanche P, Latouche A, Viallon V. Time-dependent AUC with right-censored data: a survey study. arXiv:12106805 [stat] [Internet]. 2012 Oct 25 [cited 2022 Feb 7]; Available from: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttp://arxiv.org/abs/1210.6805\u003c/span\u003e\u003cspan address=\"http://arxiv.org/abs/1210.6805\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHeagerty. Survival Model Predictive Accuracy and ROC Curves [Internet]. 2005 [cited 2021 Sep 7]. Available from: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://onlinelibrary.wiley.com/doi/\u003c/span\u003e\u003cspan address=\"https://onlinelibrary.wiley.com/doi/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1111/j.0006-341X.2005.030814.x\u003c/span\u003e\u003cspan address=\"10.1111/j.0006-341X.2005.030814.x\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHung H, Chiang CT. Estimation methods for time-dependent AUC models with survival data. Can J Stat. 2010;38(1):8\u0026ndash;26.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSatten GA, Datta S. The Kaplan-Meier Estimator as an Inverse-Probability-of-Censoring Weighted Average. Am Stat. 2001;55(3):207\u0026ndash;10.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eVan Calster B, McLernon DJ, van Smeden M, Wynants L, Steyerberg EW, Bossuyt P, et al. Calibration: the Achilles heel of predictive analytics. BMC Med. 2019;17(1):230.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSteyerberg EW, Calster BV, Pencina MJ. Performance Measures for Prediction Models and Markers: Evaluation of Predictions and Classifications. Rev Esp Cardiol. 2011;64(9):788\u0026ndash;94.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMcLernon DJ, Giardiello D, Calster BV, Wynants L, van Geloven N, van Smeden M et al. Assessing performance and clinical usefulness in prediction models with survival outcomes: practical guidance for Cox proportional hazards models [Internet]. medRxiv; 2022 [cited 2022 May 30]. p. 2022.03.17.22272411. Available from: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.medrxiv.org/content/\u003c/span\u003e\u003cspan address=\"https://www.medrxiv.org/content/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1101/2022.03.17.22272411v1\u003c/span\u003e\u003cspan address=\"10.1101/2022.03.17.22272411v1\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eIhaka R, Gentleman R. R: A Language for Data Analysis and Graphics. J Comput Graphical Stat. 1996;5(3):299\u0026ndash;314.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eVan Houwelingen H, Putter H. Dynamic prediction in clinical survival analysis. CRC Press; 2011.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eVickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making. 2006;26(6):565\u0026ndash;74.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-3866618/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-3866618/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003ePredicting the occurrence of an event over time for a newly diagnosed individual is a common aim in medical statistics. For time-to-event outcomes, this prediction is typically based on a regression model. The Cox proportional hazard (PH) model represents one of the most popular regression models for analysing time-to-event data. However, several flexible models that go beyond the assumption of proportionality of hazards have been recently developed. These include flexible hazard-based models using splines or models based on more general hazard structures. In these 2 types of models, non-linear associations and time-varying regression coefficient(s) can be easily included. Assessing the predictive ability of a hazard-based regression model is necessary to validate a predictive model but it might prove difficult for models other than the Cox PH model. We present a tutorial which explains how the predictive ability of hazard-based regression models can be assessed, focusing on the 3 commonly used performance measures. We report (i) the overall prediction ability using prediction error curve and the Brier score, (ii) the discriminative ability using the cumulative/dynamic area under the receiving operator characteristic curve, and (iii) the calibration ability, i.e., the agreement between observed and predicted probabilities, using calibration plots and graphical comparison between predicted and observed survival. We provide an implementation of these methods in R together with an illustrative example using a publicly available data set.\u003c/p\u003e","manuscriptTitle":"Assessing predictive abilities of hazard-based regression models for survival data: a tutorial for prognosis modelling","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-01-18 15:44:39","doi":"10.21203/rs.3.rs-3866618/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"2a1b8f6a-0d1d-469c-b251-ad1fa38219c1","owner":[],"postedDate":"January 18th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2024-05-09T15:29:43+00:00","versionOfRecord":[],"versionCreatedAt":"2024-01-18 15:44:39","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-3866618","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-3866618","identity":"rs-3866618","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}