{"paper_id":"658af9d6-b09c-4572-8c75-40a669c985c4","body_text":"A recent systematic review identified over 30 clinical prediction models which estimate\nindividualized chances of pregnancy outcomes following IVF treatment ( Ratna  et al. , 2020 ). These models can help\nclinicians communicate chances of treatment success to couples undergoing IVF, but their use\nin clinical practice has been limited. The quality of these models is impacted by issues\nsuch as small sample sizes, lack of external validation and failure to demonstrate clinical\nimpact ( Leushuis  et al. ,\n2009 ;  Van Loendersloot  et\nal. , 2014 ;  Ratna  et\nal. , 2020 ).\nFive IVF prediction model studies have been conducted using large national databases ( Templeton  et al. , 1996 ;  Nelson and Lawlor, 2011 ;  Luke  et al. , 2014 ;  McLernon  et al. , 2016 ;  McLernon  et al. , 2021 ). Of these, three utilized\ndata from the Human Fertilisation and Embryology Authority (HFEA) registry in the UK to\nestimate the chances of a live birth after IVF ( Templeton  et al. , 1996 ;  Nelson and Lawlor, 2011 ;  McLernon\n et al. , 2016 ). Two of these articles published models that\npredict cumulative live birth over complete cycles of IVF, where a complete cycle is defined\nas all fresh and frozen-thawed embryo transfers associated with one episode of ovarian\nstimulation ( McLernon  et al. ,\n2016 ;  McLernon  et al. ,\n2021 ). With the increasing use of frozen-thawed embryos in IVF ( Wong  et al. , 2014 ), cumulative\nlive birth rate (LBR) over multiple complete cycles is a more clinically relevant outcome\nthan the chance of live birth following a single embryo transfer ( Maheshwari  et al. , 2015 ) and clinical prediction\nmodels need to make sure that they address this need ( McLernon and Bhattacharya, 2023 ).\nTwo UK models by McLernon  et al.  were developed to predict the chances of\ncumulative live birth over multiple complete cycles of IVF: a pre-treatment model which\npredicts cumulative live birth in women before the first complete cycle commences; and a\npost-treatment model which updates predictions of cumulative live birth after the first\nfresh embryo transfer episode ( McLernon  et\nal. , 2016 ). The models were converted into an online prediction tool\ncalled OPIS (Outcome Prediction In Subfertility) ( https://w3.abdn.ac.uk/clsm/opis/ )\nand used by 3000 patients and clinicians on average each month. The models which were\ndeveloped using data from IVF treatments conducted from 1999 to 2008 showed good predictive\nperformance in the development dataset but have not been validated in the UK since. External\nvalidation in an independent cohort is essential as it supports the generalizability of the\nmodel ( Harrell  et al. , 1996 ;\n Steyerberg, 2019 ). Using prospectively\ncollected Dutch data between 2011 and 2014, a study externally validated the performance of\nthe McLernon  et al.  up to three complete cycles ( Leijdekkers  et al. , 2018 ). The findings revealed\nthat the pre-treatment model systematically overestimated the probability of cumulative live\nbirth in the external cohort but provided more accurate predictions after recalibration,\nwhilst the post-treatment model calibrated well in the external cohort.\nIVF practice in the UK has since undergone major changes, with greater emphasis on elective\nsingle embryo transfer and increasing use of frozen-thawed embryo transfers ( Human Fertilisation and Embryology Authority,\n2018 ;  Ishihara  et al. ,\n2014 ). Therefore, it is important that the McLernon models are externally validated\non a more up-to-date cohort of patients to ensure that the predictions are still accurate.\nTherefore, the aim of the study is to conduct a temporal external validation of the McLernon\nmodels in order to demonstrate the continued generalizability of these models to the current\nUK IVF population.\n\nTo perform external validation of the McLernon models, this study used the HFEA database\nwhich links all fresh and frozen IVF treatment cycles to individual women. Database access\nwas granted following approval by the North of Scotland Research Ethics Committee, the\nConfidentiality Advisory Group, and the HFEA register research panel. The data were\nanonymized and transferred to the University of Aberdeen where they were stored on the\nData Safe Haven (DaSH) server for analysis.\nInformation was collected from 91 035 women who started their first ovarian stimulation\nin the UK between January 2010 and December 2016. The records of all complete IVF cycles\nwhich began during this period were extracted. Data on frozen embryo transfers associated\nwith these between 1 January 2017 and 31 December 2017 were also collected. No data\nrecorded after the 31 December 2017 were extracted. This data selection method ensured a\nminimum of 1-year exposure to all embryo transfer attempts within a complete cycle. Women\nwhose treatment involved donor insemination, egg donation and/or surrogacy were\nexcluded.\nFor this validation study, the same baseline characteristics that were used in the\noriginal McLernon pre- and post-treatment models were selected from the new dataset (with\nthe exception of duration of infertility and pregnancy history which are discussed in the\nmissing data section). The McLernon pre-treatment model predicts the probability of a live\nbirth over six complete cycles at the start of a first complete cycle. Predictions are\nbased on couple characteristics and the type of treatment (IVF or ICSI) to be used. The\nincluded predictors are female age (years), duration of infertility (years), causes of\ninfertility (tubal, male factor, anovulation, or unexplained), pregnancy history (yes or\nno), type of treatment (IVF or ICSI), and treatment year.\nAfter the first fresh embryo transfer, the McLernon post-treatment model revises the\npredictions using additional treatment-specific data from this cycle. The added predictors\nare number of eggs collected, availability of cryopreserved embryos, number of embryos\ntransferred (one, two, or three), and stage of transferred embryos, i.e. blastocyst (Day 5\nor 6) or cleavage stage (Day 2 or 3). For the validation of the post-treatment model,\nwomen who had no eggs collected were excluded as it is impossible for them to achieve a\nlive birth in the first complete cycle.\nThe number of complete cycles was included in both models as a discrete time variable to\npredict the probability of a live birth in the  i th cycle, assuming no\nlive birth occurred in the previous cycle(s). The formulae for calculating the cumulative\npredicted probability of a live birth over six complete cycles can be seen in  Supplementary data files S1  and  S2  ( McLernon  et al. , 2016 ).\nData on the duration of infertility were missing for 97% of women, and pregnancy\nhistory was entirely missing. This is because the HFEA stopped collecting this\ninformation since 2008 (HFEA communication) ( Supplementary data file S3 ). Since these variables were fully recorded\nfrom 1999 to 2007, data from this period were used to impute the missing values of these\nvariables in the validation dataset (2010 to 2016).\nIn the study, three predictor variables had missing values: duration of infertility,\npregnancy history, and stage and number of embryos transferred. Multiple imputation of\nthese predictors was performed to increase the statistical power of the model and to\nadjust for any biases caused by excluding women with missing information ( Greenland and Finkle, 1995 ). Ten imputed\ndatasets were created using the chained equation (MICE) method (to attain a monotone\nmissing data pattern) ( Sterne  et\nal. , 2009 ). Then each missing variable was considered as a dependent\nvariable in its own imputation model where it was regressed onto all the other\nvariables. The following variables were included to inform the imputation process:\nfemale age, year of treatment, cause of infertility, IVF versus ICSI, and whether\nembryos were cryopreserved. For the continuous variable ‘duration of infertility’, a\npredicted mean matching regression model was used; to impute the binary variable\n‘pregnancy history’, a logistic regression model was used; and to impute the nominal\ncategorical variable ‘stage and number of embryos transferred’, a discriminant function\nmethod was used. This imputation was performed under the assumption that the data were\nmissing at random (MAR) which means that the missing data depend on the values of the\nobserved variables and treatment outcome.\nThe predictor values for women in the validation cohort were multiplied by the\ncorresponding parameter estimates of the predictors from the original pre-treatment model\nand then added together. The same was done for the post-treatment model ( McLernon  et al. , 2016 ). The\npredicted probabilities were calculated using the formulas in  Supplementary data files S1  and  S2 .\nThe predictive performance of the McLernon models was evaluated in terms of\ndiscrimination and calibration. Discrimination refers to the ability of the models to\ndistinguish between women who will achieve a live birth and those who will not ( Moons  et al. , 2012 ) and was\nassessed using the c-statistic.\nCalibration refers to the degree of agreement between the observed live birth in the\nexternal cohort and predicted live birth ( Moons\n et al. , 2012 ). This was formally assessed using\ncalibration-in-the-large (CIL) and the calibration slope, and graphically assessed using a\ncalibration plot ( Cox, 1958 ;  Miller  et al. , 1993 ). For\nperfect calibration, the calibration slope and calibration intercept should be 1 and 0\nrespectively. We calculated c-statistics, CIL, and calibration slope on each imputed\ndataset and separate results were pooled using the metamisc package in R version 4.1.1\n( Debray  et al. , 2017 ).\nCalibration plots and predicted curves for hypothetical couples were generated using the\nfirst imputed dataset.  Supplementary data\nfile S4  gives a detailed description of all calibration techniques used in the\nstudy.\nWhere any model demonstrated poor calibration in the validation cohort, the models were\nupdated using the following three methods to try to improve performance ( Steyerberg  et al. , 2004 ;  Janssen  et al. , 2008 ;  Moons  et al. , 2012 ):\nUpdate intercept (Method 1): adjustment of the intercept using the calibration\nintercept;\nLogistic recalibration (Method 2): adjustment of the intercept and the regression\ncoefficients using the calibration intercept and calibration slope; and\nModel revision (Method 3): further model adjustment for individual predictors which\nhad a different effect in the validation cohort compared to the development\ncohort.\nThe method which demonstrated the best agreement between the predictions and observed\noutcomes was used to update each model.\nSupplementary data file S5 \nincludes a detailed description of these methods.\nAll statistical analyses were conducted using STATA version 16 ( StataCorp, 2019 ) and R version 4.1.1 ( R Core Team, 2021 ;  Posit team, 2023 ).\nNo patients were involved in framing the research question, choosing the outcome\nmeasures, or developing plans for the design or implementation of the study. Patient input\nwas not sought on interpreting or writing up the results of the study. We have plans to\ndisseminate the results of this research study to patients via national fertility\ncharities and the HFEA.\n\nFollowing exclusions, the dataset included 91 035 women who underwent 144 734 complete\ncycles of IVF/ICSI between January 2010 and December 2016 ( Supplementary Fig. S1 ). The baseline\ncharacteristics of couples and the treatments they underwent before initiating IVF are\npresented in  Table 1  for each cohort. The\ndevelopment cohort comprised women who started IVF between 1999 and 2008, whereas the\nvalidation cohort consisted of women who started IVF between 2010 and 2016. Women included\nin the validation cohort had a similar distribution of age to the women in the development\ncohort. There was also a similar distribution in causes of infertility between the two\ncohorts.\nBaseline characteristics of couples and their treatment before undergoing the first\ncomplete cycle of IVF in the cohorts used for model development and validation.\nThe variables listed were included as predictors in the development sample (HFEA\n1999–2008 cohort) and the validation sample (HFEA 2010–2017 cohort).\n93% of women had missing data on duration of infertility in 2010 which increased to\nalmost 100% in 2017.\nFrom 2008, the Human Fertilisation and Embryology Authority (HFEA) changed the layout\nof their forms for recording data and removed questions regarding pregnancy history\nand duration of infertility (HFEA communication). Therefore, since these variables\nwere fully recorded from 1998 to 2007, previous pregnancy status (which was 100%\nmissing) and duration of infertility (97% missing) were imputed in the validation\ndataset using this data to inform the imputation process. IQR: interquartile\nrange.\nA higher proportion of women underwent ICSI in the validation cohort compared to women in\nthe development cohort (51% versus 41%). After the first IVF/ICSI cycle, embryo\ncryopreservation was more frequently performed in women belonging to the validation cohort\ncompared to the development cohort (35% versus 25%). Only 32% of women in the validation\ncohort had a double cleavage embryo transfer compared to 66% in the development cohort.\nAbout half of the women in the validation sample had a single embryo transfer (17.8% had\nsingle cleavage-stage transfer and 30.1% had a single blastocyst transfer), whereas only 9%\nof women in the development dataset had a single embryo transfer.\nThe number of women in each cohort who started a treatment cycle, had a live birth, or\ndiscontinued treatment without having a live birth is presented in  Fig. 1 . The LBRs per woman were higher in the validation cohort\n(HFEA 2010–2016) compared to the development cohort (HFEA 1999–2008) for all complete cycles\nof IVF. Over six complete cycles of IVF/ICSI, the overall LBR of both recent and old HFEA\ncohorts was 45% and 43%, respectively.\nFlowchart of number of treatments and live birth outcomes over six complete\ncycles.  Frequency and percentage of women having a live birth, continuing\ntreatment without having a live birth, and discontinuing treatment without having a live\nbirth over six complete cycles of IVF/ICSI in the HFEA 1999–2008 development cohort and\n2010–2016 validation cohort. Percentages are in parentheses. HFEA: Human Fertilisation\nand Embryology Authority.\nIn the validation cohort, the pooled c-statistic for the pre-treatment model was 0.68\n(95% CI: 0.68 to 0.68) and for the post-treatment model 0.75 (95% CI: 0.75 to 0.75).  Figure 2a  shows the calibration plot for the\nfirst imputed dataset (representative of all 10 imputations) depicting the observed\ncumulative LBR in the validation cohort versus the predicted probability of cumulative\nlive birth from the pre-treatment model ( Fig. 3a  shows the post-treatment model calibration plot) ( McLernon  et al. , 2016 ). The pre-treatment\ncalibration plot had a calibration slope of 0.74 (95% CI: 0.72 to 0.76), and the\npost-treatment calibration plot had a calibration slope of 0.68 (95% CI: 0.67 to 0.70)\n( Supplementary Tables S1  and\n S2 ). The CIL analyses showed\nlittle systematic underestimation for the pre-treatment model (CIL = 0.01 or O/E = 1.01).\nSystematic overestimation was evidenced for the post-treatment model (CIL = −0.12 or\nO/E = 0.94) (see the  Supplementary Tables\nS1  and  S2 ). Both the\ncalibration slopes indicated that the original regression coefficient estimates were too\nlarge, resulting in extreme predictions in new patients. For example, the calibration\nslope of 0.74 of the pre-treatment model indicates that the original regression\ncoefficient estimates of the model are over-optimistic by around 26% in new patients, i.e.\nlow chances of live birth calculated by the model are too low and high probabilities are\ntoo high compared with the observed LBRs.\nCalibration plots for the pre-treatment model showing the association between\nthe predicted and observed cumulative live birth rates over six complete IVF/ICSI\ncycles in the validation dataset.  ( a ) Calibration plot for the\noriginal McLernon pre-treatment model as explained by  McLernon  et al.  (2016)  applied to the\nvalidation dataset; ( b ) calibration plot for the recalibrated\npre-treatment model following adjustment of the intercept in the validation dataset\n(update intercept method); ( c ) calibration plot for the recalibrated\nmodel following adjustment of both the intercept and slope in the validation dataset\n(logistic recalibration method 2); and ( d ) calibration plot for the\nrevised model after updating some coefficients using the validation dataset (model\nrevision method).\nCalibration plots for the post-treatment model showing the association between\nthe predicted and observed cumulative live birth rates over six complete IVF/ICSI\ncycles in the validation dataset.  ( a ) Calibration plot for the\noriginal McLernon post-treatment model as explained by  McLernon  et al.  (2016)  applied to the\nvalidation dataset; ( b ) calibration plot for the recalibrated\npost-treatment model following adjustment of the intercept in the validation dataset\n(update intercept method); ( c ) the recalibrated model following\nadjustment of both the intercept and slope in the validation dataset (logistic\nrecalibration method); and ( d ) calibration plot for the revised model\nafter updating some coefficients using the validation dataset (model revision\nmethod).\nGiven the poor calibration, both the pre- and post-treatment models were updated in an\neffort to improve performance in the validation cohort.\nThe estimated parameters of the original pre-treatment model and the three different\nupdated versions of the pre-treatment model (i.e. updated intercept (Method 1), logistic\nrecalibration (Method 2), and model revision (Method 3)) are summarized in  Table 2 .\nCoefficients of the predictors from the original McLernon pre-treatment model and\nupdated coefficients using three different methods in the validation dataset.\nThe estimates of the main parameters of the pre-treatment model using the different\nupdating methods and the details of estimating these parameters are presented in  Supplementary Table S3  and  Supplementary data file S6 ,\nrespectively.\nThe Method 1 updating approach did not lead to any improvement in calibration for the\npre-treatment model when reapplied to the validation cohort. However, Methods 2 and 3\ndid result in improved calibration ( Fig. 2b–d ). We compared these approaches using  Fig. 2  in order to identify the one which had the most\nbeneficial impact on calibration. After recalibrating the original model by adjusting\nthe intercept and slope (Method 2), calibration was good for all tenths except the\nseventh tenth (as 95% CI of the seventh decile does not overlap with the diagonal\nreference line) ( Fig. 2c ).  Figure 2d  shows a better update of the model\n(as 95% CIs of all deciles overlap the diagonal reference line) after model revision\n(Method 3) in the validation cohort and therefore was chosen as the best method to\nupdate the pre-treatment model ( Supplementary data file S6 ).\nThe c-statistic of the model updated by the model revision (Method 3) method decreased\nvery slightly to 0.67 (95% CI: 0.66 to 0.68) (using imputed dataset 1).\nThe estimated parameters of the original post-treatment model and the three versions of\nthe updated post-treatment model (i.e. updated intercept (Method 1), logistic\nrecalibration (Method 2), and model revision (Method 3)) are summarized in  Table 3 .\nCoefficients of the predictors from the original McLernon post-treatment model and\nupdated coefficients using three different methods in the validation dataset.\nSupplementary Table S4  and  Supplementary data file S7  present\nthe different updated estimates of the main post-treatment model parameters and the\ndetails of estimating these parameters respectively.\nAll three updating approaches can be compared to the original model in  Fig. 3 . In  Fig. 3b , some deciles were still outside the diagonal line,\nindicating that the model updated with Method 1 still needs further improvement. Method\n2 ( Fig. 3c ) showed the best improvement in\ncalibration and was chosen as the best method to update the post-treatment model.\nThe c-statistic of the model updated by the logistic recalibration (Method 2) method\nwas 0.75 (95% CI: 0.74 to 0.76) (using imputed dataset 1) ( Supplementary data file S7 ).\nFigure 4  shows examples of both the pre- and\npost-treatment model predictions in different case scenarios using the final updated\nmodels.\nExamples of the updated models predicting cumulative live birth over three\ncomplete cycles of ICSI for couples with different characteristics. \n( a ) couples with either 2 or 5 years of primary male factor\ninfertility, where the female partner is aged either 30 or 40 years (pre-treatment\nmodel); ( b ) couples with 2 years of primary male factor infertility,\nwhere the female partner is aged either 30 or 40 years, with either 5 or 15 oocytes\ncollected in the first complete cycle. Those with five oocytes have a single cleavage\nembryo transfer with no embryos cryopreserved, and those with 15 oocytes have a single\nblastocyst embryo transfer with embryos cryopreserved. S: single.\nFigure 4a  shows the cumulative predictions of\nlive birth from the updated pre-treatment model over three complete ICSI cycles. These are\npresented for women aged 30 and 40 years with either a 2- or 5-year duration of male\nfactor infertility. As shown in the figure, younger women have a much higher chance of\nsuccess. A 30-year-old woman with 2 years of infertility has a 41% predicted chance of\nhaving a live birth in the first complete ICSI cycle. This increases to 75% over three\ncomplete cycles. For a 40-year-old woman with 2 years of primary infertility, these\nprobabilities are 14% and 31% for one complete cycle and three complete cycles,\nrespectively. In contrast, for a similar woman with 5 years of infertility the\nprobabilities are 13% and 30% for one complete cycle and three complete cycles,\nrespectively.\nFigure 4b  presents the predictions from the\nupdated post-treatment model. The predicted probability of a live birth was updated for a\ncouple with the following characteristics: 30-year-old woman, 2 years of male factor\nprimary infertility, 15 oocytes collected at the start of the first cycle, embryos\ncryopreserved after fertilization, and a single fresh blastocyst embryo transferred in the\nfirst cycle. The predicted probability of live birth after the first complete ICSI cycle\nis 40%. Cumulatively, this increases to 75% over three complete cycles. A woman who is\n40 years old, has five oocytes collected, no embryos cryopreserved, and has a single\ncleavage stage embryo transferred has a 9% chance of a live birth after the first complete\ncycle. Cumulatively, this rises to 22% over three complete cycles.\n\nThe results of this study show that the pre- and post-treatment models discriminate\nreasonably well between couples with and without live birth when applied to a more recent\ncohort of IVF patients. However, both models required updating owing to poor calibration\nin the external dataset. The updated models should provide more accurate predictions in\nfuture patients, and, like the original models, will be incorporated within the OPIS\nonline calculator for regular clinical use.\nFor external validation, this study selected the McLernon models which were developed\nusing appropriate methodology, showed good predictive performance ability at both internal\nand external validation, and scored better on the Transparent Reporting of a multivariable\nprediction model for Individual Prognosis Or Diagnosis (TRIPOD) checklist than other IVF\nprediction models ( Collins  et al. ,\n2015 ;  Ratna  et al. ,\n2020 ).\nIn the study, calibration was assessed with multiple methods including CIL, logistic\ncalibration, and by visualizing the agreement between the predicted and observed LBRs\n( Bouwmeester  et al. ,\n2012 ). To improve predictions, the study updated the pre-treatment model using a\nmore extensive model revision method, while the post-treatment model was updated through\nthe simpler approach of recalibration. The recalibration methods (intercept updating and\nlogistic recalibration) are simple and stable because of the low number of parameters\nestimated. However, the model revision method is expected to lead to a lower bias in the\nupdated model since more parameters are estimated ( Steyerberg  et al. , 2004 ).\nThis study has some limitations. First, the external validation exercise involved a\ndataset with a very high proportion of missing values for duration of infertility (97%)\nand no data on previous pregnancy. Therefore, both predictors had to be imputed in our\nanalysis. These variables were assumed to be MAR as the missingness is assumed to be\nconditional on observed variables and treatment outcome. Since these variables were\nconsistently recorded between 1998 and 2007, the patient data from that time period were\nused to inform the imputation. Multiple, rather than single, imputation was performed as a\nlarge amount of missing data may lead to an underestimation of the uncertainty associated\nwith the imputed values ( Steyerberg, 2019 ).\nFemale age explained most of the variation from all of the predictors included in the\npre-treatment model, and female age, number of eggs and cryopreservation status explained\nmost in the post-treatment model. However, we cannot rule out the possibility that imputed\nvalues for duration of infertility and previous pregnancy could have accounted for some of\nthe difference in model performance in the external cohort compared to the development\ncohort.\nThe McLernon models estimate the individualized cumulative chances of live birth under\nthe optimistic assumption that couples who discontinue IVF treatment without a live birth\nhave the same chances of a live birth as couples who continue further treatment cycles.\nThis assumption may lead to an overestimation of the predicted cumulative probability of\nlive birth, as some of the women who discontinue treatment will have stopped because of\npoor prognosis ( Olivius  et al. ,\n2004 ;  Brandes  et al. ,\n2009 ) meaning they will have an almost zero chance of conceiving.\nThe original models were not able to account for other potential predictors, such as BMI,\novarian reserve tests and ethnicity, because they were absent in the HFEA database. We\nemphasize that the models can only be used in heterosexual couples using their own eggs\nand sperm and not undergoing preimplantation genetic testing. It should also be noted that\nthe predictions from our models will represent an average prediction over all clinics\nwithin the UK. Clinic identifiers are not accessible from the HFEA and so it was not\npossible to adjust at the individual clinic level.\nIn October 2009, the HFEA changed their consent policy so that patients had to opt-in for\ntheir IVF data to be used for research purposes. Our validation study used data from 2010\nand so only would have included couples who opted in. We do not expect there to be a\ndifference in characteristics and outcome between those who opt-in and those who opt-out,\nbut it is difficult to know for sure without access to the data of those who opted\nout.\nWe were able to reassess calibration and discrimination after updating both models. This\nwould be considered a type of internal validation as it involves assessing the performance\nof the updated models in the dataset used to update them. Ideally, we would like to be\nable to validate the updated models using a dataset from a separate population or to\nconduct a further temporal validation on a more up to date version of the HFEA dataset.\nThe latter would be preferable from a practical perspective, as the models were developed\nfor, and validated on, UK national data and are intended for use by UK couples. We aim to\ncontinue validating the models periodically in the future using UK data to ensure that\nthey remain fit for purpose ( Van Calster\n et al. , 2023 ).\nOur results show that when applied to more recently treated patients, our models\nunderpredicted outcomes in women with low observed LBRs and slightly overpredicted in\nwomen with high observed LBRs. Therefore, it was very important to update these models to\nreflect current practice and to provide more accurate predictions for patients and\nclinicians. After updating, the models showed improved agreement between live birth\npredictions and observed LBRs, as expected. As such, they can be considered suitable for\nclinical use and can be used to inform future couples of their likely chances of treatment\nsuccess ( Arvis  et al. ,\n2012 ;  Zarinara  et al. ,\n2016 ). When updating the McLernon models for the later time period, the\ndifferences in the relative weights of the variables was probably a result of a\ncombination of differences in IVF protocols, improved IVF success rates, and differences\nin case mix between the two cohorts ( McLernon\n et al. , 2016 ;  Leijdekkers  et al. , 2018 ). The proportion of women having\nembryo cryopreservation, single embryo transfer, and blastocyst transfer were higher in\nthe validation cohort than the development cohort. This is a result of the increased use\nof single embryo transfer following the introduction of the UK ‘one-at-a-time’ policy in\n2007. It also reflects the increased use of embryo cryopreservation owing to improvements\nin embryo freezing techniques ( Human Fertilisation\nand Embryology Authority, 2018 ;  Ishihara\n et al. , 2014 ). These changes in practice and techniques may\nhave resulted in a degree of calibration drift which could explain the different\nperformances of the McLernon models in the validation cohort ( Jenkins  et al. , 2018 ). Even our updated models\nwill have suffered some calibration drift since the end of our study period in 2016. Since\nthen, the national LBR per embryo transferred has only increased by 1%, from 22% to 23% in\n2018, which suggests that not much changed in the following 2 years ( Human Fertilisation and Embryology Authority, 2018 ,  2020 ). The HFEA has yet to publish data on UK\nLBRs for 2019–2022 so it is difficult to estimate how much calibration drift has affected\nour updated model.\nBoth live birth and treatment discontinuation rates in all complete cycles of IVF were\nhigher in the validation cohort than the development cohort. Year of treatment was\nstrongly positively associated with live birth, reflecting improvements in ART over time\n( McLernon  et al. , 2016 ).\nFrom October 2009, the HFEA patient consent forms were changed so that patients had to\nexplicitly agree that their data could be used for research purposes. This change led to\nhigher discontinuation rates owing to many women opting not to disclose their treatment\ninformation. Therefore, only couples who provided explicit consent for their information\nto be used in research were included in this study. Data collected in 2009 were also\nexcluded from this study to ensure that the dataset only encompassed the time period after\nwhich the new forms were introduced across the whole of the UK.\nRegarding discrimination, the updated pre-treatment model had a slightly lower\nc-statistic (0.68, 95% CI: 0.67 to 0.69) in the validation cohort than in the development\ncohort (0.69, 95% CI: 0.68 to 0.69) ( McLernon\n et al. , 2018 ). The recalibrated post-treatment model had a\ngood c-statistic of 0.75 (95% CI: 0.74 to 0.76) in the validation cohort which is slightly\nlower than the c-statistic of 0.76 (95% CI: 0.75 to 0.77) in the development cohort ( McLernon  et al. , 2018 ). A\nprevious validation study also reported lower c-statistics of 0.62 (95% Cl: 0.59 to 0.64)\nand 0.71 (95% CI: 0.69 to 0.74) for the recalibrated pre-treatment McLernon model and the\ncalibrated post-treatment McLernon model, respectively ( Leijdekkers  et al. , 2018 ). These reductions in\nmodel discrimination ability are likely due to the differences in couple and treatment\ncharacteristics and outcome prevalence between the two cohorts ( Moons  et al. , 2012 ).\nPoor calibration was evidenced in the external validation for McLernon models. Three\nincreasingly complicated methods were explored for updating the models (i.e. intercept\nupdating, logistic recalibration, and model revision). The method that led to the most\nimprovement (as evidenced by the calibration plot) was selected to update the models\n( Janssen  et al. , 2008 ).\nModel updating over time is expected given improvements in IVF practice and technology,\nand changes in patient case mix.\nAlthough our post-treatment model showed good discrimination after recalibration, the\ndiscriminatory ability of the pre-treatment model remained reasonably low, as is the case\nfor almost all fertility-based prediction models ( Leushuis  et al. , 2009 ). The literature suggests that the low\nc-statistic reflects the homogeneity of the study population e.g. infertile women of\nreproductive age ( Cook, 2008 ;  Coppus  et al. , 2009 ). However,\na low c-statistic does not necessarily imply that such prediction models have limited use\nin clinical practice. Couples with a fertility problem are more interested in knowing\ntheir chances of live birth (calibration) rather than the ability of the model to\ndiscriminate between couples who will have a live birth and couples who will not.\nTherefore, assessment by calibration is more relevant.\nTwo prediction models were developed using national US data from the Society for Assisted\nReproductive Technology (SART) ( McLernon\n et al. , 2021 ). The first model is a pre-treatment model,\nsimilar to that validated in our current study. The second model is a post-treatment model\nbut differs from the one validated here because it predicts cumulative live birth chances\nin couples starting a second complete cycle whose first complete cycle was unsuccessful.\nThe pre-treatment model was adjusted for BMI, which was not available for the UK models.\nFurthermore, anti-Müllerian hormone (AMH) was included in a second pre-treatment model\ndeveloped using a sub-population who had an AMH measurement. The SART data did not have\nduration of infertility which was available in the HFEA data and included as a predictor\nin the UK model. The US models have yet to be externally validated but the c-statistic of\nthe pre-treatment model in the development dataset was slightly higher than that for the\nUK pre-treatment model (0.71 versus 0.69) ( McLernon  et al. , 2018 ).\nThe amount of electronic data produced and stored in the field of reproductive medicine\nhas increased considerably. Artificial intelligence (AI) (or machine learning) is\nprogressively used in medical research to predict future outcomes and is often used in\nplace of regression-based models. Approaches such as Bayesian neural networks and boosting\nalgorithms are more suited to high dimensional datasets, i.e. containing a large number of\npotential predictors which may include imaging information. Because of this, they require\nmany patients to avoid risk of bias ( Andaur Navarro\n et al. , 2021 ). Models using such approaches that are developed\nin a single clinic may not be transportable to other clinics as they tend to detect\npatterns unique to that particular clinic ( Chen\n et al. , 2022 ). However, if clinics are able to share and\ncombine their data to develop such models and then assess heterogeneity in predictive\nperformance between clinics then they may be transportable ( Riegler MA  et al. , 2021 ). High dimensional\nelectronic health records are not commonly available yet in reproductive medicine ( Shingshetty  et al. , 2022 ). Our\nregression-based models will be useful until a reliable and tested AI model has been\ndeveloped, validated and shown to perform better than our model. There are many\npublications showing that traditional statistical regression models can match or even\noutperform AI models ( Liew  et\nal. , 2022 ;  Lynam  et\nal. , 2020 ). Indeed, statistical models are more generalizable to other\npopulations and easier to interpret.\nBoth the updated models provide more accurate predictions for the current IVF population\nand can be used as counselling tools in fertility clinics within the UK. Before initiating\ntreatment, the revised pre-treatment model can be used to inform clinicians and couples of\ntheir individualized estimates of treatment success over multiple complete cycles of IVF.\nThen, after the first fresh embryo transfer, the recalibrated post-treatment model can\nprovide a revised estimate of treatment success using treatment-related information.\nClinicians can use these models in their daily practice to shape couples’ expectations by\ninforming them of their individualized chances of live birth over a sequence of multiple\ncomplete cycles of IVF.\nOur models should not be used for excluding couples from treatment. A model which is\nintended for use in clinical decisions, such as whether or not to have treatment, should\nbe developed using data from patients who were not treated as well as patients who were\ntreated, preferably using data from randomized controlled trials with treated and\nuntreated patients. This would allow us to assess treatment effectiveness (i.e. are\ncouples more likely to have a baby with or without IVF?) and treatment benefit (if they\nare more likely to have a baby with IVF, is the increase in the predicted chance worth the\nphysical, emotional and financial burden of the treatment?). Our prediction models are not\nmeant to aid decisions around whether to have IVF or ICSI. Such a decision must be made\nbefore using the models to make predictions in new patients. For models that aim to\nfacilitate decisions on treatment type, a different causal modelling approach is required\nwhen only observational data is available ( Sperrin\n et al. , 2019 ).\nThe original McLernon models were converted into the OPIS online calculator so that they\ncould be used in clinical practice to estimate the probability of live birth based on the\ncharacteristics of the couple and treatment ( https://w3.abdn.ac.uk/clsm/opis ). Since both the original models underestimate\npredicted cumulative live birth for couples in the recent UK IVF cohort, conversion of the\nupdated models into a new online calculator is required. The updated online calculator\nwill be able to provide accurate and more up-to-date predictions to both clinicians and\ncouples considering IVF/ICSI treatment.\nWhile we did not involve patients and clinicians in this validation study, our online\nOPIS calculator has been updated with an optional questionnaire for patients and\nhealthcare professionals to obtain feedback on the tool. We will use the findings to make\nfuture refinements to the models and our calculator.\n\nThe updated McLernon prediction models provide accurate predictions of cumulative live\nbirth over multiple complete cycles of treatment which reflect current UK IVF practice.\nThese models, which will be available in our updated OPIS calculator ( http://w3.abdn.ac.uk/clsm/opis ),\ncan be used as counselling tools to inform couples of their prognosis before commencing\nIVF/ICSI treatment as well as after the first fresh embryo transfer. They will help couples\nprepare emotionally and financially for their future treatment.","source_license":"CC-BY-4.0","license_restricted":false}