An artificial intelligence-based model to reduce the no-show rate in outpatient clinics of an academic hospital

An artificial intelligence-based model to reduce the no-show rate in outpatient clinics of an academic hospital | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article An artificial intelligence-based model to reduce the no-show rate in outpatient clinics of an academic hospital Kjeld Aij, Josta Knoester, Ben Werkhoven This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-3743388/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Purpose non-attendance of patients for outpatient appointments, known as "no-shows," poses a persistent challenge for healthcare facilities, with significant repercussions for both patients and healthcare systems. This study aimed to investigate whether targeting high-risk individuals with interventions could effectively reduce the rate of no-shows within reasonable resource allocation. Methods we developed an artificial intelligence (AI) algorithm-based prediction model to estimate the likelihood of an appointment resulting in a no-show. Utilizing retrospective data from 24 outpatient clinics, a machine learning (ML) model was constructed and trained to identify patients at high risk of no-show. Subsequently, over a 6-month period, 35% of the highest-risk patients were randomly assigned to either the intervention group (receiving a reminder phone call three workdays before their appointment) or the control group (no reminder call). Results following the intervention, the intervention group experienced a notable 26.2% reduction in no-shows. This reduction translates to a 14.3% decrease in the overall number of no-shows, demonstrating the efficacy of the reminder service. Moreover, this intervention led to additional benefits, including the ability to schedule new patients on previously avoided no-show slots, enhanced patient experience, reduced staff preparation time for missed appointments, and a decrease in administrative burden associated with rescheduling no-shows. Conclusions Our AI-powered model proved to be an effective tool for identifying high-risk patients prone to missing their outpatient appointments. This allowed for targeted interventions, such as reminder phone calls, to be implemented. The substantial reduction in no-show rates underscores the potential impact of this approach on optimizing healthcare resource allocation and improving patient attendance. AI (Artificial intelligence) Appointment reminder ML (Machine learning) Patient Non-attendance Patient No-show Predictive modelling Figures Figure 1 Highlights In our institution, over 40,000 patients do not attend their appointments (known as a ‘no-shows’) every year. Patient non-attendance can prevent patients from receiving optimal care, increase waiting times, and contribute to an inefficient use of healthcare resources. Artificial intelligence-based models can help to estimate which patients would have the highest risk of not showing up for their appointments, so they can receive targeted interventions. Through using a predictive model followed by a targeted phone call intervention, we show that it is possible to reduce the rate of no-shows by 14.3%, which translates into 6,070 no-shows prevented every year. Fewer no-shows can shorten waiting lists, improve patient care and experience, and improve workload and cost efficiencies in healthcare facilities. 1 Introduction Hospitals, outpatient clinics and healthcare systems analyse the number of missed appointments to optimise the use of healthcare resources and improve the quality of care. The National Healthcare System (NHS) in England defines failed attendance as the ‘number of patients for whom admission was arranged but failed to attend and did not tell the hospital in advance that they would not be coming’ [ 1 ]. Researchers from the Adult Congenital Heart Disease Clinic in Belgium defined missed appointments as ‘no-show of the patient for a scheduled outpatient visit without sufficient notification, or any notification at all’ [ 2 ]. The percentage of missed outpatient visits reported by NHS England is substantially high. Of the 122 million scheduled visits in 2021‒2022, 7.8 million (6.4%) were missed, which translated into approximately 650,000 missed outpatients appointments per month [ 3 ]. A systematic review by Dantas et al. [ 4 ] analysed the findings from 105 papers investigating patient non-attendance as well as patient characteristics associated with no-show behaviour: a mean no-show rate of 23% was identified, ranging from 13.2–43.0%, depending on the study. Researchers from the outpatient Adult Congenital Heart Disease Clinic in Belgium followed up patients 16 years of age or above with congenital heart disease from 2007 to 2009 [ 2 ]. The study reported that 8.2% (n = 281) of the 3,432 scheduled visits across 230 unique patients were missed due to patient non-attendance (i.e., without patients notifying the clinic). A Spanish prospective observational study analysed attendance data in an outpatient HIV/AIDS clinic of a tertiary hospital from January to December 2006 [ 5 ]. The study found that the percentage of no-shows was 5.9% (103 out of 1,733 patients). A hepatology clinic in a tertiary academic centre in the United States collected data between January 2016 and December 2021 [ 6 ]. Of 3,404 scheduled appointments, 13.5% (n = 460) were recorded as missed visits. A randomised controlled trial from the United States analysed no-shows among adults with substance use disorders who were seeking treatment; here, the no-show rate ranged from 28–54% depending on the patient’s profile [ 7 ]. Overall, results show that outpatient non-attendance is a common challenge observed worldwide and across a wide range of specialties and healthcare facilities. A variety of patient- and appointment-related characteristics have been identified as potential predictive factors that may predispose individuals to not showing up for scheduled medical appointments. Among patients with congenital heart disease, no-shows were more common in men than women (odds ratio [OR] 1.57; 95% confidence interval [CI]: 1.18–2.09); in patients without previous cardiac surgery (OR 1.46; 95% CI: 1.08–1.97); and for morning appointments (OR 1.45; 95% CI: 1.10–1.92) [ 2 ]. Among patients with substance abuse disorders, the rates of no-shows were significantly higher in patients with comorbidities versus those without (54% vs 28%; p < 0.01) [ 7 ]. Among patients with HIV, no-shows were more common in those with hepatitis C and alcohol-associated liver disease (OR 4.0; 95% CI: 3.2–4.9 and OR 2.7; 95% CI: 1.7–4.2, respectively) compared with those with other liver diseases [ 6 ]. A longer waiting time (> 30 days) from the time the appointment is scheduled to the actual appointment day has also been identified as a predictive factor for patient no-shows (OR 1.8; 95% CI: 1.5–2.2) [ 6 ]. The systematic literature review by Dantas et al. [ 4 ] cited above identified the following patient characteristics as most associated with no-shows: adult of younger age, a lower socioeconomic status, distance between the patient’s home and the clinic, and lack of private insurance. Moreover, key appointment-related factors leading to patient non-attendance included the time between the appointment was scheduled and the actual appointment date, as well as a previous history of no-show behaviour [ 4 ]. Missing scheduled medical appointments can have a negative impact on a patient’s health status. No-shows have been linked to an increased risk of morbidity and mortality as well as gaps in patient care. Among patients with HIV from an outpatient infectious disease clinic, rates of hospital admissions and mortality were significantly higher for no-show patients compared to those who attended their appointments (27.2% vs 8.9%, p < 0.001; and 5.8% vs 0.7%, p < 0.001, respectively) [ 5 ]. The researchers concluded that missed scheduled appointments can serve as an independent predictor of hospital admission and mortality [ 5 ]. In a study conducted among patients with congenital heart disease, Goossens et al. [ 2 ] found that missed appointments can be considered a predicting factor for care gaps, as patients who missed appointments in the preceding 2 years had 19 times greater probability of care gaps in comparison to patients without missed appointments; however, no-shows were not associated with increased mortality in this particular study [ 2 ]. In addition to gaps in the care process of patients, patient non-attendance has also been linked to increased waiting time for medical appointments and a higher rate of admissions to emergency departments [ 8 ]. In addition to having a negative impact on patients’ health, missed appointments also have negative consequences for healthcare systems, as they disrupt the normal functioning and efficiency of healthcare institutions, increase the use of healthcare resources and can lead to lost revenue [ 8 ]. An outpatient endoscopy unit performing 24 procedures per day evaluated the cost related to no-shows [ 9 ]. Three main cost categories were used for colonoscopy procedures: operating costs (capital investment, equipment costs and staffing costs), overtime costs and patient waiting time costs. Considering an estimated 18% rate of no-shows using historical data, the net loss due to no-shows was 16.4% or $ 725.42 per day [ 9 ]. In an outpatient vascular unit located in the United States with 12% rate of no-shows and an average of 7.6 missed visits per week, the gross revenue loss due to no-shows was estimated at $ 89,107 annually [ 10 ]. Considering a nationwide scale, the annual cost of missed outpatient hospital appointments has been estimated at £1 billion for the NHS England and $ 150 billion for the US healthcare system [ 11 ]. Given the high financial impact associated with missed appointments, measures to prevent no-shows can translate into cost savings for healthcare systems and institutions. For example, the development of an online, patient-led booking system at Guy’s and St Thomas’ NHS Foundation Trust in the UK reduced missed appointments by 11% in a 6-month period and led to savings of about £2.6 million [ 3 ]. Several intervention strategies have been investigated with the aim of decreasing the rate of patient no-shows and the associated revenue loss. Some of these interventions include overbooking appointments, appointment reminders, e-mail communication, or the use of various appointment scheduling systems. Overbooking is a relatively inexpensive and easy method to implement [ 9 ]. However, a disadvantage of this method is the level of staff dissatisfaction associated with overtime when more patients than initially expected show up for their appointments. Also, while overbooking is helpful in reducing healthcare costs related to no-shows, it does not help patients who are left without proper care due to their non-attendance [ 9 ]. Reminders sent to patients before their visits can be helpful; however, patients may need to be carefully selected for this type of intervention to be successful. As an example, reminders were not effective in a small vascular unit, in which patients who agreed, received an automated reminder phone call three to four days before their visit. Here, the rate of no-shows was significantly higher in patients who received a phone reminder versus those who did not (8.9% vs 5.9%; p < 0.0001). To explain this, the authors acknowledged the fact that they had not compared the compliance of each study group, so the group who did not agree to receive reminders could inherently be a more compliant group who did not see any benefit from receiving reminders, whereas the group who agreed to receive automated reminder phone calls could be more inclined to not attend their appointments, regardless of an intervention being implemented [ 10 ]. A systematic literature review of studies reporting non-attendance in the context of hospital admissions, showed that automated cell phone and short message service reminders were less effective in comparison to manual phone calls (i.e., calls made directly by the staff) [ 12 ]. The pooled estimates based on identified studies showed a reduction in the non-attendance rates of 39% and 29% versus baseline for phone calls and automated reminders, respectively [ 12 ]. In the past, e-mails generated through a triage-based system have been found to be ineffective in reducing no-shows at two university-affiliated primary care centres in the United States [ 13 ]. Overall, appointment reminders, whether e-mail or phone-based, can be ineffective if they are not tailored to the individual service/institution and its patient population [ 14 ]. Regarding the various appointment scheduling systems, the use of an online-scheduling system versus traditionally-scheduled appointments (i.e., via reception staff) was not more effective in reducing waiting time or missed appointments, except for new patients [ 15 ]. Given the ongoing challenge of patient non-attendance, several predictive models have been developed to try to anticipate which patients are at high risk of non-attendance, with the aim of designing interventions which can reduce the number of missed appointments and their negative consequences for patients and healthcare systems [ 16 ]. Among these, the use of artificial intelligence (AI) and machine learning (ML) techniques can further reduce the uncertainty of patient non-attendance because they can better capture and analyse variability among patient populations [ 17 ]. At our institution, missed outpatient appointments are an ongoing concern despite attempts to mitigate their negative impact on the use of resources and personnel time while ensuring patients’ access to medical services. The aim of this study was to explore the use of an ML model to estimate the risk of future no-show patients, and how this can help support targeted, phone call-based interventions for patients who are at high risk of not showing up for their outpatient appointments. While a personal phone-based reminder has been shown to be effective, its main downside is the amount of resources required to contact every single patient. By using an ML model we focus our resources on the population with the highest risk of non-attendance, such that most patients who will be no-show have been contacted while significantly reducing the number of patients that need to be called. Additionally, the number of patients that is contacted can be easily adjusted and therefore fit to the available capacity, resources or needs of the organisation. We hypothesised that directing an intervention to target the population at high risk of no-show may reduce the rate of non-attendance within reasonable costs. 2 Material and methods 2.1 Definition no-show In order to consistently measure and analyse the data it was important to carefully define what we consider to be a ‘no-show’. Since the negative consequences of a no-show occur when a timeslot is reserved but unused, we defined an appointment as a no-show when this occurred. Therefore, for the purpose of the reminder service, we considered an appointment to be a ‘no-show’ when either: 1. The appointment was fulfilled and registered by the staff as a ‘non-attendance’ in the EHR or 2a. The appointment was cancelled or rescheduled less than 24 hours before the scheduled time and 2b. The registered cause of cancellation or rescheduling was related to the patient. Condition 2a was included because, just like condition 1, this also results in an unused but reserved timeslot. According to the clinical staff, if an appointment is cancelled or rescheduled shortly before the appointed time, it is typically very difficult for the staff to fill the empty timeslot. Therefore, condition 2a also results in the same negative consequences, for both patient and hospital, as condition 1. Condition 2b was included specifically for the reminder service targeted at patients at risk of no-show, since some appointments may be registered as a no-show when in fact the cause was unrelated to the patient (e.g., absence of the healthcare professional). A reminder service for the patient is ineffective on such cases. Note that our definition of no-show is in line with the definition of ‘failed attendance’ of the NHS [1], although we expanded on this definition slightly to include cases where the clinic is notified of the patients non-attendance shortly before the appointment. 2.2 Design reminder service Not every single outpatient appointment was included when planning the reminder service. Appointments were only taken into account for the reminder service if the appointment was: 1. Planned to take place with the patient physically present at the hospital 2. Planned at least 7 days ahead 3. Of such a type or (medical) content to be excluded on request of the outpatient clinic. Among these three criteria, condition 1 was included because the negative consequences of a no-show for phone or virtual appointments (i.e., e-mail, video) are minimal in contrast to face-to-face appointments. Condition 2 was included because it was deemed unnecessary to remind patients of appointments that they had made within the same week. Condition 3 was included so outpatient clinics could exclude certain appointment types, e.g., appointments for which every patient is called beforehand for medical reasons. Every working day, the list of patients to be called was constructed as follows. The ML model was used to generate a risk score for the entire outpatient population eligible for the reminder service as defined by the criteria outlined above. For every single outpatient clinic included, 35% of patients with the highest risk score of being a no-show were classified as the “high-risk” group, whereas 65% of patients were considered “lower risk” (“low-risk group). Patients in the high-risk group were then randomly allocated to either the intervention group or the control group. Patients in the intervention group received a reminder phone call three working days before their appointment. Phone calls were made by either a call centre or students from our institution (50% each, randomly selected from the intervention group). Patients in the control group had no intervention (i.e., no reminder phone call). The patient flow during the study is illustrated in Fig. 1 . Every patient was called after working hours (between 17:00 and 20:00) in order to increase to chance of reaching the patient. Since personal information would be discussed during the phone call, the patients were first asked to identify themselves (name and date of birth) before continuing with the call. If a patient could not be reached with a first attempt, a second attempt was done later that evening. All results were logged and saved in a secured database for later analysis. The staff who performed the phone calls only had access to the list of patients to be called for that day, in order to minimise privacy risks. Fig. 1 Schematic representation of the patient flow during the study. Red figures represent a no-show patient, a green figure represents a show patient. 2.3 AI model and strategy for predicting no-shows The AI-based model was developed to generate a risk score of non-attendance for every patient eligible for the reminder service. We used Structured Query Language queries to gather data from Electronic Health Records (EHR), Python to analyse the data and train the model, Azure DevOps to schedule the daily data preparations and predictions, and a web application (developed with C#) which showed the details of the patients allocated to receive the intervention each day. We used a gradient boosted decision-tree model, which is based on the extreme gradient boosting (XGBoost) algorithm, for applied ML for structured or tabular data [18]. The XGBoost ML library produces scalable and distributed decision trees and performs regression analysis as well as classification and ranking of variables using parallel tree boosting [19, 20]. Our model was trained on data from appointments that would have been eligible for the reminder service for outpatients treated at the participating clinics. Data included patient demographic and clinical characteristics as well as reasons for missing an appointment, which are routinely recorded in our database by technologists as part of the quality assurance system used at our institution. We tested selected variables and how they increased or decreased the risk of no-shows. Only variables that considerably improved model performance were included. In addition, all the predictive variables considered were stable over time to avoid issues with data drift. The list of variables used to determine the risk of non-attendance is presented in Table 1. Table 1 Predictive variables for no-shows by patient and appointment characteristics or history Patient characteristics Appointment characteristics Appointment history · Distance (‘as the crow flies’) from home to the medical institution in km · Age · Time between scheduling the appointment and the actual appointment · Multiple appointments that day · Number of days since previous appointment · Duration of the appointment · Type of consultation · Appointment previously rescheduled · Referral · Appointment rescheduled by a physician · Number of previous no-shows · Patient punctuality: average arrival time before appointment (late, just in time, on time, early) · Number of previous appointments · Percentage of no-shows 2.3 Pilot study and timeframe for main study To assess the feasibility of using our AI model for predicting no-shows, the study was preceded by a pilot study. The pilot study was performed in August 2021 at the dermatology outpatient clinic. The dermatology clinic was chosen for the pilot study since this clinic has a relatively high percentage of no-shows in our institution and staff members from this clinic were willing and motivated to help with implementing and testing the reminder service. An ML model was trained on previous appointments at the dermatology outpatient clinic. Several different types of ML models were tested (Random Forest, Logistic Regression and XGBoost) and the XGBoost model showed the highest predictive power. Patients predicted by the AI model to be at high risk for no-show received a reminder call before their appointment. The pilot trial showed a 12.8% reduction in no-shows, which was considered sufficient to proceed to the next stage of the project, which involved further refinement of the AI model and testing across a wider variety of outpatient clinics at our institution. The main study, described in detail in this publication, was conducted from June 2022 to January 2023 (6-month period). The AI model was trained with retrospective data and kept unchanged for the duration of the study to ensure the reproducibility of the results. In this main study, the entire outpatient population from 24 outpatient clinics at our institution were included. 2.4 Ethical considerations All variables used by the model underwent review and successfully completed the Data Protection Impact Assessment (DPIA). Privacy risk was considered to be very limited. For example, the variables used are not medical in nature and cannot be traced back to an individual. In addition, the negative consequences for a patient identified as being at high risk of no-shows by the prediction model are minimal (i.e., the patient will receive one possibly unnecessary phone call), and the positive consequences for both patient and hospital were deemed to outweigh the negative ones. Lastly, every single feature that the model used was tested such that only features that probably improved the predictive power of the model were included (data minimalization). 3 Results 3.1 Reduction of no-shows and additional patient- and staff-related benefits Over the 6-month period evaluated in the main study, a total of 11,778 of high-risk patients were allocated to the intervention group and 15,399 to the control group. The control group is larger than the intervention group because patients from the intervention group that were not attempted to be reached (due to, for example, staff shortage), were then allocated to the control group. In the intervention group, 70% of patients designated to receive a phone call were able to be reached. Overall, the study showed a 26.2% decline in no-shows for the intervention group. Different levels of improvement were observed across the 24 outpatient clinics: the reduction in no-shows ranged from 5–69%, depending on the outpatient clinic profile and size. However, all outpatient clinics showed a reduction in no-shows following the intervention, even those historically linked to a lower proportion of patient non-attendance. We did not observe an increase in rescheduled appointments due to the implementation of reminder calls. Only 4.5% of the patients who were reached through a reminder phone call wanted to reschedule their appointments. If this was the case, the outpatient clinic staff received an email notification that the patient wanted to reschedule their appointment so they could be reached the following working day. In addition, when appointments were rescheduled due to the reminder phone call, new appointments for other patients could be often scheduled in the cancelled slots because there was sufficient time. Reminder calls were also useful to update contact details for many patients as, during the study period, 110 e-mail addresses and 40 phone numbers were able to be updated in patients’ medical records. In addition, patient feedback around the intervention was overall positive, as patients frequently appreciated being able to speak directly to a member of the staff. This allowed administrative errors to be found, for example a patient who had asked for their appointment to be rescheduled but was not (yet) correctly adjusted. Some additional benefits were also observed in relation to our employees. Typically, staff will prepare ahead of time for any scheduled appointments, hence, through reducing patient no-shows, we can reduce the time that staff invests on the preparation of appointments which are then not attended by patients. In addition, reducing the number of no-shows is expected to decrease the administrative time necessary to reschedule these appointments which, in our institution, has been estimated to range between 5 and 20 minutes per no-show. 3.2 Estimation of no-shows avoided during a 1-year period The measured 26.2% reduction of the number of no-shows in the intervention group can be extrapolated to an expected reduction in the total number of no-shows if all high-risk patients would have been called. Assuming that the control group would have had the same reduction in the number of no-shows if they had been called, we can extrapolate that the reminder service would lead to a reduction of 14.3% in the total number of no-shows occurring at the outpatient clinics participating in the reminder service. As there were 42,445 no-shows in our institution in 2021, this extrapolation of the data from the 6-month study period would result in 6,070 no-shows which could be prevented during a year. Consequently, after the implementation of the AI model and the introduction of targeted phone calls, it was estimated that our institution could reschedule up to 6,070 visits and provide additional care for up to 6,070 patients. 3.3 Estimated cost of the intervention during a 1-year period We estimated the cost of the reminder phone calls during an entire year. Reminder phone calls were to be made every working day to 35% of the patients identified as “high-risk” of not showing up. The cost of the call was €1.50 per patient reached for those calls made from the call centre, and €0.72 per patient for calls made by students. In addition, a cost of €30,000 per year was also estimated, to account for information technology (IT) management and hardware acquisition for the AI model. Overall, the total annual cost was estimated at €262,276 for the call centre versus €114,454 for calls made by students. 4 Discussion Our study showed that the use of an AI model to predict patients at high risk of non-attendance, followed by a targeted intervention via phone call reminders, allowed for a 14.3% decrease in patient no-shows across 24 outpatient clinics from our institution. Additional observed benefits from the intervention were the ability to see more patients through scheduling appointments in the prevented “no-show” slots, the possibility of updating and improving the quality of data on patients’ medical records, an overall improved patient experience, reductions in the time invested by the staff to prepare for no-show appointments and a reduction in the administrative work needed to reschedule no-shows. Predicting patient non-attendance to medical appointments is a complex process. Typically, researchers select different sets and numbers of variables to increase prediction accuracy. The grouping of risk factors facilitates their analysis, but differences across healthcare institutions and patient populations mean that predictive variables are not universal. Mieloszyk et al. [ 21 ] reported a model for predicting no-shows in the radiology department of a US clinic. Data were grouped into three categories: patient-related, exam-related and scheduling-related. The final model included 16 variables, of which the type of examination and those related to an appointment were most informative in the prediction of no-shows. A model developed by Alaidah et al. [ 22 ] indicated that the patient history of missed appointments had the highest predictive weight; however, factors such as a longer time between booking the appointment and the appointment itself also increased the risk of non-attendance. The number of variables inputted in models tends to increase with the complexity of the model. In a recent model published by Nelson et al. [ 23 ], which included 81 variables, previous non-attendance was the eighth most important predictive factor after time between referral and appointment, number of previous appointments for imaging studies, home latitude, distance from home to hospital, number of previous magnetic resonance imaging (MRI) appointments, home longitude and total care cost. Dashtban et al. [ 24 ] considered eight variable groups for prediction of non-attendance, including demographic data, patient history, appointment characteristics, time variables, patient appointment history, socioeconomic data of the patient, weather conditions and admission history. A study by Dunstan et al. [ 25 ] from Chile concluded that factors pertaining to a socioeconomically disadvantaged background were the most relevant to predict no-shows. The predictive value of variables linked to low socioeconomic status have also been observed in other studies, although there seems to be a country-specific component [ 6 , 26 ], which further strengthen the need for models highly adjusted to the local patient population. We grouped the predictive variables into three categories, including patient characteristics, appointment characteristics and patient appointment history, which is similar to the approach followed by other studies. In addition, the predictive features selected for our model were based on what variables have been shown to be effective by other studies, as well as features that were relevant and possible to attain from our own institution, to tailor the model as much as possible to the outpatient population from our geographical area. We specifically selected features that would be applicable to a wide variety of outpatient clinics and which are expected to remain stable and well-registered over time. Every feature included in the final model was tested whether the feature improved the predictive performance of the model. Features that added no or negligible predictive power were excluded from the model. Several ML predictive models have been investigated in healthcare, showing different levels of accuracy and efficacy. We tested different types of predictive models and found that XGBoost resulted in the highest predictive power for no-shows. This result was also found by the following studies. Yang et al. [ 27 ] tested different models, including logistic regression, random forest, support vector machine and XGBoost, to predict early neurological deterioration in patients after ischemic stroke treated with mechanical thrombectomy. XGBoost had the highest prediction power [area under the curve (AUC) = 0.826; 95% CI: 0.781‒0.871] in comparison to logistic regression (AUC = 0.644), random forest (AUC = 0.819) and support vector machine (AUC = 0.643) [ 27 ]. Lee et al. [ 28 ] analysed several models to compare their accuracy and complexity of deployment for predicting no-shows. The variables with the strongest prediction power were the number of days since the last visit, the last appointment status and the appointment waiting time. The XGBoost model gave the highest prediction power (AUC = 0.832) and highest precision (0.785), in comparison to other models such as decision trees (AUC = 0.760; precision = 0.718), logistic regression (AUC = 0.776; precision = 0.718), random forest (AUC = 0.805; precision = 0.729), elastic net (AUC = 0.808; precision = 0.734) and gradient boosting trees (AUC = 0.819; precision = 0.733) [ 28 ]. Chen et al. [ 29 ] reported that the prediction power exerted by the XGBoost model for predicting no-shows for follow-up visits in an academic paediatric ophthalmology clinic was very high and reached an AUC of 0.90 and precision of 0.74. These results were better than those achieved when using other algorithms to train the model. Random forest was the second-best model (AUC of 0.88 and precision of 0.69). The remaining models investigated were the support vector regression (AUC = 0.81 and precision = 0.50) and LASSO regression (AUC = 0.74 and precision = 0.27). Chen et al. [ 29 ] also reported that the predictive capabilities of the model were much lower for new patient appointments in comparison to follow-up visits. Of note, the final predictive power and precision of a model have also been shown to depend on the type of medical institution and patient characteristics [ 30 ]. These two factors were also considered when training our AI model. Non-attendance rates remain high despite the implementation of various interventions to reduce it. For the NHS England, the percentage of no-shows dropped from 8.3% in 2007‒2008 to 6.7% in 2017‒2018 [ 31 ], even after the introduction of a wide range of interventions, highlighting the need for additional, more refined solutions based on advanced technologies, such as AI. One of the first AI models to predict no-shows was developed by Dravenstott et al. [ 30 ]; this model used artificial neural networks to predict the no-show risk separately for primary care and endocrinology clinics. To teach the model, historical data on 3 million visits over the previous two years were included, which covered clinic, provider and patient-specific variables. The predictive performance of the model was below the attendance rate with an AUC of 0.78 and 0.81 for the primary care and endocrinology clinics, respectively [ 30 ]. Another model, which also used deep neural networks, was proposed by Dashtban and Li [ 24 , 32 ]; this model was based on sparse stacked denoising autoencoders (SDAEs). The model was trained using in-hospital data collected from an acute care NHS hospital in the UK. Approximately 1.6 million records were included. The model used SDAEs with logistic regression and softmax layer to produce an AUC of 0.70, which was claimed sufficient to be used in real practice [ 24 , 32 ]. Lee et al. [ 28 ] tested several ML models and finally opted for the XGBoost model, which provided the highest AUC (0.832) and highest precision (0.785). The proposed variables for the model by Lee et al. were sourced from the data warehouse, extracted from the literature and advised by domain experts. Twenty-two variables were finally engineered for modelling and linked to more than a million outpatient appointment records [ 28 ]. Nelson et al. [ 23 ] reported results from a high-dimensional gradient boosting machine-based model to predict no-shows, which was tested in 22,318 consecutive scheduled MRI appointments at two hospitals from the University College London. Their high-dimensional gradient boosting machine-based model showed an AUC of 0.852 and an average precision of 0.511. Optimal predictive performance of the model was observed when using 81 variables, which represents 2 to 4 times more variables in comparison to other models from the literature. The added value of the study by Nelson et al. [ 23 ] is the inclusion of potential cost benefit calculations that had not been reported in other studies. The authors made the following assumptions regarding costs of services: cost of model infrastructure and development was approximately £20,000, cost of an MRI study was £150 and cost of reminder calls was £6 per patient. The efficacy of the intervention (i.e., reminder calls) was estimated at 33%, meaning there was a 33% reduction of no-shows following the use of the predictive model plus the intervention. The net operating benefit of using the model was £3.15 per appointment with a break-even point at around 6,350 scheduled MRIs. In an average NHS hospital trust performing about 20,000 MRI scans annually, it was estimated the break-even point would be reached after about 4.5 months (83 working days) [ 23 ]. It is difficult to compare the effectiveness of our AI-based predictive model with that of other models. Several of the models reported so far have been implemented in radiology departments [ 21 , 23 ], while our model is intended to be used across different specialties. In addition, only the AI-based model by Nelson et al. [ 23 ] described earlier has provided an estimate of financial/cost benefits, although this was also provided in the context of radiology [ 23 ]. A high-level comparison of the model reported by Nelson et al. [ 23 ]. and our model reveals several differences, including model structure (high-dimensional gradient boosting machine-based models vs XGBoost), number of variables analysed (81 vs 13), effectiveness of the intervention (33% vs 14.3%), phone call cost (£6 vs €0.72‒€1.50) or IT investment (£20,000 vs €30,000). Due to substantial differences between studies, the financial results cannot be compared; however, both studies concluded that the high predictive performance offered by AI models provides operating and financial benefits. Our model also showed that reducing the risk of no-shows brings benefits to several stakeholders. First, there were positive effects for patients. The total number of missed appointments prevented, which for our institution was estimated to be up to 6,070 appointments per year, could be either rescheduled or offered to other patients, potentially reducing waiting time for patients. Lee et al. [ 28 ] compared the rate of released appointments, defined as the percentage of cancelled or rescheduled appointment slots over the total number of scheduled appointments, among different groups of patients as stratified by an AI model. The highest percentage of released appointments was observed in the group at high risk of not showing up: 3.9% of appointments were released in the low-risk group, 14.6% in the medium-risk group, and 31.9% in the high-risk group (p < 0.05). The results reported by Lee et al. [ 28 ] also support that interventions to reduce no-shows should be directed towards patients at high risk of missing their appointment. Moreover, for patients with low digital literacy, receiving a phone call could translate into more inclusive care, as they will be able to be reminded of their appointment through an analogue method which is more familiar to them, rather than through digital reminder strategies, such as phone text messages or e-mails [ 33 ]. In our study, the overall patient experience for those patients receiving the intervention appeared to improve, as patients often shared positive feedback when they had the opportunity to speak directly on the phone with a member of the call centre or student staff. In addition, if the appointment needed to be rescheduled, patients would also speak with a member of the outpatient clinic staff for rescheduling purposes. Lastly, the intervention also provided an opportunity to update personal contact details for some of the patients at high risk of no-shows, which could help facilitate communication with them in the future to continue to reduce their risk of being a no-show. We also observed benefits for our staff following the use of the AI-based model and targeted intervention. The possibility of seeing additional patients who covered the slots of prevented no-shows did not seem to be associated with additional workload for the staff. On the contrary, reducing no-shows may help improve efficiencies and minimise the time invested by healthcare professionals on preparing for appointments that then go onto being a no-show. On the other hand, some staff reported that a no-show gave them time to catch up with other tasks; this would not be possible if there are less no-shows. However, since no-show appointments need to be rescheduled, this apparent time gain to catch up on other tasks can be considered a short-term gain but a long-term loss (from the need to eventually reschedule the no-shows). Another benefit is the potential for improved distribution of workload and operational efficiency: as there are fewer missed appointments that need to be rescheduled, it is expected that there will be less risk of waiting times increasing and appointment dairies getting overbooked. In addition, the reduction in no-shows was perceived by the staff to be associated with less administrative work through avoiding the extra time needed from reception workers to reschedule appointments. Similar effects were shown by other researchers who reported that, by reducing no-shows, technologists and physicians could devote more time to care for patients, rather than investing it on administrative tasks [ 34 , 35 ]. Lastly, the potential for increasing revenue through the reimbursement of more appointments who can be seen in place of the no-shows is expected to have a positive financial impact on the overall institution. 4.1 Strengths, limitations and future research A key strength from our study lies in its ability to show the real-world benefits of using an AI-based model in the outpatient setting from a large medical institution. Many previous studies, although based on real-world data, have only described the mathematical approach to predicting no-shows based on patient characteristics that place patients at high risk of missing medical appointments [ 28 , 30 , 32 , 36 ]. An additional strength of our study is that the reduction in no-shows was shown across a wide range of outpatient services from various specialties, including those with lower rates of historical no-shows, suggesting the use of predictive models based on AI to predict patients at high risk of non-attendance followed by a targeted intervention can be of benefit regardless of the specialty. Lastly, the reduction in no-shows was associated with observed benefits for patients, hospital staff, and the institution, including some benefits that may not be as obvious when the model and intervention are being designed, such as the improved patient experience or the enhanced patient care for patients with poor digital literacy. This study also has some limitations. Firstly, it is important to note that some no-shows are caused by reasons that a reminder service cannot prevent, e.g., a no-show due to unavailability of the healthcare professional or if the contact details of the patient are incorrect. Additionally, a reminder service is ineffective for appointments scheduled a few days in advance; however, these cases are a minority, although they require a different strategy to prevent non-attendance. Like any other AI-based model, if our model is to be used in a different healthcare institution, it would need to be adapted to the specific target population and relevant predictive variables, which may be different than those used in the current version of the model. Moreover, the prediction model would need to be trained with the new dataset and adapted in order to function with different data and software settings, since it is expected that different hospitals will have significantly different patient populations. Currently, the list of patients to be called is comprised of the 35% of patients with the highest risk of not showing up for every outpatient clinic separately. This setup was chosen in order to show the efficacy of the reminder service for each individual clinic. Now that this goal has been achieved, an improvement which can be easily implemented is to create a list of patients to be called including the 35% of patients at the highest risk of not showing up from the entire outpatient patient population. A more challenging improvement to the remainder service is to analyse the intervention effects. As the design includes both an intervention and a control group, it is possible to analyse for which patient profiles the reminder call was most effective. This would allow us to compose the list of patients to be called not solely based on which patients have the highest risk of not showing up, but also those for whom the intervention is expected to be most effective. The existing model is comprised of data currently available in the EHR, but future work from both engineers and healthcare institutions should focus on understanding additional variables which may have not been included yet in databases, but which may be important for predicting no-shows. Examples of such variables could be real commuting time, the most effective type of intervention based on the patient’s profile, or reminders sent to adolescents/young adults. Determination of additional variables based on real-world data may increase the effectiveness of predictive models in the future. Lastly, we advise to combine such a reminder service with other interventions to reduce the number of no-shows. There are many reasons why no-shows may occur, and not all of them may be prevented with a single intervention. 5 Conclusions The AI-based predictive model we report here is an effective tool for identifying patients at high risk of missing their outpatient appointments, who can then be targeted with a reminder intervention such as a phone call. Based on the results of the randomised control trial, we show that the no-show percentage of the targeted group is reduced to 26.2%, which is equivalent to a reduction of no-show appointments of 14.3% in the total patient population. The observed benefits after implementation of the model included improved patient experience, more inclusive care for patients with limited digital literacy, the ability to update patient contact information during calls, and improved overall operational efficiencies (including reductions in the administrative burden to reschedule appointments, as well as less time wasted by healthcare professionals and staff in preparing for no-show appointments). Further research and adaptations of the model to other healthcare settings and institutions as well as other patient populations are needed to confirm the high predictive performance of AI in the identification and management of no-show appointments. Declarations Competing Interests: The authors declare that they have no conflict of interest. Human ethics: Ethics approval was not required for this research. This study has been conducted in accordance with the World Medical Association Declaration of Helsinki. Funding: This study was funded by the Dijkzigt department of Erasmus MC in Rotterdam, the Netherlands. Human ethics: Ethics approval was not applicable for this research because according to the Dutch law of medical research in humans (WMO), the participants of our study were not subjected to actions or prescribed rules of conduct. Therefore, the consent to participate was not required. This study has been conducted in accordance with the World Medical Association Declaration of Helsinki (2013). Authors’ contributions: KA, JK and BW contributed to the study conception and design. Data collection and analysis were performed by JK and BW All authors contributed to data interpretation. KA participated in the drafting of the manuscript and all authors commented on draft versions of the manuscript. All authors read and approved the final manuscript. All authors agreed to be accountable for all aspects of the work and take responsibility for the integrity of the work as a whole. Data availability: The data sets used and/or analysed during the current study are available from the corresponding author upon reasonable request. Acknowledgments: We would like to thank Małgorzata Biernikiewicz and Paula Martin of Valid Insight, Macclesfield, UK for providing medical writing support. We also acknowledge Joris Tukker, Steffen Greup, Enrico Timmerman, Jaap Auwema, Raymond de Vos for their contributions to developing and implementing the model and the implementation software. References NHS England and NHS Improvement (2020) NHS inpatient admission and outpatient referrals and attendances. 2020. https://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2020/05/QAR-Commentary-Q4-201920-aIu8F.pdf . Accessed 3 May 2023 Goossens E, van Deyk K, Budts W, Moons P (2022) Are missed appointments in an outpatient clinic for adults with congenital heart disease the harbinger for care gaps? Eur J Cardiovasc Nurs 21(2):127–134. 10.1093/eurjcn/zvab012 NHS drive to reduce (2023) ‘no shows’ to help tackle long waits for care. NHS England. 2023. https://www.england.nhs.uk/2023/01/nhs-drive-to-reduce-no-shows-to-help-tackle-long-waits-for-care/ . Accessed 4 May 2023 Dantas LF, Fleck JL, Cyrino Oliveira FL, Hamacher S (2018) No-shows in appointment scheduling - a systematic literature review. Health Policy 122(4):412–421. 10.1016/j.healthpol.2018.02.002 Colubi MM, Pérez-Elías MJ, Elías L, Pumares M, Muriel A, Zamora AM, Casado JL, Dronda F, López D, Moreno S (2012) Missing scheduled visits in the outpatient clinic as a marker of short-term admissions and death. HIV Clin Trials 13(5):289–295. 10.1310/hct1305-289 Rustam LB, Vander Weg M, Chrischilles E, Tanaka T (2023) Sociodemographic and Clinical Factors Associated with Nonattendance at the Hepatology Clinic. Dig Dis Sci. 10.1007/s10620-023-07951-z Angarita GA, Reif S, Pirard S, Lee S, Sharon E, Gastfriend DR (2007) No-Show for Treatment in Substance Abuse Patients with Comorbid Symptomatology: Validity Results from a Controlled Trial of the ASAM Patient Placement Criteria. J Addict Med 1(2):79–87. 10.1097/ADM.0b013e3180634c1d Marbouh D, Khaleel I, Al Shanqiti K, Al Tamimi M, Simsekler MCE, Ellahham S, Alibazoglu D, Alibazoglu H (2020) Evaluating the Impact of Patient No-Shows on Service Quality. Risk Manag Healthc Policy 13:509–517. 10.2147/rmhp.S232114 Berg BP, Murr M, Chermak D, Woodall J, Pignone M, Sandler RS, Denton BT (2013) Estimating the cost of no-shows and evaluating the effects of mitigation strategies. Med Decis Making 33(8):976–985. 10.1177/0272989x13478194 Satiani B, Miller S, Patel D (2009) No-show rates in the vascular laboratory: analysis and possible solutions. J Vasc Interv Radiol 20(1):87–91. 10.1016/j.jvir.2008.09.027 Philpott-Morgan S, Thakrar DB, Symons J, Ray D, Ashrafian H, Darzi A (2021) Characterising the nationwide burden and predictors of unkept outpatient appointments in the National Health Service in England: A cohort study using a machine learning approach. PLoS Med 18(10):e1003783. 10.1371/journal.pmed.1003783 Hasvold PE, Wootton R (2011) Use of telephone and SMS reminders to improve attendance at hospital appointments: a systematic review. J Telemed Telecare 17(7):358–364. 10.1258/jtt.2011.110707 Katz SJ, Moyer CA, Cox DT, Stern DT (2003) Effect of a triage-based E-mail system on clinic resource use and patient and physician satisfaction in primary care: a randomized controlled trial. J Gen Intern Med 18(9):736–744. 10.1046/j.1525-1497.2003.20756.x McLean SM, Booth A, Gee M, Salway S, Cobb M, Bhanbhro S, Nancarrow SA (2016) Appointment reminder systems are effective but not optimal: results of a systematic review and evidence synthesis employing realist principles. Patient Prefer Adherence 10:479–499. 10.2147/ppa.S93046 Kachooei A, Plusch K, Kasper A, D'Amore T, Beredjiklian P (2023) The effect of outpatient web-based online scheduling versus traditional staff scheduling systems on progression to surgery and no-show rates. J Res Med Sci 28:23. 10.4103/jrms.jrms_738_22 Carreras-García D, Delgado-Gómez D, Llorente-Fernández F, Arribas-Gil A (2020) Patient No-Show Prediction: A Systematic Literature Review. Entropy (Basel) 22(6). 10.3390/e22060675 AlMuhaideb S, Alswailem O, Alsubaie N, Ferwana I, Alnajem A (2019) Prediction of hospital no-show appointments through artificial intelligence algorithms. Ann Saudi Med 39(6):373–381. 10.5144/0256-4947.2019.373 Friedman JH (2001) Greedy function approximation: A gradient boosting machine. Ann Stat 29:1189–1232 Wang K, Dick T, Balcan M-F (2020) Scalable and Provably Accurate Algorithms for Differentially Private Distributed Decision Tree Learning. ArXiv. abs/2012.10602 xgboost developers (2022) XGBoost Documentation. 2022. https://xgboost.readthedocs.io/en/stable/ . Accessed 25 May 2023 Mieloszyk RJ, Rosenbaum JI, Bhargava P, Hall CS (2017) Predictive modeling to identify scheduled radiology appointments resulting in non-attendance in a hospital setting. Annu Int Conf IEEE Eng Med Biol Soc 2017:2618–2621. 10.1109/embc.2017.8037394 Alaidah A, Alamoudi E, Shalabi D, AlQahtani M, Alnamshan H, Abubacker NF (eds) Mining and Predicting No-Show Medical Appointments: Using Hybrid Sampling Technique. Communication and Intelligent Systems; 2021 2021//; Singapore: Springer Singapore Nelson A, Herron D, Rees G, Nachev P (2019) Predicting scheduled hospital attendance with artificial intelligence. NPJ Digit Med 2:26. 10.1038/s41746-019-0103-3 Dashtban M, Li W (eds) (2019) Deep Learning for Predicting Non-attendance in Hospital Outpatient Appointments. Hawaii International Conference on System Sciences; Dunstan J, Villena F, Hoyos JP, Riquelme V, Royer M, Ramírez H, Peypouquet J (2023) Predicting no-show appointments in a pediatric hospital in Chile using machine learning. Health Care Manag Sci. 10.1007/s10729-022-09626-z Tarabichi Y, Higginbotham J, Riley N, Kaelber DC, Watts B (2023) Reducing Disparities in No Show Rates Using Predictive Model-Driven Live Appointment Reminders for At-Risk Patients: a Randomized Controlled Quality Improvement Initiative. J Gen Intern Med. 10.1007/s11606-023-08209-0 Yang T, Hu Y, Pan X, Lou S, Zou J, Deng Q, Zhang Q, Zhou J, Zhu J (2023) Interpretable Machine Learning Model Predicting Early Neurological Deterioration in Ischemic Stroke Patients Treated with Mechanical Thrombectomy: A Retrospective Study. Brain Sci 13(4). 10.3390/brainsci13040557 Lee G, Wang S, Dipuro F, Hou J, Grover P, Low LL, Liu N, Loke CY (eds) (2017) Leveraging on Predictive Analytics to Manage Clinic No Show and Improve Accessibility of Care. IEEE International Conference on Data Science and Advanced Analytics (DSAA); 2017 19–21 Oct. 2017 Chen J, Goldstein IH, Lin WC, Chiang MF, Hribar MR (2020) Application of Machine Learning to Predict Patient No-Shows in an Academic Pediatric Ophthalmology Clinic. AMIA Annu Symp Proc. 2020:293–302 Dravenstott R, Kirchner HL, Strömblad C, Boris D, Leader J, Devapriya P (2014) Applying predictive modeling to identify patients at risk to no-show. IIE Annual Conference and Expo 2014.2370-8 Hospital Outpatient Activity (2018) NHS Digital. 2018. https://files.digital.nhs.uk/97/20440A/hosp-epis-stat-outp-summ-rep-2017-18-rep.pdf Dashtban M, Li W (2022) Predicting non-attendance in hospital outpatient appointments using deep learning approach. Health Syst (Basingstoke) 11(3):189–210. 10.1080/20476965.2021.1924085 Neves BB, Mead G (2021) Digital Technology and Older People: Towards a Sociological Approach to Technology Adoption in Later Life. Sociology 55(5):888–905. 10.1177/0038038520975587 Bentayeb D, Lahrichi N, Rousseau L-M (2019) Patient scheduling based on a service-time prediction model: a data-driven study for a radiotherapy center. Health Care Manag Sci 22(4):768–782. 10.1007/s10729-018-9459-1 Babayoff O, Shehory O, Geller S, Shitrit-Niselbaum C, Weiss-Meilik A, Sprecher E (2022) Improving Hospital Outpatient Clinics Appointment Schedules by Prediction Models. J Med Syst 47(1):5. 10.1007/s10916-022-01902-3 Harris SL, May JH, Vargas LG (2016) Predictive analytics model for healthcare planning and scheduling. Eur J Oper Res 253(1):121–131. https://doi.org/10.1016/j.ejor.2016.02.017 Additional Declarations No competing interests reported. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-3743388","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":264336892,"identity":"de1efeb9-9853-46b2-90ff-9ce7d3701ee9","order_by":0,"name":"Kjeld Aij","email":"","orcid":"","institution":"Erasmus MC","correspondingAuthor":false,"prefix":"","firstName":"Kjeld","middleName":"","lastName":"Aij","suffix":""},{"id":264336893,"identity":"caf5b53c-07de-4103-b074-3600d74869ab","order_by":1,"name":"Josta Knoester","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA70lEQVRIie3PMQrCMBSA4YRAXFq6JnTwCi2COAgexEUXt+wOEiJCXKRzvUWlII6FQh2MdHVUHFzbRXARFVEcbOvokH8KeXwkDwCd7g+DAr+OKPoctCoJAQD3Pgek5J03MZzfCBJ4sM+GnFu2yo/DEe866aafwVUxuX9s7foqJtRjoauSmAU7FhKoykhN2qaMiKPMJRU4uhNzCaCsIldOOso4XcSVsyDdVhGc2KZAxDEMDMcSsSBiFWSCB9RPYuor3KBjL2bzxy79EuJOZZNkI25ZM3TIxZkzL90usly2i8nk63WvEABQL5npdDqd7tkNGspRqB0lkJgAAAAASUVORK5CYII=","orcid":"","institution":"Erasmus MC","correspondingAuthor":true,"prefix":"","firstName":"Josta","middleName":"","lastName":"Knoester","suffix":""},{"id":264336894,"identity":"1a9670cc-b8de-444b-aed5-c93a2da477ba","order_by":2,"name":"Ben Werkhoven","email":"","orcid":"","institution":"Erasmus MC","correspondingAuthor":false,"prefix":"","firstName":"Ben","middleName":"","lastName":"Werkhoven","suffix":""}],"badges":[],"createdAt":"2023-12-12 10:59:33","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-3743388/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-3743388/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":49135536,"identity":"4fa5ebf9-7f65-40d9-a456-a8d1845aad7f","added_by":"auto","created_at":"2024-01-03 17:09:25","extension":"jpeg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":274395,"visible":true,"origin":"","legend":"\u003cp\u003eSchematic representation of the patient flow during the study. Red figures represent a no-show patient, a green figure represents a show patient.\u003c/p\u003e","description":"","filename":"floatimage1.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-3743388/v1/48f9b221c5b376f241412469.jpeg"},{"id":49136860,"identity":"b7890a65-a4f2-4cb8-825d-a72fb4f2560a","added_by":"auto","created_at":"2024-01-03 17:25:28","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":379814,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3743388/v1/2620e85d-5f83-4dbf-aec4-59e590fd1297.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"An artificial intelligence-based model to reduce the no-show rate in outpatient clinics of an academic hospital","fulltext":[{"header":"Highlights","content":"\u003cul class=\"decimal_type\"\u003e\n \u003cli\u003eIn our institution, over 40,000 patients do not attend their appointments (known as a \u0026lsquo;no-shows\u0026rsquo;) every year.\u003c/li\u003e\n \u003cli\u003ePatient non-attendance can prevent patients from receiving optimal care, increase waiting times, and contribute to an inefficient use of healthcare resources.\u0026nbsp;\u003c/li\u003e\n \u003cli\u003eArtificial intelligence-based models can help to estimate which patients would have the highest risk of not showing up for their appointments, so they can receive targeted interventions.\u0026nbsp;\u003c/li\u003e\n \u003cli\u003eThrough using a predictive model followed by a targeted phone call intervention, we show that it is possible to reduce the rate of no-shows by 14.3%, which translates into 6,070 no-shows prevented every year.\u003c/li\u003e\n \u003cli\u003eFewer no-shows can shorten waiting lists, improve patient care and experience, and improve workload and cost efficiencies in healthcare facilities.\u003c/li\u003e\n\u003c/ul\u003e"},{"header":"1 Introduction","content":"\u003cp\u003eHospitals, outpatient clinics and healthcare systems analyse the number of missed appointments to optimise the use of healthcare resources and improve the quality of care. The National Healthcare System (NHS) in England defines failed attendance as the \u0026lsquo;number of patients for whom admission was arranged but failed to attend and did not tell the hospital in advance that they would not be coming\u0026rsquo; [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]. Researchers from the Adult Congenital Heart Disease Clinic in Belgium defined missed appointments as \u0026lsquo;no-show of the patient for a scheduled outpatient visit without sufficient notification, or any notification at all\u0026rsquo; [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]. The percentage of missed outpatient visits reported by NHS England is substantially high. Of the 122\u0026nbsp;million scheduled visits in 2021‒2022, 7.8\u0026nbsp;million (6.4%) were missed, which translated into approximately 650,000 missed outpatients appointments per month [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. A systematic review by Dantas et al. [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e] analysed the findings from 105 papers investigating patient non-attendance as well as patient characteristics associated with no-show behaviour: a mean no-show rate of 23% was identified, ranging from 13.2\u0026ndash;43.0%, depending on the study. Researchers from the outpatient Adult Congenital Heart Disease Clinic in Belgium followed up patients 16 years of age or above with congenital heart disease from 2007 to 2009 [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]. The study reported that 8.2% (n\u0026thinsp;=\u0026thinsp;281) of the 3,432 scheduled visits across 230 unique patients were missed due to patient non-attendance (i.e., without patients notifying the clinic). A Spanish prospective observational study analysed attendance data in an outpatient HIV/AIDS clinic of a tertiary hospital from January to December 2006 [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. The study found that the percentage of no-shows was 5.9% (103 out of 1,733 patients). A hepatology clinic in a tertiary academic centre in the United States collected data between January 2016 and December 2021 [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e]. Of 3,404 scheduled appointments, 13.5% (n\u0026thinsp;=\u0026thinsp;460) were recorded as missed visits. A randomised controlled trial from the United States analysed no-shows among adults with substance use disorders who were seeking treatment; here, the no-show rate ranged from 28\u0026ndash;54% depending on the patient\u0026rsquo;s profile [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]. Overall, results show that outpatient non-attendance is a common challenge observed worldwide and across a wide range of specialties and healthcare facilities.\u003c/p\u003e \u003cp\u003eA variety of patient- and appointment-related characteristics have been identified as potential predictive factors that may predispose individuals to not showing up for scheduled medical appointments. Among patients with congenital heart disease, no-shows were more common in men than women (odds ratio [OR] 1.57; 95% confidence interval [CI]: 1.18\u0026ndash;2.09); in patients without previous cardiac surgery (OR 1.46; 95% CI: 1.08\u0026ndash;1.97); and for morning appointments (OR 1.45; 95% CI: 1.10\u0026ndash;1.92) [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]. Among patients with substance abuse disorders, the rates of no-shows were significantly higher in patients with comorbidities versus those without (54% vs 28%; p\u0026thinsp;\u0026lt;\u0026thinsp;0.01) [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]. Among patients with HIV, no-shows were more common in those with hepatitis C and alcohol-associated liver disease (OR 4.0; 95% CI: 3.2\u0026ndash;4.9 and OR 2.7; 95% CI: 1.7\u0026ndash;4.2, respectively) compared with those with other liver diseases [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e]. A longer waiting time (\u0026gt;\u0026thinsp;30 days) from the time the appointment is scheduled to the actual appointment day has also been identified as a predictive factor for patient no-shows (OR 1.8; 95% CI: 1.5\u0026ndash;2.2) [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e]. The systematic literature review by Dantas et al. [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e] cited above identified the following patient characteristics as most associated with no-shows: adult of younger age, a lower socioeconomic status, distance between the patient\u0026rsquo;s home and the clinic, and lack of private insurance. Moreover, key appointment-related factors leading to patient non-attendance included the time between the appointment was scheduled and the actual appointment date, as well as a previous history of no-show behaviour [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eMissing scheduled medical appointments can have a negative impact on a patient\u0026rsquo;s health status. No-shows have been linked to an increased risk of morbidity and mortality as well as gaps in patient care. Among patients with HIV from an outpatient infectious disease clinic, rates of hospital admissions and mortality were significantly higher for no-show patients compared to those who attended their appointments (27.2% vs 8.9%, p\u0026thinsp;\u0026lt;\u0026thinsp;0.001; and 5.8% vs 0.7%, p\u0026thinsp;\u0026lt;\u0026thinsp;0.001, respectively) [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. The researchers concluded that missed scheduled appointments can serve as an independent predictor of hospital admission and mortality [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. In a study conducted among patients with congenital heart disease, Goossens et al. [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e] found that missed appointments can be considered a predicting factor for care gaps, as patients who missed appointments in the preceding 2 years had 19 times greater probability of care gaps in comparison to patients without missed appointments; however, no-shows were not associated with increased mortality in this particular study [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]. In addition to gaps in the care process of patients, patient non-attendance has also been linked to increased waiting time for medical appointments and a higher rate of admissions to emergency departments [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eIn addition to having a negative impact on patients\u0026rsquo; health, missed appointments also have negative consequences for healthcare systems, as they disrupt the normal functioning and efficiency of healthcare institutions, increase the use of healthcare resources and can lead to lost revenue [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]. An outpatient endoscopy unit performing 24 procedures per day evaluated the cost related to no-shows [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. Three main cost categories were used for colonoscopy procedures: operating costs (capital investment, equipment costs and staffing costs), overtime costs and patient waiting time costs. Considering an estimated 18% rate of no-shows using historical data, the net loss due to no-shows was 16.4% or \u003cspan\u003e$\u003c/span\u003e725.42 per day [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. In an outpatient vascular unit located in the United States with 12% rate of no-shows and an average of 7.6 missed visits per week, the gross revenue loss due to no-shows was estimated at \u003cspan\u003e$\u003c/span\u003e89,107 annually [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. Considering a nationwide scale, the annual cost of missed outpatient hospital appointments has been estimated at \u0026pound;1\u0026nbsp;billion for the NHS England and \u003cspan\u003e$\u003c/span\u003e150\u0026nbsp;billion for the US healthcare system [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]. Given the high financial impact associated with missed appointments, measures to prevent no-shows can translate into cost savings for healthcare systems and institutions. For example, the development of an online, patient-led booking system at Guy\u0026rsquo;s and St Thomas\u0026rsquo; NHS Foundation Trust in the UK reduced missed appointments by 11% in a 6-month period and led to savings of about \u0026pound;2.6\u0026nbsp;million [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eSeveral intervention strategies have been investigated with the aim of decreasing the rate of patient no-shows and the associated revenue loss. Some of these interventions include overbooking appointments, appointment reminders, e-mail communication, or the use of various appointment scheduling systems. Overbooking is a relatively inexpensive and easy method to implement [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. However, a disadvantage of this method is the level of staff dissatisfaction associated with overtime when more patients than initially expected show up for their appointments. Also, while overbooking is helpful in reducing healthcare costs related to no-shows, it does not help patients who are left without proper care due to their non-attendance [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. Reminders sent to patients before their visits can be helpful; however, patients may need to be carefully selected for this type of intervention to be successful. As an example, reminders were not effective in a small vascular unit, in which patients who agreed, received an automated reminder phone call three to four days before their visit. Here, the rate of no-shows was significantly higher in patients who received a phone reminder versus those who did not (8.9% vs 5.9%; p\u0026thinsp;\u0026lt;\u0026thinsp;0.0001). To explain this, the authors acknowledged the fact that they had not compared the compliance of each study group, so the group who did not agree to receive reminders could inherently be a more compliant group who did not see any benefit from receiving reminders, whereas the group who agreed to receive automated reminder phone calls could be more inclined to not attend their appointments, regardless of an intervention being implemented [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. A systematic literature review of studies reporting non-attendance in the context of hospital admissions, showed that automated cell phone and short message service reminders were less effective in comparison to manual phone calls (i.e., calls made directly by the staff) [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. The pooled estimates based on identified studies showed a reduction in the non-attendance rates of 39% and 29% versus baseline for phone calls and automated reminders, respectively [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. In the past, e-mails generated through a triage-based system have been found to be ineffective in reducing no-shows at two university-affiliated primary care centres in the United States [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e]. Overall, appointment reminders, whether e-mail or phone-based, can be ineffective if they are not tailored to the individual service/institution and its patient population [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e]. Regarding the various appointment scheduling systems, the use of an online-scheduling system versus traditionally-scheduled appointments (i.e., via reception staff) was not more effective in reducing waiting time or missed appointments, except for new patients [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eGiven the ongoing challenge of patient non-attendance, several predictive models have been developed to try to anticipate which patients are at high risk of non-attendance, with the aim of designing interventions which can reduce the number of missed appointments and their negative consequences for patients and healthcare systems [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e]. Among these, the use of artificial intelligence (AI) and machine learning (ML) techniques can further reduce the uncertainty of patient non-attendance because they can better capture and analyse variability among patient populations [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eAt our institution, missed outpatient appointments are an ongoing concern despite attempts to mitigate their negative impact on the use of resources and personnel time while ensuring patients\u0026rsquo; access to medical services. The aim of this study was to explore the use of an ML model to estimate the risk of future no-show patients, and how this can help support targeted, phone call-based interventions for patients who are at high risk of not showing up for their outpatient appointments. While a personal phone-based reminder has been shown to be effective, its main downside is the amount of resources required to contact every single patient. By using an ML model we focus our resources on the population with the highest risk of non-attendance, such that most patients who will be no-show have been contacted while significantly reducing the number of patients that need to be called. Additionally, the number of patients that is contacted can be easily adjusted and therefore fit to the available capacity, resources or needs of the organisation. We hypothesised that directing an intervention to target the population at high risk of no-show may reduce the rate of non-attendance within reasonable costs.\u003c/p\u003e"},{"header":"2 Material and methods","content":"\u003ch2\u003e2.1 Definition no-show\u003c/h2\u003e\n\u003cp\u003eIn order to consistently measure and analyse the data it was important to carefully define what we consider to be a \u0026lsquo;no-show\u0026rsquo;. Since the negative consequences of a no-show occur when a timeslot is reserved but unused, we defined an appointment as a no-show when this occurred. Therefore, for the purpose of the reminder service, we considered an appointment to be a \u0026lsquo;no-show\u0026rsquo; when either:\u003c/p\u003e\n\u003cp\u003e1.\u0026nbsp; \u0026nbsp; \u0026nbsp;The appointment was fulfilled and registered by the staff as a \u0026lsquo;non-attendance\u0026rsquo; in the EHR or\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e2a. The appointment was cancelled or rescheduled less than 24 hours before the scheduled time and\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e2b. The registered cause of cancellation or rescheduling was related to the patient.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eCondition 2a was included because, just like condition 1, this also results in an unused but reserved timeslot. According to the clinical staff, if an appointment is cancelled or rescheduled shortly before the appointed time, it is typically very difficult for the staff to fill the empty timeslot. Therefore, condition 2a also results in the same negative consequences, for both patient and hospital, as condition 1. Condition 2b was included specifically for the reminder service targeted at patients at risk of no-show, since some appointments may be registered as a no-show when in fact the cause was unrelated to the patient (e.g., absence of the healthcare professional). A reminder service for the patient is ineffective on such cases. Note that our definition of no-show is in line with the definition of \u0026lsquo;failed attendance\u0026rsquo; of the NHS [1], although we expanded on this definition slightly to include cases where the clinic is notified of the patients non-attendance shortly before the appointment.\u003c/p\u003e\n\u003ch2\u003e2.2 Design reminder service\u003c/h2\u003e\n\u003cp\u003eNot every single outpatient appointment was included when planning the reminder service. Appointments were only taken into account for the reminder service if the appointment was:\u003c/p\u003e\n\u003cp\u003e1.\u0026nbsp; \u0026nbsp; \u0026nbsp;Planned to take place with the patient physically present at the hospital\u003c/p\u003e\n\u003cp\u003e2.\u0026nbsp; \u0026nbsp; \u0026nbsp;Planned at least 7 days ahead\u003c/p\u003e\n\u003cp\u003e3.\u0026nbsp; \u0026nbsp; \u0026nbsp;Of such a type or (medical) content to be excluded on request of the outpatient clinic.\u003c/p\u003e\n\u003cp\u003eAmong these three criteria, condition 1 was included because the negative consequences of a no-show for phone or virtual appointments (i.e., e-mail, video) are minimal in contrast to face-to-face appointments. Condition 2 was included because it was deemed unnecessary to remind patients of appointments that they had made within the same week. Condition 3 was included so outpatient clinics could exclude certain appointment types, e.g., appointments for which every patient is called beforehand for medical reasons.\u003c/p\u003e\n\u003cp\u003eEvery working day, the list of patients to be called was constructed as follows. The ML model was used to generate a risk score for the entire outpatient population eligible for the reminder service as defined by the criteria outlined above. For every single outpatient clinic included, 35% of patients with the highest risk score of being a no-show were classified as the \u0026ldquo;high-risk\u0026rdquo; group, whereas 65% of patients were considered \u0026ldquo;lower risk\u0026rdquo; (\u0026ldquo;low-risk group). Patients in the high-risk group were then randomly allocated to either the intervention group or the control group. Patients in the intervention group received a reminder phone call three working days before their appointment. Phone calls were made by either a call centre or students from our institution (50% each, randomly selected from the intervention group). Patients in the control group had no intervention (i.e., no reminder phone call). The patient flow during the study is illustrated in\u0026nbsp;\u003cstrong\u003eFig. 1\u003c/strong\u003e.\u003c/p\u003e\n\u003cp\u003eEvery patient was called after working hours (between 17:00 and 20:00) in order to increase to chance of reaching the patient. Since personal information would be discussed during the phone call, the patients were first asked to identify themselves (name and date of birth) before continuing with the call. If a patient could not be reached with a first attempt, a second attempt was done later that evening. All results were logged and saved in a secured database for later analysis. The staff who performed the phone calls only had access to the list of patients to be called for that day, in order to minimise privacy risks.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFig.\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003e1\u003c/strong\u003e Schematic representation of the patient flow during the study. Red figures represent a no-show patient, a green figure represents a show patient.\u0026nbsp;\u003c/p\u003e\n\u003ch2\u003e2.3 AI model and strategy for predicting no-shows\u003c/h2\u003e\n\u003cp\u003eThe AI-based model was developed to generate a risk score of non-attendance for every patient eligible for the reminder service. We used Structured Query Language queries to gather data from Electronic Health Records (EHR), Python to analyse the data and train the model, Azure DevOps to schedule the daily data preparations and predictions, and a web application (developed with C#) which showed the details of the patients allocated to receive the intervention each day.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eWe used a gradient boosted decision-tree model, which is based on the extreme gradient boosting (XGBoost) algorithm, for applied ML for structured or tabular data\u0026nbsp;[18]. The XGBoost ML library produces scalable and distributed decision trees and performs regression analysis as well as classification and ranking of variables using parallel tree boosting\u0026nbsp;[19, 20]. Our model was trained on data from appointments that would have been eligible for the reminder service for outpatients treated at the participating clinics. Data included patient demographic and clinical characteristics as well as reasons for missing an appointment, which are routinely recorded in our database by technologists as part of the quality assurance system used at our institution.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eWe tested selected variables and how they increased or decreased the risk of no-shows. Only variables that considerably improved model performance were included. In addition, all the predictive variables considered were stable over time to avoid issues with data drift. The list of variables used to determine the risk of non-attendance is presented in Table 1.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003e1\u003c/strong\u003e Predictive variables for no-shows by patient and appointment characteristics or history\u003c/p\u003e\n\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd width=\"33.333333333333336%\" valign=\"top\"\u003e\n \u003cp\u003ePatient characteristics\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"33.333333333333336%\" valign=\"top\"\u003e\n \u003cp\u003eAppointment characteristics\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"33.333333333333336%\" valign=\"top\"\u003e\n \u003cp\u003eAppointment history\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"33.333333333333336%\" valign=\"top\"\u003e\n \u003cp\u003e\u0026middot;\u0026nbsp; \u0026nbsp; \u0026nbsp;\u0026nbsp;Distance (\u0026lsquo;as the crow flies\u0026rsquo;) from home to the medical institution in km\u003c/p\u003e\n \u003cp\u003e\u0026middot;\u0026nbsp; \u0026nbsp; \u0026nbsp;\u0026nbsp;Age\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"33.333333333333336%\" valign=\"top\"\u003e\n \u003cp\u003e\u0026middot;\u0026nbsp; \u0026nbsp; \u0026nbsp;\u0026nbsp;Time between scheduling the appointment and the actual appointment\u003c/p\u003e\n \u003cp\u003e\u0026middot;\u0026nbsp; \u0026nbsp; \u0026nbsp;\u0026nbsp;Multiple appointments that day\u003c/p\u003e\n \u003cp\u003e\u0026middot;\u0026nbsp; \u0026nbsp; \u0026nbsp;\u0026nbsp;Number of days since previous appointment\u003c/p\u003e\n \u003cp\u003e\u0026middot;\u0026nbsp; \u0026nbsp; \u0026nbsp;\u0026nbsp;Duration of the appointment\u003c/p\u003e\n \u003cp\u003e\u0026middot;\u0026nbsp; \u0026nbsp; \u0026nbsp;\u0026nbsp;Type of consultation\u003c/p\u003e\n \u003cp\u003e\u0026middot;\u0026nbsp; \u0026nbsp; \u0026nbsp;\u0026nbsp;Appointment previously rescheduled\u003c/p\u003e\n \u003cp\u003e\u0026middot;\u0026nbsp; \u0026nbsp; \u0026nbsp;\u0026nbsp;Referral\u003c/p\u003e\n \u003cp\u003e\u0026middot;\u0026nbsp; \u0026nbsp; \u0026nbsp;\u0026nbsp;Appointment rescheduled by a physician\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"33.333333333333336%\" valign=\"top\"\u003e\n \u003cp\u003e\u0026middot;\u0026nbsp; \u0026nbsp; \u0026nbsp;\u0026nbsp;Number of previous no-shows\u003c/p\u003e\n \u003cp\u003e\u0026middot;\u0026nbsp; \u0026nbsp; \u0026nbsp;\u0026nbsp;Patient punctuality: average arrival time before appointment (late, just in time, on time, early)\u003c/p\u003e\n \u003cp\u003e\u0026middot;\u0026nbsp; \u0026nbsp; \u0026nbsp;\u0026nbsp;Number of previous appointments\u003c/p\u003e\n \u003cp\u003e\u0026middot;\u0026nbsp; \u0026nbsp; \u0026nbsp;\u0026nbsp;Percentage of no-shows\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003ch2\u003e\u0026nbsp;\u003c/h2\u003e\n\u003ch2\u003e2.3 Pilot study and timeframe for main study\u0026nbsp;\u003c/h2\u003e\n\u003cp\u003e\u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp;\u0026nbsp;To assess the feasibility of using our AI model for predicting no-shows, the study was preceded by a pilot study. The pilot study was performed in August 2021 at the dermatology outpatient clinic. The dermatology clinic was chosen for the pilot study since this clinic has a relatively high percentage of no-shows in our institution and staff members from this clinic were willing and motivated to help with implementing and testing the reminder service. An ML model was trained on previous appointments at the dermatology outpatient clinic. Several different types of ML models were tested (Random Forest, Logistic Regression and XGBoost) and the XGBoost model showed the highest predictive power. Patients predicted by the AI model to be at high risk for no-show received a reminder call before their appointment. The pilot trial showed a 12.8% reduction in no-shows, which was considered sufficient to proceed to the next stage of the project, which involved further refinement of the AI model and testing across a wider variety of outpatient clinics at our institution.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThe main study, described in detail in this publication, was conducted from June 2022 to January 2023 (6-month period). The AI model was trained with retrospective data and kept unchanged for the duration of the study to ensure the reproducibility of the results. In this main study, the entire outpatient population from 24 outpatient clinics at our institution were included.\u003c/p\u003e\n\u003ch2\u003e2.4 Ethical considerations\u003c/h2\u003e\n\u003cp\u003eAll variables used by the model underwent review and successfully completed the Data Protection Impact Assessment (DPIA). Privacy risk was considered to be very limited. For example, the variables used are not medical in nature and cannot be traced back to an individual. In addition, the negative consequences for a patient identified as being at high risk of no-shows by the prediction model are minimal (i.e., the patient will receive one possibly unnecessary phone call), and the positive consequences for both patient and hospital were deemed to outweigh the negative ones. Lastly, every single feature that the model used was tested such that only features that probably improved the predictive power of the model were included (data minimalization).\u003c/p\u003e"},{"header":"3 Results","content":"\u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003e3.1 Reduction of no-shows and additional patient- and staff-related benefits\u003c/h2\u003e \u003cp\u003eOver the 6-month period evaluated in the main study, a total of 11,778 of high-risk patients were allocated to the intervention group and 15,399 to the control group. The control group is larger than the intervention group because patients from the intervention group that were not attempted to be reached (due to, for example, staff shortage), were then allocated to the control group. In the intervention group, 70% of patients designated to receive a phone call were able to be reached.\u003c/p\u003e \u003cp\u003eOverall, the study showed a 26.2% decline in no-shows for the intervention group. Different levels of improvement were observed across the 24 outpatient clinics: the reduction in no-shows ranged from 5\u0026ndash;69%, depending on the outpatient clinic profile and size. However, all outpatient clinics showed a reduction in no-shows following the intervention, even those historically linked to a lower proportion of patient non-attendance.\u003c/p\u003e \u003cp\u003eWe did not observe an increase in rescheduled appointments due to the implementation of reminder calls. Only 4.5% of the patients who were reached through a reminder phone call wanted to reschedule their appointments. If this was the case, the outpatient clinic staff received an email notification that the patient wanted to reschedule their appointment so they could be reached the following working day. In addition, when appointments were rescheduled due to the reminder phone call, new appointments for other patients could be often scheduled in the cancelled slots because there was sufficient time.\u003c/p\u003e \u003cp\u003eReminder calls were also useful to update contact details for many patients as, during the study period, 110 e-mail addresses and 40 phone numbers were able to be updated in patients\u0026rsquo; medical records.\u003c/p\u003e \u003cp\u003eIn addition, patient feedback around the intervention was overall positive, as patients frequently appreciated being able to speak directly to a member of the staff. This allowed administrative errors to be found, for example a patient who had asked for their appointment to be rescheduled but was not (yet) correctly adjusted.\u003c/p\u003e \u003cp\u003eSome additional benefits were also observed in relation to our employees. Typically, staff will prepare ahead of time for any scheduled appointments, hence, through reducing patient no-shows, we can reduce the time that staff invests on the preparation of appointments which are then not attended by patients. In addition, reducing the number of no-shows is expected to decrease the administrative time necessary to reschedule these appointments which, in our institution, has been estimated to range between 5 and 20 minutes per no-show.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003e3.2 Estimation of no-shows avoided during a 1-year period\u003c/h2\u003e \u003cp\u003eThe measured 26.2% reduction of the number of no-shows in the intervention group can be extrapolated to an expected reduction in the total number of no-shows if all high-risk patients would have been called. Assuming that the control group would have had the same reduction in the number of no-shows if they had been called, we can extrapolate that the reminder service would lead to a reduction of 14.3% in the total number of no-shows occurring at the outpatient clinics participating in the reminder service. As there were 42,445 no-shows in our institution in 2021, this extrapolation of the data from the 6-month study period would result in 6,070 no-shows which could be prevented during a year. Consequently, after the implementation of the AI model and the introduction of targeted phone calls, it was estimated that our institution could reschedule up to 6,070 visits and provide additional care for up to 6,070 patients.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003e3.3 Estimated cost of the intervention during a 1-year period\u003c/h2\u003e \u003cp\u003eWe estimated the cost of the reminder phone calls during an entire year. Reminder phone calls were to be made every working day to 35% of the patients identified as \u0026ldquo;high-risk\u0026rdquo; of not showing up. The cost of the call was \u0026euro;1.50 per patient reached for those calls made from the call centre, and \u0026euro;0.72 per patient for calls made by students. In addition, a cost of \u0026euro;30,000 per year was also estimated, to account for information technology (IT) management and hardware acquisition for the AI model. Overall, the total annual cost was estimated at \u0026euro;262,276 for the call centre versus \u0026euro;114,454 for calls made by students.\u003c/p\u003e \u003c/div\u003e"},{"header":"4 Discussion","content":"\u003cp\u003e Our study showed that the use of an AI model to predict patients at high risk of non-attendance, followed by a targeted intervention via phone call reminders, allowed for a 14.3% decrease in patient no-shows across 24 outpatient clinics from our institution. Additional observed benefits from the intervention were the ability to see more patients through scheduling appointments in the prevented \u0026ldquo;no-show\u0026rdquo; slots, the possibility of updating and improving the quality of data on patients\u0026rsquo; medical records, an overall improved patient experience, reductions in the time invested by the staff to prepare for no-show appointments and a reduction in the administrative work needed to reschedule no-shows.\u003c/p\u003e \u003cp\u003ePredicting patient non-attendance to medical appointments is a complex process. Typically, researchers select different sets and numbers of variables to increase prediction accuracy. The grouping of risk factors facilitates their analysis, but differences across healthcare institutions and patient populations mean that predictive variables are not universal. Mieloszyk et al. [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e] reported a model for predicting no-shows in the radiology department of a US clinic. Data were grouped into three categories: patient-related, exam-related and scheduling-related. The final model included 16 variables, of which the type of examination and those related to an appointment were most informative in the prediction of no-shows. A model developed by Alaidah et al. [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e] indicated that the patient history of missed appointments had the highest predictive weight; however, factors such as a longer time between booking the appointment and the appointment itself also increased the risk of non-attendance. The number of variables inputted in models tends to increase with the complexity of the model. In a recent model published by Nelson et al. [\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e], which included 81 variables, previous non-attendance was the eighth most important predictive factor after time between referral and appointment, number of previous appointments for imaging studies, home latitude, distance from home to hospital, number of previous magnetic resonance imaging (MRI) appointments, home longitude and total care cost. Dashtban et al. [\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e] considered eight variable groups for prediction of non-attendance, including demographic data, patient history, appointment characteristics, time variables, patient appointment history, socioeconomic data of the patient, weather conditions and admission history. A study by Dunstan et al. [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e] from Chile concluded that factors pertaining to a socioeconomically disadvantaged background were the most relevant to predict no-shows. The predictive value of variables linked to low socioeconomic status have also been observed in other studies, although there seems to be a country-specific component [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e, \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e], which further strengthen the need for models highly adjusted to the local patient population. We grouped the predictive variables into three categories, including patient characteristics, appointment characteristics and patient appointment history, which is similar to the approach followed by other studies. In addition, the predictive features selected for our model were based on what variables have been shown to be effective by other studies, as well as features that were relevant and possible to attain from our own institution, to tailor the model as much as possible to the outpatient population from our geographical area. We specifically selected features that would be applicable to a wide variety of outpatient clinics and which are expected to remain stable and well-registered over time. Every feature included in the final model was tested whether the feature improved the predictive performance of the model. Features that added no or negligible predictive power were excluded from the model.\u003c/p\u003e \u003cp\u003eSeveral ML predictive models have been investigated in healthcare, showing different levels of accuracy and efficacy. We tested different types of predictive models and found that XGBoost resulted in the highest predictive power for no-shows. This result was also found by the following studies. Yang et al. [\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e] tested different models, including logistic regression, random forest, support vector machine and XGBoost, to predict early neurological deterioration in patients after ischemic stroke treated with mechanical thrombectomy. XGBoost had the highest prediction power [area under the curve (AUC)\u0026thinsp;=\u0026thinsp;0.826; 95% CI: 0.781‒0.871] in comparison to logistic regression (AUC\u0026thinsp;=\u0026thinsp;0.644), random forest (AUC\u0026thinsp;=\u0026thinsp;0.819) and support vector machine (AUC\u0026thinsp;=\u0026thinsp;0.643) [\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e]. Lee et al. [\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e] analysed several models to compare their accuracy and complexity of deployment for predicting no-shows. The variables with the strongest prediction power were the number of days since the last visit, the last appointment status and the appointment waiting time. The XGBoost model gave the highest prediction power (AUC\u0026thinsp;=\u0026thinsp;0.832) and highest precision (0.785), in comparison to other models such as decision trees (AUC\u0026thinsp;=\u0026thinsp;0.760; precision\u0026thinsp;=\u0026thinsp;0.718), logistic regression (AUC\u0026thinsp;=\u0026thinsp;0.776; precision\u0026thinsp;=\u0026thinsp;0.718), random forest (AUC\u0026thinsp;=\u0026thinsp;0.805; precision\u0026thinsp;=\u0026thinsp;0.729), elastic net (AUC\u0026thinsp;=\u0026thinsp;0.808; precision\u0026thinsp;=\u0026thinsp;0.734) and gradient boosting trees (AUC\u0026thinsp;=\u0026thinsp;0.819; precision\u0026thinsp;=\u0026thinsp;0.733) [\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e]. Chen et al. [\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e] reported that the prediction power exerted by the XGBoost model for predicting no-shows for follow-up visits in an academic paediatric ophthalmology clinic was very high and reached an AUC of 0.90 and precision of 0.74. These results were better than those achieved when using other algorithms to train the model. Random forest was the second-best model (AUC of 0.88 and precision of 0.69). The remaining models investigated were the support vector regression (AUC\u0026thinsp;=\u0026thinsp;0.81 and precision\u0026thinsp;=\u0026thinsp;0.50) and LASSO regression (AUC\u0026thinsp;=\u0026thinsp;0.74 and precision\u0026thinsp;=\u0026thinsp;0.27). Chen et al. [\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e] also reported that the predictive capabilities of the model were much lower for new patient appointments in comparison to follow-up visits. Of note, the final predictive power and precision of a model have also been shown to depend on the type of medical institution and patient characteristics [\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e]. These two factors were also considered when training our AI model.\u003c/p\u003e \u003cp\u003eNon-attendance rates remain high despite the implementation of various interventions to reduce it. For the NHS England, the percentage of no-shows dropped from 8.3% in 2007‒2008 to 6.7% in 2017‒2018 [\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e], even after the introduction of a wide range of interventions, highlighting the need for additional, more refined solutions based on advanced technologies, such as AI. One of the first AI models to predict no-shows was developed by Dravenstott et al. [\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e]; this model used artificial neural networks to predict the no-show risk separately for primary care and endocrinology clinics. To teach the model, historical data on 3\u0026nbsp;million visits over the previous two years were included, which covered clinic, provider and patient-specific variables. The predictive performance of the model was below the attendance rate with an AUC of 0.78 and 0.81 for the primary care and endocrinology clinics, respectively [\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e]. Another model, which also used deep neural networks, was proposed by Dashtban and Li [\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e, \u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e]; this model was based on sparse stacked denoising autoencoders (SDAEs). The model was trained using in-hospital data collected from an acute care NHS hospital in the UK. Approximately 1.6\u0026nbsp;million records were included. The model used SDAEs with logistic regression and softmax layer to produce an AUC of 0.70, which was claimed sufficient to be used in real practice [\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e, \u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e]. Lee et al. [\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e] tested several ML models and finally opted for the XGBoost model, which provided the highest AUC (0.832) and highest precision (0.785). The proposed variables for the model by Lee et al. were sourced from the data warehouse, extracted from the literature and advised by domain experts. Twenty-two variables were finally engineered for modelling and linked to more than a million outpatient appointment records [\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e]. Nelson et al. [\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e] reported results from a high-dimensional gradient boosting machine-based model to predict no-shows, which was tested in 22,318 consecutive scheduled MRI appointments at two hospitals from the University College London. Their high-dimensional gradient boosting machine-based model showed an AUC of 0.852 and an average precision of 0.511. Optimal predictive performance of the model was observed when using 81 variables, which represents 2 to 4 times more variables in comparison to other models from the literature. The added value of the study by Nelson et al. [\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e] is the inclusion of potential cost benefit calculations that had not been reported in other studies. The authors made the following assumptions regarding costs of services: cost of model infrastructure and development was approximately \u0026pound;20,000, cost of an MRI study was \u0026pound;150 and cost of reminder calls was \u0026pound;6 per patient. The efficacy of the intervention (i.e., reminder calls) was estimated at 33%, meaning there was a 33% reduction of no-shows following the use of the predictive model plus the intervention. The net operating benefit of using the model was \u0026pound;3.15 per appointment with a break-even point at around 6,350 scheduled MRIs. In an average NHS hospital trust performing about 20,000 MRI scans annually, it was estimated the break-even point would be reached after about 4.5 months (83 working days) [\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eIt is difficult to compare the effectiveness of our AI-based predictive model with that of other models. Several of the models reported so far have been implemented in radiology departments [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e, \u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e], while our model is intended to be used across different specialties. In addition, only the AI-based model by Nelson et al. [\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e] described earlier has provided an estimate of financial/cost benefits, although this was also provided in the context of radiology [\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e]. A high-level comparison of the model reported by Nelson et al. [\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e]. and our model reveals several differences, including model structure (high-dimensional gradient boosting machine-based models vs XGBoost), number of variables analysed (81 vs 13), effectiveness of the intervention (33% vs 14.3%), phone call cost (\u0026pound;6 vs \u0026euro;0.72‒\u0026euro;1.50) or IT investment (\u0026pound;20,000 vs \u0026euro;30,000). Due to substantial differences between studies, the financial results cannot be compared; however, both studies concluded that the high predictive performance offered by AI models provides operating and financial benefits.\u003c/p\u003e \u003cp\u003eOur model also showed that reducing the risk of no-shows brings benefits to several stakeholders. First, there were positive effects for patients. The total number of missed appointments prevented, which for our institution was estimated to be up to 6,070 appointments per year, could be either rescheduled or offered to other patients, potentially reducing waiting time for patients. Lee et al. [\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e] compared the rate of released appointments, defined as the percentage of cancelled or rescheduled appointment slots over the total number of scheduled appointments, among different groups of patients as stratified by an AI model. The highest percentage of released appointments was observed in the group at high risk of not showing up: 3.9% of appointments were released in the low-risk group, 14.6% in the medium-risk group, and 31.9% in the high-risk group (p\u0026thinsp;\u0026lt;\u0026thinsp;0.05). The results reported by Lee et al. [\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e] also support that interventions to reduce no-shows should be directed towards patients at high risk of missing their appointment. Moreover, for patients with low digital literacy, receiving a phone call could translate into more inclusive care, as they will be able to be reminded of their appointment through an analogue method which is more familiar to them, rather than through digital reminder strategies, such as phone text messages or e-mails [\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e]. In our study, the overall patient experience for those patients receiving the intervention appeared to improve, as patients often shared positive feedback when they had the opportunity to speak directly on the phone with a member of the call centre or student staff. In addition, if the appointment needed to be rescheduled, patients would also speak with a member of the outpatient clinic staff for rescheduling purposes. Lastly, the intervention also provided an opportunity to update personal contact details for some of the patients at high risk of no-shows, which could help facilitate communication with them in the future to continue to reduce their risk of being a no-show. We also observed benefits for our staff following the use of the AI-based model and targeted intervention. The possibility of seeing additional patients who covered the slots of prevented no-shows did not seem to be associated with additional workload for the staff. On the contrary, reducing no-shows may help improve efficiencies and minimise the time invested by healthcare professionals on preparing for appointments that then go onto being a no-show. On the other hand, some staff reported that a no-show gave them time to catch up with other tasks; this would not be possible if there are less no-shows. However, since no-show appointments need to be rescheduled, this apparent time gain to catch up on other tasks can be considered a short-term gain but a long-term loss (from the need to eventually reschedule the no-shows). Another benefit is the potential for improved distribution of workload and operational efficiency: as there are fewer missed appointments that need to be rescheduled, it is expected that there will be less risk of waiting times increasing and appointment dairies getting overbooked. In addition, the reduction in no-shows was perceived by the staff to be associated with less administrative work through avoiding the extra time needed from reception workers to reschedule appointments. Similar effects were shown by other researchers who reported that, by reducing no-shows, technologists and physicians could devote more time to care for patients, rather than investing it on administrative tasks [\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e, \u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e]. Lastly, the potential for increasing revenue through the reimbursement of more appointments who can be seen in place of the no-shows is expected to have a positive financial impact on the overall institution.\u003c/p\u003e \u003cdiv id=\"Sec15\" class=\"Section2\"\u003e \u003ch2\u003e4.1 Strengths, limitations and future research\u003c/h2\u003e \u003cp\u003eA key strength from our study lies in its ability to show the real-world benefits of using an AI-based model in the outpatient setting from a large medical institution. Many previous studies, although based on real-world data, have only described the mathematical approach to predicting no-shows based on patient characteristics that place patients at high risk of missing medical appointments [\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e, \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e, \u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e, \u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e]. An additional strength of our study is that the reduction in no-shows was shown across a wide range of outpatient services from various specialties, including those with lower rates of historical no-shows, suggesting the use of predictive models based on AI to predict patients at high risk of non-attendance followed by a targeted intervention can be of benefit regardless of the specialty. Lastly, the reduction in no-shows was associated with observed benefits for patients, hospital staff, and the institution, including some benefits that may not be as obvious when the model and intervention are being designed, such as the improved patient experience or the enhanced patient care for patients with poor digital literacy. This study also has some limitations. Firstly, it is important to note that some no-shows are caused by reasons that a reminder service cannot prevent, e.g., a no-show due to unavailability of the healthcare professional or if the contact details of the patient are incorrect. Additionally, a reminder service is ineffective for appointments scheduled a few days in advance; however, these cases are a minority, although they require a different strategy to prevent non-attendance. Like any other AI-based model, if our model is to be used in a different healthcare institution, it would need to be adapted to the specific target population and relevant predictive variables, which may be different than those used in the current version of the model. Moreover, the prediction model would need to be trained with the new dataset and adapted in order to function with different data and software settings, since it is expected that different hospitals will have significantly different patient populations.\u003c/p\u003e \u003cp\u003eCurrently, the list of patients to be called is comprised of the 35% of patients with the highest risk of not showing up for every outpatient clinic separately. This setup was chosen in order to show the efficacy of the reminder service for each individual clinic. Now that this goal has been achieved, an improvement which can be easily implemented is to create a list of patients to be called including the 35% of patients at the highest risk of not showing up from the entire outpatient patient population. A more challenging improvement to the remainder service is to analyse the intervention effects. As the design includes both an intervention and a control group, it is possible to analyse for which patient profiles the reminder call was most effective. This would allow us to compose the list of patients to be called not solely based on which patients have the highest risk of not showing up, but also those for whom the intervention is expected to be most effective. The existing model is comprised of data currently available in the EHR, but future work from both engineers and healthcare institutions should focus on understanding additional variables which may have not been included yet in databases, but which may be important for predicting no-shows. Examples of such variables could be real commuting time, the most effective type of intervention based on the patient\u0026rsquo;s profile, or reminders sent to adolescents/young adults. Determination of additional variables based on real-world data may increase the effectiveness of predictive models in the future. Lastly, we advise to combine such a reminder service with other interventions to reduce the number of no-shows. There are many reasons why no-shows may occur, and not all of them may be prevented with a single intervention.\u003c/p\u003e \u003c/div\u003e"},{"header":"5 Conclusions","content":"\u003cp\u003eThe AI-based predictive model we report here is an effective tool for identifying patients at high risk of missing their outpatient appointments, who can then be targeted with a reminder intervention such as a phone call. Based on the results of the randomised control trial, we show that the no-show percentage of the targeted group is reduced to 26.2%, which is equivalent to a reduction of no-show appointments of 14.3% in the total patient population. The observed benefits after implementation of the model included improved patient experience, more inclusive care for patients with limited digital literacy, the ability to update patient contact information during calls, and improved overall operational efficiencies (including reductions in the administrative burden to reschedule appointments, as well as less time wasted by healthcare professionals and staff in preparing for no-show appointments). Further research and adaptations of the model to other healthcare settings and institutions as well as other patient populations are needed to confirm the high predictive performance of AI in the identification and management of no-show appointments.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003eCompeting Interests: The authors declare that they have no conflict of interest.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eHuman ethics: Ethics approval was not required for this research. This study has been conducted in accordance with the World Medical Association Declaration of Helsinki.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eFunding: This study was funded by the Dijkzigt department of Erasmus MC in Rotterdam, the Netherlands.\u003c/p\u003e\n\u003cp\u003eHuman ethics: Ethics approval was not applicable for this research because according to the Dutch law of medical research in humans (WMO), the participants of our study were not subjected to actions or prescribed rules of conduct. Therefore, the consent to participate was not required. This study has been conducted in accordance with the World Medical Association Declaration of Helsinki (2013).\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eAuthors\u0026rsquo; contributions: KA, JK and BW contributed to the study conception and design. Data collection and analysis were performed by JK and BW All authors contributed to data interpretation. KA participated in the drafting of the manuscript and all authors commented on draft versions of the manuscript. All authors read and approved the final manuscript. All authors agreed to be accountable for all aspects of the work and take responsibility for the integrity of the work as a whole.\u003c/p\u003e\n\u003cp\u003eData availability: The data sets used and/or analysed during the current study are available from the corresponding author upon reasonable request.\u003c/p\u003e\n\u003cp\u003eAcknowledgments: We would like to thank Małgorzata Biernikiewicz and Paula Martin of Valid Insight, Macclesfield, UK for providing medical writing support. We also acknowledge Joris Tukker, Steffen Greup, Enrico Timmerman, Jaap Auwema, Raymond de Vos \u0026nbsp;for their contributions to developing and implementing the model and the implementation software.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eNHS England and NHS Improvement (2020) NHS inpatient admission and outpatient referrals and attendances. 2020. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2020/05/QAR-Commentary-Q4-201920-aIu8F.pdf\u003c/span\u003e\u003cspan address=\"https://www.england.nhs.uk/statistics/wp-content/uploads/sites/2/2020/05/QAR-Commentary-Q4-201920-aIu8F.pdf\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. Accessed 3 May 2023\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGoossens E, van Deyk K, Budts W, Moons P (2022) Are missed appointments in an outpatient clinic for adults with congenital heart disease the harbinger for care gaps? Eur J Cardiovasc Nurs 21(2):127\u0026ndash;134. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1093/eurjcn/zvab012\u003c/span\u003e\u003cspan address=\"10.1093/eurjcn/zvab012\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNHS drive to reduce (2023) \u0026lsquo;no shows\u0026rsquo; to help tackle long waits for care. NHS England. 2023. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.england.nhs.uk/2023/01/nhs-drive-to-reduce-no-shows-to-help-tackle-long-waits-for-care/\u003c/span\u003e\u003cspan address=\"https://www.england.nhs.uk/2023/01/nhs-drive-to-reduce-no-shows-to-help-tackle-long-waits-for-care/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. Accessed 4 May 2023\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDantas LF, Fleck JL, Cyrino Oliveira FL, Hamacher S (2018) No-shows in appointment scheduling - a systematic literature review. Health Policy 122(4):412\u0026ndash;421. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.healthpol.2018.02.002\u003c/span\u003e\u003cspan address=\"10.1016/j.healthpol.2018.02.002\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eColubi MM, P\u0026eacute;rez-El\u0026iacute;as MJ, El\u0026iacute;as L, Pumares M, Muriel A, Zamora AM, Casado JL, Dronda F, L\u0026oacute;pez D, Moreno S (2012) Missing scheduled visits in the outpatient clinic as a marker of short-term admissions and death. HIV Clin Trials 13(5):289\u0026ndash;295. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1310/hct1305-289\u003c/span\u003e\u003cspan address=\"10.1310/hct1305-289\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRustam LB, Vander Weg M, Chrischilles E, Tanaka T (2023) Sociodemographic and Clinical Factors Associated with Nonattendance at the Hepatology Clinic. Dig Dis Sci. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/s10620-023-07951-z\u003c/span\u003e\u003cspan address=\"10.1007/s10620-023-07951-z\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAngarita GA, Reif S, Pirard S, Lee S, Sharon E, Gastfriend DR (2007) No-Show for Treatment in Substance Abuse Patients with Comorbid Symptomatology: Validity Results from a Controlled Trial of the ASAM Patient Placement Criteria. J Addict Med 1(2):79\u0026ndash;87. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1097/ADM.0b013e3180634c1d\u003c/span\u003e\u003cspan address=\"10.1097/ADM.0b013e3180634c1d\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMarbouh D, Khaleel I, Al Shanqiti K, Al Tamimi M, Simsekler MCE, Ellahham S, Alibazoglu D, Alibazoglu H (2020) Evaluating the Impact of Patient No-Shows on Service Quality. Risk Manag Healthc Policy 13:509\u0026ndash;517. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.2147/rmhp.S232114\u003c/span\u003e\u003cspan address=\"10.2147/rmhp.S232114\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBerg BP, Murr M, Chermak D, Woodall J, Pignone M, Sandler RS, Denton BT (2013) Estimating the cost of no-shows and evaluating the effects of mitigation strategies. Med Decis Making 33(8):976\u0026ndash;985. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1177/0272989x13478194\u003c/span\u003e\u003cspan address=\"10.1177/0272989x13478194\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSatiani B, Miller S, Patel D (2009) No-show rates in the vascular laboratory: analysis and possible solutions. J Vasc Interv Radiol 20(1):87\u0026ndash;91. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.jvir.2008.09.027\u003c/span\u003e\u003cspan address=\"10.1016/j.jvir.2008.09.027\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePhilpott-Morgan S, Thakrar DB, Symons J, Ray D, Ashrafian H, Darzi A (2021) Characterising the nationwide burden and predictors of unkept outpatient appointments in the National Health Service in England: A cohort study using a machine learning approach. PLoS Med 18(10):e1003783. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1371/journal.pmed.1003783\u003c/span\u003e\u003cspan address=\"10.1371/journal.pmed.1003783\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHasvold PE, Wootton R (2011) Use of telephone and SMS reminders to improve attendance at hospital appointments: a systematic review. J Telemed Telecare 17(7):358\u0026ndash;364. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1258/jtt.2011.110707\u003c/span\u003e\u003cspan address=\"10.1258/jtt.2011.110707\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKatz SJ, Moyer CA, Cox DT, Stern DT (2003) Effect of a triage-based E-mail system on clinic resource use and patient and physician satisfaction in primary care: a randomized controlled trial. J Gen Intern Med 18(9):736\u0026ndash;744. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1046/j.1525-1497.2003.20756.x\u003c/span\u003e\u003cspan address=\"10.1046/j.1525-1497.2003.20756.x\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMcLean SM, Booth A, Gee M, Salway S, Cobb M, Bhanbhro S, Nancarrow SA (2016) Appointment reminder systems are effective but not optimal: results of a systematic review and evidence synthesis employing realist principles. Patient Prefer Adherence 10:479\u0026ndash;499. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.2147/ppa.S93046\u003c/span\u003e\u003cspan address=\"10.2147/ppa.S93046\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKachooei A, Plusch K, Kasper A, D'Amore T, Beredjiklian P (2023) The effect of outpatient web-based online scheduling versus traditional staff scheduling systems on progression to surgery and no-show rates. J Res Med Sci 28:23. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.4103/jrms.jrms_738_22\u003c/span\u003e\u003cspan address=\"10.4103/jrms.jrms_738_22\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCarreras-Garc\u0026iacute;a D, Delgado-G\u0026oacute;mez D, Llorente-Fern\u0026aacute;ndez F, Arribas-Gil A (2020) Patient No-Show Prediction: A Systematic Literature Review. Entropy (Basel) 22(6). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.3390/e22060675\u003c/span\u003e\u003cspan address=\"10.3390/e22060675\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAlMuhaideb S, Alswailem O, Alsubaie N, Ferwana I, Alnajem A (2019) Prediction of hospital no-show appointments through artificial intelligence algorithms. Ann Saudi Med 39(6):373\u0026ndash;381. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.5144/0256-4947.2019.373\u003c/span\u003e\u003cspan address=\"10.5144/0256-4947.2019.373\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFriedman JH (2001) Greedy function approximation: A gradient boosting machine. Ann Stat 29:1189\u0026ndash;1232\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang K, Dick T, Balcan M-F (2020) Scalable and Provably Accurate Algorithms for Differentially Private Distributed Decision Tree Learning. ArXiv. abs/2012.10602\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003exgboost developers (2022) XGBoost Documentation. 2022. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://xgboost.readthedocs.io/en/stable/\u003c/span\u003e\u003cspan address=\"https://xgboost.readthedocs.io/en/stable/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. Accessed 25 May 2023\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMieloszyk RJ, Rosenbaum JI, Bhargava P, Hall CS (2017) Predictive modeling to identify scheduled radiology appointments resulting in non-attendance in a hospital setting. Annu Int Conf IEEE Eng Med Biol Soc 2017:2618\u0026ndash;2621. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1109/embc.2017.8037394\u003c/span\u003e\u003cspan address=\"10.1109/embc.2017.8037394\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAlaidah A, Alamoudi E, Shalabi D, AlQahtani M, Alnamshan H, Abubacker NF (eds) Mining and Predicting No-Show Medical Appointments: Using Hybrid Sampling Technique. Communication and Intelligent Systems; 2021 2021//; Singapore: Springer Singapore\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNelson A, Herron D, Rees G, Nachev P (2019) Predicting scheduled hospital attendance with artificial intelligence. NPJ Digit Med 2:26. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1038/s41746-019-0103-3\u003c/span\u003e\u003cspan address=\"10.1038/s41746-019-0103-3\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDashtban M, Li W (eds) (2019) Deep Learning for Predicting Non-attendance in Hospital Outpatient Appointments. Hawaii International Conference on System Sciences;\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDunstan J, Villena F, Hoyos JP, Riquelme V, Royer M, Ram\u0026iacute;rez H, Peypouquet J (2023) Predicting no-show appointments in a pediatric hospital in Chile using machine learning. Health Care Manag Sci. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/s10729-022-09626-z\u003c/span\u003e\u003cspan address=\"10.1007/s10729-022-09626-z\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTarabichi Y, Higginbotham J, Riley N, Kaelber DC, Watts B (2023) Reducing Disparities in No Show Rates Using Predictive Model-Driven Live Appointment Reminders for At-Risk Patients: a Randomized Controlled Quality Improvement Initiative. J Gen Intern Med. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/s11606-023-08209-0\u003c/span\u003e\u003cspan address=\"10.1007/s11606-023-08209-0\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYang T, Hu Y, Pan X, Lou S, Zou J, Deng Q, Zhang Q, Zhou J, Zhu J (2023) Interpretable Machine Learning Model Predicting Early Neurological Deterioration in Ischemic Stroke Patients Treated with Mechanical Thrombectomy: A Retrospective Study. Brain Sci 13(4). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.3390/brainsci13040557\u003c/span\u003e\u003cspan address=\"10.3390/brainsci13040557\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLee G, Wang S, Dipuro F, Hou J, Grover P, Low LL, Liu N, Loke CY (eds) (2017) Leveraging on Predictive Analytics to Manage Clinic No Show and Improve Accessibility of Care. IEEE International Conference on Data Science and Advanced Analytics (DSAA); 2017 19\u0026ndash;21 Oct. 2017\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen J, Goldstein IH, Lin WC, Chiang MF, Hribar MR (2020) Application of Machine Learning to Predict Patient No-Shows in an Academic Pediatric Ophthalmology Clinic. AMIA Annu Symp Proc. 2020:293\u0026ndash;302\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDravenstott R, Kirchner HL, Str\u0026ouml;mblad C, Boris D, Leader J, Devapriya P (2014) Applying predictive modeling to identify patients at risk to no-show. IIE Annual Conference and Expo 2014.2370-8\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHospital Outpatient Activity (2018) NHS Digital. 2018. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://files.digital.nhs.uk/97/20440A/hosp-epis-stat-outp-summ-rep-2017-18-rep.pdf\u003c/span\u003e\u003cspan address=\"https://files.digital.nhs.uk/97/20440A/hosp-epis-stat-outp-summ-rep-2017-18-rep.pdf\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDashtban M, Li W (2022) Predicting non-attendance in hospital outpatient appointments using deep learning approach. Health Syst (Basingstoke) 11(3):189\u0026ndash;210. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1080/20476965.2021.1924085\u003c/span\u003e\u003cspan address=\"10.1080/20476965.2021.1924085\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNeves BB, Mead G (2021) Digital Technology and Older People: Towards a Sociological Approach to Technology Adoption in Later Life. Sociology 55(5):888\u0026ndash;905. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1177/0038038520975587\u003c/span\u003e\u003cspan address=\"10.1177/0038038520975587\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBentayeb D, Lahrichi N, Rousseau L-M (2019) Patient scheduling based on a service-time prediction model: a data-driven study for a radiotherapy center. Health Care Manag Sci 22(4):768\u0026ndash;782. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/s10729-018-9459-1\u003c/span\u003e\u003cspan address=\"10.1007/s10729-018-9459-1\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBabayoff O, Shehory O, Geller S, Shitrit-Niselbaum C, Weiss-Meilik A, Sprecher E (2022) Improving Hospital Outpatient Clinics Appointment Schedules by Prediction Models. J Med Syst 47(1):5. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/s10916-022-01902-3\u003c/span\u003e\u003cspan address=\"10.1007/s10916-022-01902-3\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHarris SL, May JH, Vargas LG (2016) Predictive analytics model for healthcare planning and scheduling. Eur J Oper Res 253(1):121\u0026ndash;131. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.ejor.2016.02.017\u003c/span\u003e\u003cspan address=\"10.1016/j.ejor.2016.02.017\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"AI (Artificial intelligence), Appointment reminder, ML (Machine learning), Patient Non-attendance, Patient No-show, Predictive modelling","lastPublishedDoi":"10.21203/rs.3.rs-3743388/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-3743388/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003ePurpose\u003c/h2\u003e \u003cp\u003enon-attendance of patients for outpatient appointments, known as \"no-shows,\" poses a persistent challenge for healthcare facilities, with significant repercussions for both patients and healthcare systems. This study aimed to investigate whether targeting high-risk individuals with interventions could effectively reduce the rate of no-shows within reasonable resource allocation.\u003c/p\u003e\u003ch2\u003eMethods\u003c/h2\u003e \u003cp\u003ewe developed an artificial intelligence (AI) algorithm-based prediction model to estimate the likelihood of an appointment resulting in a no-show. Utilizing retrospective data from 24 outpatient clinics, a machine learning (ML) model was constructed and trained to identify patients at high risk of no-show. Subsequently, over a 6-month period, 35% of the highest-risk patients were randomly assigned to either the intervention group (receiving a reminder phone call three workdays before their appointment) or the control group (no reminder call).\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e \u003cp\u003efollowing the intervention, the intervention group experienced a notable 26.2% reduction in no-shows. This reduction translates to a 14.3% decrease in the overall number of no-shows, demonstrating the efficacy of the reminder service. Moreover, this intervention led to additional benefits, including the ability to schedule new patients on previously avoided no-show slots, enhanced patient experience, reduced staff preparation time for missed appointments, and a decrease in administrative burden associated with rescheduling no-shows.\u003c/p\u003e\u003ch2\u003eConclusions\u003c/h2\u003e \u003cp\u003eOur AI-powered model proved to be an effective tool for identifying high-risk patients prone to missing their outpatient appointments. This allowed for targeted interventions, such as reminder phone calls, to be implemented. The substantial reduction in no-show rates underscores the potential impact of this approach on optimizing healthcare resource allocation and improving patient attendance.\u003c/p\u003e","manuscriptTitle":"An artificial intelligence-based model to reduce the no-show rate in outpatient clinics of an academic hospital","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-01-03 17:09:20","doi":"10.21203/rs.3.rs-3743388/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"75527935-036d-445f-a0ec-cf04c6eda494","owner":[],"postedDate":"January 3rd, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2024-01-03T17:09:22+00:00","versionOfRecord":[],"versionCreatedAt":"2024-01-03 17:09:20","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-3743388","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-3743388","identity":"rs-3743388","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}