Predicting voided computerized physician order entry in oral and maxillofacial surgery inpatients: development and validation of machine learning model

Predicting voided computerized physician order entry in oral and maxillofacial surgery inpatients: development and validation of machine learning model | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Predicting voided computerized physician order entry in oral and maxillofacial surgery inpatients: development and validation of machine learning model John M. Nathan, Kevin Arce, Vitaly Herasevich This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-3868326/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 8 You are reading this latest preprint version Abstract Objective The aim of this study is to determine if supervised machine learning algorithms can accurately predict cancelled and self-intercepted computerized physician order entry in oral and maxillofacial surgery inpatients. Methods Data from Electronic Medical Record included patient demographics, comorbidities, procedures, vital signs, laboratory values, and medication orders were retrospectively collected. Predictor variables included patient demographics, comorbidities, procedures, vital signs, and laboratory values. Outcome of interest is if a medication order was voided or not. Data was cleaned and pro1cessed using Microsoft Excel and Python v3.12. Gradient Boosted Decision Trees, Random Forest, K-Nearest Neighbor, and Naïve Bayes were trained, validated, and tested for accuracy of the prediction of voided medication orders. Results 37,546 medication orders from 1,204 patient admissions over 5 years were used for this study included 3,892 (10.4%) medication orders that were voided. Gradient Boosted Decision Trees, Random Forest, K-Nearest Neighbor, and Naïve Bayes had an Area Under the Receiver Operating Curve of 0.802 802 with 95% CI [0.787, 0.825], 0.746 with 95% CI [0.722, 0.765], 0.685 with 95% CI [0.667, 0.699], and 0.505 with 95% CI [0.489, 0.539], respectively. Area Under the Precision Recall Curve was 0.684 with 95% CI [0.679, 0.702], 0.647 with 95% CI [0.638, 0.664], 0.429 with 95% CI [0.417, 0.434], and 0.551 with 95% CI [0.551, 0.552], respectively. Conclusion Gradient Boosted Decision Trees is the best model of the supervised machine learning algorithms with satisfactory performance in validation cohort for predicting voided Computerized Physician Order Entry in Oral and Maxillofacial Surgery inpatients. Figures Figure 1 Figure 2 Introduction Healthcare systems have been heavily investing in leveraging data generated and stored by Electronic Health Records (EHRs) in order to mitigate practice inefficiencies and decrease the frequency of costly medical errors [ 1 ]. Rule based Clinical Decision Support Systems (CDSS) that detect potential errors such as prescribing contraindicated medications and duplicate medication orders are commonly deployed within modern EHRs [ 2 ]. CDSS were intended to prevent adverse drug events and decrease medical errors, however, high frequency of warnings have led to alert fatigue, and providers ultimately dismiss alerts without fully acknowledging them [ 3 – 4 ]. Using artificial intelligence and machine learning (AI/ML) to create more precise and clinically relevant alerts could potentially circumvent alert fatigue while also catching preventable adverse drug errors (pADEs) [ 5 ]. King et al demonstrated promising results using various supervised machine learning algorithms including logistic regression (LR), decision trees (DT), random forest (RF), and gradient boosted decision trees (GBDT) to predict self-intercepted medication ordering errors [ 6 ]. In this study, the dataset included inpatient and outpatient medication orders from all hospital services at a tertiary hospital. Studies analyzing specific specialties have not been reported on. There is a rapidly increasing number of AI/ML studies being published within the specialty of oral and maxillofacial surgery (OMFS). Data scientists and clinicians are leveraging the power of AI/ML for improving cancer detection, automating virtual surgical planning, and predicting disease prognosis [ 7 – 9 ]. However, barriers such as lack of readily available data, computing power availability, and collaboration between data scientists and clinicians must be overcome to train, test, implement, and maintain AI/ML models that can improve OMFS patient care, outcomes, and clinician efficiency [ 10 ]. OMFS practices are not immune to medical errors that can lead to increased morbidity, mortality, and cost. AI//ML has the potential to supplement safeguards that are already in place to help prevent these errors. Anomaly detection methods have continuously evolved over many years. Many fields including E-commerce, cybersecurity, banking, and manufacturing analyze large amounts of data and must detect anomalous events that could represent fraud or unwanted results [ 11 – 15 ]. Current techniques in practice and under development for anomaly detection include the use of supervised and unsupervised machine learning. Different models have demonstrated promising results with accuracy, precision, and area under the receiver operating curve (AUROC) over 90% [ 15 ]. Many of the ML techniques used for anomaly detection in other industries have the potential to be applied to medication error detection in healthcare. In addition, applying data augmentation strategies used in other industries such as oversampling, undersampling, and generative adversarial networks (GAN) could be implemented when detecting infrequent events in medicine. OMFS data generation from EHRs and Computerized Physician Order Entry (CPOE) has had limited exploration for AI/ML model development. OMFS hospital services treat a diverse patient population ranging from pediatrics to geriatrics that require a wide range of procedures. This also includes a wide range of patient demographics, primary diagnosis, and patient comorbidities. Patient turnover is typically high with many patients hospitalized for less than a week at a time. In addition, depending on the institution type, OMFS providers range from the surgeon, trainees, advanced practice providers, and hospitalists. The natural variability within OMFS practices creates the potential for medication errors that can lead to increased patient morbidity and cost. Being able to precisely predict medication order errors in OMFS patients using AI/ML could decrease patient morbidity, cost, and alert fatigue. The purpose of this study was to determine if supervised machine learning algorithms can predict intercepted medication ordering errors within OMFS inpatients. Retrospective data including patient demographics, medications, comorbidities, surgeries, vital signs, lab values, and providers were collected over a 5-year span from a single OMFS hospital inpatient service. Supervised machine learning algorithms were then trained, validated, tested, and analyzed for diagnostic performance. The investigators hypothesize that supervised machine learning can accurately predict intercepted medication orders. Study aims: Characterize factors associated with intercepted medication orders Determine if supervised machine learning algorithms can accurately predict intercepted medication orders Compare performance of machine learning algorithms predicting intercepted medication orders Methods A TRIPOD checklist was used for development of the prediction model. See S1 for detailed list. Setting The Mayo Clinic Institutional Review Board approved this study (22-007742). Mayo Clinic Hospital in Rochester is a tertiary level I trauma center and teaching institution. CPOE and patient data entry was performed using EPIC. Mayo Clinic implemented EPIC EHR in May of 2018. Data was collected from the Mayo Clinic Unified Data Platform (UDP) using custom queries. Data Medication order history was retrospectively collected from all patients admitted to the OMFS service at the Mayo Clinic in Rochester, MN from 5/1/2018 to 12/31/2022. Data was cleaned, formatted, and analyzed using Microsoft Excel and Python version 3.12. For this study, only medication orders placed during an admission to service floor were used. Intra-operative and discharge medication orders were excluded. Medication orders entered by non-OMFS providers were not included in this study. At this institution, laboratory work is not indicated or routinely ordered for a significant number of the OMFS patients. For this study, missing laboratory values were imputed with the cohort median value. Variables Independent variables included patient age, gender, race, weight, height, comorbidities problem list, vital signs, lab values, type of medication ordered, medication frequency, medication dose, medication form, procedures underwent during admission, time of order placement, day of order placement, provider placing the order, and admitting attending. Dependent variable of interest is if a medication order was cancelled or not. In addition, medication orders discontinued within 2 hours of being ordered, without being completed, were considered a cancelled order for this study. Distribution of time to order discontinuation was analyzed and orders discontinued within 2 hours were determined to likely represent an error. In addition, previously published medication error rates were similar and thus the 2 hour mark was used as the cutoff for this study [ 16 ]. Medication order, form, route, units, frequency, and dose were all recorded for each admission. Active problem list comorbidities were collected from all admissions. Vital signs including heart rate, respiratory rate, blood pressure, temperature, and oxygen saturation were collected from each hospital admission. Lab values from admissions including potassium, blood glucose, sodium, creatinine, magnesium, phosphorous, hemoglobin, white blood cell count, and platelets were collected. Vital sign and lab data was winsorized and inspected for outliers likely caused by entry error. Outliers thought to be caused by entry error were removed. Provider placing the order was categorized into resident and advanced practice provider. Medication doses were z-scored using unique combination of medication, unit, and route. Procedures underwent during the admission were also collected. Procedures underwent during admission were categorized into dentoalveolar, orthognathic/craniofacial, pathology/reconstruction, temporomandibular joint surgery, infection related, and other surgeries. Problem list comorbidities were binned into ICD-9 categories of infection, neoplasm, autoimmune related, endocrine, psychiatric, neurological, ophthalmologic, ear, circulatory, gastrointestinal, dermatology, musculoskeletal, urinary, pregnancy related, and trauma related. The maximum and minimum value for vital signs and lab values during an admission were used for modeling. Time of order placement was sub-categorized into the hour of the day and day of the week. Statistical Analysis Descriptive statistics and univariable regression were performed where indicated to analyze the dataset and determine which independent variables were most closely associated with self-intercepted medication orders. Algorithms were validated using stratified k-fold cross validation to account for class imbalance. The dataset was divided into 5 folds. One of the folds was held out for testing while the other folds were used for training. Stratified k-fold cross validation splits the data into k subsets of equal size and maintains the overall class distribution for each fold. Synthetic Minority Oversampling Technique (SMOTE) was also used on the training data folds to address class imbalance. SMOTE synthetically creates data that closely mirrors the minority class using k-nearest neighbor to create a more balanced dataset. Scikit-Learn was used to train and test Gradient Boosted Decision Trees (GBDT), Random Forest (RF), k-Nearest Neighbors (kNN), and Naïve Bayes (NB). Hyperparameters that were associated with the greatest area under the receiver operating characteristic curve (AUROC) during training and validation were used for algorithm testing. Performance of machine learning algorithms including AUROC, Area under Precision-Recall Curve (AUPRC), precision, recall, F1-score, and accuracy was compared using the unseen k-fold dataset. Parameter grid search was used for to search for hyperparameters that resulted in the highest algorithm performance. For GBDT, best hyperparameters were learning_rate: 0.1, max_depth: 7, and n_estimators: 50. For RF, best hyperparameters were max_depth: 20, min_samples_leaf: 4, min_samples_leaf: 4, n_estimators: 100, and random_state: 42. For kNN, best hyperparameters were n_neighbors: 3, weights: distance, and algorithm: auto. Results After data processing and cleaning, 37,493 medication orders from 1,047 patients accounting for 1,204 admissions were used for analysis in this study. 3,892 (10.4%) medication orders were cancelled or voided within 2 hours of order placement without administration. 15.99% of medication orders placed on a weekend were cancelled compared to 9.97% of medication orders placed on a weekday. 11.87% of orders placed by residents were cancelled compared to 6.21% of orders placed by APPs. See Table 1 for medication order characteristics. Area under the receiver operating characteristic (AUROC) for GBDT was 0.802 with 95% CI [0.787, 0.825]. Area under the precision recall curve (AUPRC) was 0.6864 with 95% CI [0.679, 0.702] (Figure 1). GBDT had a weighted precision and recall of 0.93 and 0.93, respectively (Figure 2). F1-score for GBDT was 0.93. Top 5 most important features for GBDT with F score were hour_of_day (350), age (207), weight_kg (189), height_cm (160), and Z_Score_dose (143) (Figure 2). AUROC for RF was 0.746 with 95% CI [0.722, 0.765]. AUPRC was 0.647 with 95% CI [0.638, 0.664] (Figure 1). RF had a weighted precision and recall of 0.92 and 0.93, respectively (Figure 2). F1-score for RF was 0.92. AUROC for kNN was 0.685 with 95% CI [0.667, 0.699]. AUPRC was 0.429 with 95% CI [0.417, 0.434] (Figure 1). kNN had a weighted precision and recall of 0.87 and 0.82, respectively. F1-score for kNN was 0.84 (Figure 2). AUROC for NB was 0.505 with 95% CI [0.489, 0.539]. AUPRC was 0.551 with 95% CI [0.551, 0.552] (Figure 1). NB had a weighted precision and recall of 0.89 and 0.12, respectively (Figure 2). F1-score for NB was 0.04. Discussion Healthcare systems and providers are constantly attempting to minimize medication errors, however, various strategies to reduce and catch errors have had limited success and unintended consequences. We believe that this study is the first to report on predicting intercepted CPOE using data from a single hospital service. We retrospectively collected data generated from OMFS inpatients at a single institution over approximately 5 years. The data was then used to train, validate, and test GBDT, RF, KNN, and NB. GBDT (AUROC = 0.802) resulted in the best performance of the algorithms tested. While GBDT does not give feature coefficients like traditional regression models, feature importance can be viewed. In this study, the most important feature was the hour of day that the order was placed. This aligns with clinical factors such as increased provider fatigue and decreased trainee oversight at night. This study helps to characterize factors associated with medication orders used for OMFS inpatients at a single institution. It also demonstrates that supervised machine learning algorithms can accurately predict intercepted medication orders for OMFS inpatients. Relative algorithm performance for AUROC was similar to the results of the study by King et al when self-intercepted CPOE was predicted in a hospital wide dataset. In our study, GBDT and RF performed better at precision and recall compared to King et al. This is potentially due to using a single specialty dataset in this study. Even larger datasets from specific specialties have the potential for improved algorithm performance compared to non-specific hospital wide datasets. Using specialty specific data for algorithm training can allow for improved domain specific feature selection and decrease noise from irrelevant and redundant features [ 17 ]. Domain specific features and feature selection strategies have been applied towards consumer credit risk and cybersecurity models to improve model performance [ 18 – 19 ], Khandani et al used generalized classification and regression trees (CART) in order to recursively select independent features that improved model performance while also increasing model interpretability for consumer credit risk predictions. Datasets used for model training in cybersecurity anomaly detection are also commonly composed of high-dimensionality with many features [ 20 ]. Dimensionality reduction strategies used in the cybersecurity industry include univariate feature selection, correlated feature elimination, and gradient boosting [ 18 ]. Applying these various strategies to predicting intercepted CPOE will likely continue to improve model performance. Due to the nature of the data, a significant amount of class imbalance exists. Various techniques including random oversampling and synthetic oversampling have been described to account for class imbalance during model training for credit card fraud detection. In credit card transaction datasets, large class imbalance exists between genuine transactions compared to fraudulent transactions. Synthetic Minority Over-sampling Technique (SMOTE) was described by Chawla et al in order to overcome dataset imbalance [ 21 ]. Since then many credit card fraud detection models have augmented their datasets using SMOTE during model training in order to improve prediction performance on highly imbalanced datasets [ 22 – 23 ]. SMOTE increased algorithm performance for all algorithms tested during our study. Weaknesses of the study include dataset size, missing laboratory data, and using vital signs and lab values from the entire admission. At the time of data collection, the institution had implemented EPIC as its EHR and CPOE for approximately 5 years. Combining data from the previous CPOE system and EHR was not practical due to the voiding and cancellation process of the previous system. Handling of missing data is important for optimizing algorithm performance without causing overfitting or losing statistical power. Missing lab values in the setting of OMFS inpatients are a common occurrence due to the lack of clinical indication. For this study, missing laboratory data was addressed by imputing the median value for the labs. Removing medication orders associated with admissions without lab values would lead to a high percentage of the dataset being removed. The use of AI/ML to predict outcomes should rely on data that is collected before an outcome occurs. In this study, vitals and lab values from over the course of the entire admission were used. Future studies should use vitals and lab data collected from the initial admission or vitals collected prior to medication orders being placed such as during a procedure in the operating room. In addition, the use of intercepted medication orders as a surrogate for ordering errors is not perfect. However, previous studies have demonstrated that approximately 70% − 90% of self-intercepted orders are errors [ 24 – 25 ]. Conclusions Our results demonstrated that GBDT, a supervised machine learning algorithm, can use patient factors and clinical scenario to accurately predict discontinued CPOE. Future implementation of machine learning based CDSSs at level of EMR CPOE could decrease medication errors in real time as providers place orders while also alleviating the need for rule based alert systems. This could ultimately reduce hospital costs, alert fatigue, and patient complications. Further studies and validation are still needed prior to clinical implementation of AI/ML guided CDSS. Declarations Declaration of Consent The Mayo Clinic Institutional Review Board waived the requirement for HIPAA Authorization Waiver and patient consent. IRB# 22-007742 Ethics Declaration The research was conducted in accordance with the Declaration of Helsinki. Funding None References “Computer Physician Order Entry: Benefits, Costs, and Issues | Annals of Internal Medicine.” Accessed: Nov. 06, 2023. [Online]. Available: https://www.acpjournals.org/doi/full/ 10.7326/0003-4819-139-1-200307010-00010 “Applied Sciences | Free Full-Text | Applicability of Clinical Decision Support in Management among Patients Undergoing Cardiac Surgery in Intensive Care Unit: A Systematic Review.” Accessed: Nov. 06, 2023. [Online]. Available: https://www.mdpi.com/2076-3417/11/6/2880 J. S. Ancker et al. , “Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system,” BMC Med. Inform. Decis. Mak., vol. 17, no. 1, p. 36, Apr. 2017, doi: 10.1186/s12911-017-0430-8 . T. K. Nuckols et al. , “The effectiveness of computerized order entry at reducing preventable adverse drug events and medication errors in hospital settings: a systematic review and meta-analysis,” Syst. Rev., vol. 3, no. 1, p. 56, Jun. 2014, doi: 10.1186/2046-4053-3-56 . “Reducing drug prescription errors and adverse drug events by application of a probabilistic, machine-learning based clinical decision support system in an inpatient setting | Journal of the American Medical Informatics Association | Oxford Academic.” Accessed: Nov. 06, 2023. [Online]. Available: https://academic.oup.com/jamia/article/26/12/1560/5544737 “Predicting self-intercepted medication ordering errors using machine learning | PLOS ONE.” Accessed: Nov. 06, 2023. [Online]. Available: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0254358 D. G. Krishnan, “Artificial Intelligence in Oral and Maxillofacial Surgery Education,” Oral Maxillofac. Surg. Clin., vol. 34, no. 4, pp. 585–591, Nov. 2022, doi: 10.1016/j.coms.2022.03.006 . P. Bouletreau, M. Makaremi, B. Ibrahim, A. Louvrier, and N. Sigaux, “Artificial Intelligence: Applications in orthognathic surgery,” J. Stomatol. Oral Maxillofac. Surg. , vol. 120, no. 4, pp. 347–354, Sep. 2019, doi: 10.1016/j.jormas.2019.06.001 . “Artificial intelligence for oral and maxillo-facial surgery: A narrative review - ScienceDirect.” Accessed: Nov. 06, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S2468785522000271 J. Watson et al. , “Overcoming barriers to the adoption and implementation of predictive modeling and machine learning in clinical care: what can we learn from US academic medical centers?,” JAMIA Open , vol. 3, no. 2, pp. 167–172, Jul. 2020, doi: 10.1093/jamiaopen/ooz046 . S. Shirgave, C. Awati, R. More, and S. Patil, “A Review On Credit Card Fraud Detection Using Machine Learning,” Int. J. Sci. Technol. Res. , vol. 8, pp. 1217–1220, Oct. 2019. T. Wuest, D. Weimer, C. Irgens, and K.-D. Thoben, “Machine learning in manufacturing: advantages, challenges, and applications,” Prod. Manuf. Res., vol. 4, no. 1, pp. 23–45, Jan. 2016, doi: 10.1080/21693277.2016.1192517 . A. Handa, A. Sharma, and S. K. Shukla, “Machine learning in cybersecurity: A review,” WIREs Data Min. Knowl. Discov., vol. 9, no. 4, p. e1306, 2019, doi: 10.1002/widm.1306 . J. Nanduri, Y. Jia, A. Oka, J. Beaver, and Y.-W. Liu, “Microsoft Uses Machine Learning and Optimization to Reduce E-Commerce Fraud,” Inf. J. Appl. Anal. , vol. 50, no. 1, pp. 64–79, Jan. 2020, doi: 10.1287/inte.2019.1017 . “Machine Learning for Anomaly Detection: A Systematic Review | IEEE Journals & Magazine | IEEE Xplore.” Accessed: Jan. 03, 2024. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/9439459 L. Briceland and D. S. Stein, “Factors Related to Errors in Medication Prescribing”. L. Budach et al. , “The Effects of Data Quality on Machine Learning Performance,” arXiv.org. Accessed: Jan. 03, 2024. [Online]. Available: https://arxiv.org/abs/2207.14529v4 M. Ahsan, R. Gomes, M. M. Chowdhury, and K. E. Nygard, “Enhancing Machine Learning Prediction in Cybersecurity Using Dynamic Feature Selector,” J. Cybersecurity Priv. , vol. 1, no. 1, Art. no. 1, Mar. 2021, doi: 10.3390/jcp1010011 . A. E. Khandani, A. J. Kim, and A. W. Lo, “Consumer credit-risk models via machine-learning algorithms,” J. Bank. Finance , vol. 34, no. 11, pp. 2767–2787, Nov. 2010, doi: 10.1016/j.jbankfin.2010.06.001 . A. Alshaibi, M. Al-Ani, A. Al-Azzawi, A. Konev, and A. Shelupanov, “The Comparison of Cybersecurity Datasets,” Data , vol. 7, no. 2, Art. no. 2, Feb. 2022, doi: 10.3390/data7020022 . N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “SMOTE: Synthetic Minority Over-sampling Technique,” J. Artif. Intell. Res. , vol. 16, pp. 321–357, Jun. 2002, doi: 10.1613/jair.953 . A. Thennakoon, C. Bhagyani, S. Premadasa, S. Mihiranga, and N. Kuruwitaarachchi, Real-time Credit Card Fraud Detection Using Machine Learning. 2019. doi: 10.1109/CONFLUENCE.2019.8776942 . D. Varmedja, M. Karanovic, S. Sladojevic, M. Arsenovic, and A. Anderla, “Credit Card Fraud Detection - Machine Learning methods,” in 2019 18th International Symposium INFOTEH-JAHORINA (INFOTEH) , East Sarajevo, Bosnia and Herzegovina: IEEE, Mar. 2019, pp. 1–5. doi: 10.1109/INFOTEH.2019.8717766 . T. G. Kannampallil et al. , “Learning from errors: analysis of medication order voiding in CPOE systems,” J. Am. Med. Inform. Assoc. , vol. 24, no. 4, pp. 762–768, Jul. 2017, doi: 10.1093/jamia/ocw187 . J. Abraham et al. , “Reasons for computerised provider order entry (CPOE)-based inpatient medication ordering errors: an observational study of voided orders,” BMJ Qual. Saf., vol. 27, no. 4, pp. 299–307, Apr. 2018, doi: 10.1136/bmjqs-2017-006606 . Tables Variable Non-Voided Order Voided Order 33601 (89.6%) 3892 (10.4%) Age (mean, SD) 48.7 (21.4) 51.7 (20.8) Sex Female 19646 (58%) 2330 (60%) Male 13955 (42%) 1562 (40%) Race/Ethnicity Caucasian 30635 (91%) 3579 (92%) Asian 932 (3%) 61 (2%) Black 709 (2%) 96 (2%) Other 1325 (4%) 156 (4%) Day of Week Weekend 2139 (6%) 407 (10%) Weekday 31462 (94%) 3485 (90%) Shift Day 24811 (74%) 2470 (63%) Night 8790 (26%) 1422 (37%) Ordering Provider Resident 24351 (72%) 3280 (84%) APP 9250 (28%) 612 (16%) Medication Type Antimicrobial 4362 (12%) 435 (11%) Analgesic 9716 (29%) 1391 (36%) Circulatory 5592 (17%) 723 (19%) Respiratory 898 (3%) 29 (1%) GI/Endo 3374 (10%) 195 (5%) Other 9627 (29%) 1117 (28%) SD = standard deviation APP = advanced practice provider Table 1 Medication order cohort characteristics. GBDT RF Outcome Precision Recall F1-score Outcome Precision Recall F1-score 0 0.96 0.97 0.96 0 0.95 0.98 0.96 1 0.69 0.64 0.66 1 0.73 0.51 0.6 Macro Average 0.83 0.8 0.81 0.84 0.75 0.78 Weighted Average 0.93 0.93 0.93 0.92 0.93 0.92 kNN NB Outcome Precision Recall F1-score Outcome Precision Recall F1-score 0 0.94 0.85 0.89 0 0.98 0.01 0.03 1 0.29 0.52 0.37 1 0.1 1 0.19 Macro Average 0.61 0.69 0.63 0.54 0.51 0.11 Weighted Average 0.87 0.82 0.84 0.89 0.12 0.04 Table 2 GBDT, RF, kNN, and NB classification report. Additional Declarations Competing interest reported. Non-financial interests: Author K.A. is a co-editor of Oral and Maxillofacial Surgery Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 12 May, 2024 Reviews received at journal 06 Apr, 2024 Reviewers agreed at journal 06 Apr, 2024 Reviewers agreed at journal 21 Jan, 2024 Reviewers invited by journal 21 Jan, 2024 Editor assigned by journal 16 Jan, 2024 Submission checks completed at journal 16 Jan, 2024 First submitted to journal 15 Jan, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-3868326","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":267520833,"identity":"87f89900-de69-4b73-8c71-7b04c3886bbb","order_by":0,"name":"John M. Nathan","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA7ElEQVRIiWNgGAWjYBACgwMMbCCamUGCgfEBTBCvFosDzHAtzDCl+LXYQLUwALWwSRCn5fj5Yw9+7mBg55fuPVZ1o+awHAN78zYJfFrMziSzG/aeYWCWnHMu7XbOscPGDDzHyvBrOZDMJsHbBvTHjRyz2zlshxMbJHLM8GoxPv+YTfIvVEtxzj+gFvk3+LUY3khmk4bZwpzbBrKFh5CWx2bSsm0SzJIzcoylc/vSjdl40oot8GkxOJ/4TPJtm00yv0SO4eecb9Zy/OyHN97ApwUKJJKhjGYGNiKUg4EdlK4jVsMoGAWjYBSMIAAA8hpEitFUDToAAAAASUVORK5CYII=","orcid":"","institution":"Mayo Clinic","correspondingAuthor":true,"prefix":"","firstName":"John","middleName":"M.","lastName":"Nathan","suffix":""},{"id":267520834,"identity":"c1580ea9-6f03-419f-807e-bb8a967fb65a","order_by":1,"name":"Kevin Arce","email":"","orcid":"","institution":"Mayo Clinic","correspondingAuthor":false,"prefix":"","firstName":"Kevin","middleName":"","lastName":"Arce","suffix":""},{"id":267520835,"identity":"fb1ef8da-46d3-4ce6-93d2-798b3c816cb4","order_by":2,"name":"Vitaly Herasevich","email":"","orcid":"","institution":"Mayo Clinic","correspondingAuthor":false,"prefix":"","firstName":"Vitaly","middleName":"","lastName":"Herasevich","suffix":""}],"badges":[],"createdAt":"2024-01-16 01:44:10","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-3868326/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-3868326/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":49824156,"identity":"b49d6e24-91fc-4522-b977-931b73963324","added_by":"auto","created_at":"2024-01-18 15:33:38","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":55302,"visible":true,"origin":"","legend":"\u003cp\u003eReceiver Operating Characteristic Curve and Precision-Recall Curve for GBDT, RF, kNN, and NB.\u003c/p\u003e","description":"","filename":"F1.png","url":"https://assets-eu.researchsquare.com/files/rs-3868326/v1/99f64b9848d511d1547cc52a.png"},{"id":49824155,"identity":"a53a5ddd-f87b-4e5e-9037-ff47add77c10","added_by":"auto","created_at":"2024-01-18 15:33:38","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":32472,"visible":true,"origin":"","legend":"\u003cp\u003eGBDT top 5 features and relative importance.\u003c/p\u003e","description":"","filename":"F2.png","url":"https://assets-eu.researchsquare.com/files/rs-3868326/v1/3dcb34b081533fa315a2f485.png"},{"id":49824733,"identity":"328f51ee-7e37-41f6-8e82-d89655e65bb6","added_by":"auto","created_at":"2024-01-18 15:41:38","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":311720,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-3868326/v1/c75689e2-63dd-4873-aaf5-ab6e6cf9b2e3.pdf"}],"financialInterests":"Competing interest reported. Non-financial interests: Author K.A. is a co-editor of Oral and Maxillofacial Surgery","formattedTitle":"Predicting voided computerized physician order entry in oral and maxillofacial surgery inpatients: development and validation of machine learning model","fulltext":[{"header":"Introduction","content":"\u003cp\u003eHealthcare systems have been heavily investing in leveraging data generated and stored by Electronic Health Records (EHRs) in order to mitigate practice inefficiencies and decrease the frequency of costly medical errors [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]. Rule based Clinical Decision Support Systems (CDSS) that detect potential errors such as prescribing contraindicated medications and duplicate medication orders are commonly deployed within modern EHRs [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]. CDSS were intended to prevent adverse drug events and decrease medical errors, however, high frequency of warnings have led to alert fatigue, and providers ultimately dismiss alerts without fully acknowledging them [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. Using artificial intelligence and machine learning (AI/ML) to create more precise and clinically relevant alerts could potentially circumvent alert fatigue while also catching preventable adverse drug errors (pADEs) [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. King et al demonstrated promising results using various supervised machine learning algorithms including logistic regression (LR), decision trees (DT), random forest (RF), and gradient boosted decision trees (GBDT) to predict self-intercepted medication ordering errors [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e]. In this study, the dataset included inpatient and outpatient medication orders from all hospital services at a tertiary hospital. Studies analyzing specific specialties have not been reported on.\u003c/p\u003e \u003cp\u003eThere is a rapidly increasing number of AI/ML studies being published within the specialty of oral and maxillofacial surgery (OMFS). Data scientists and clinicians are leveraging the power of AI/ML for improving cancer detection, automating virtual surgical planning, and predicting disease prognosis [\u003cspan additionalcitationids=\"CR8\" citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. However, barriers such as lack of readily available data, computing power availability, and collaboration between data scientists and clinicians must be overcome to train, test, implement, and maintain AI/ML models that can improve OMFS patient care, outcomes, and clinician efficiency [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. OMFS practices are not immune to medical errors that can lead to increased morbidity, mortality, and cost. AI//ML has the potential to supplement safeguards that are already in place to help prevent these errors.\u003c/p\u003e \u003cp\u003eAnomaly detection methods have continuously evolved over many years. Many fields including E-commerce, cybersecurity, banking, and manufacturing analyze large amounts of data and must detect anomalous events that could represent fraud or unwanted results [\u003cspan additionalcitationids=\"CR12 CR13 CR14\" citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]. Current techniques in practice and under development for anomaly detection include the use of supervised and unsupervised machine learning. Different models have demonstrated promising results with accuracy, precision, and area under the receiver operating curve (AUROC) over 90% [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]. Many of the ML techniques used for anomaly detection in other industries have the potential to be applied to medication error detection in healthcare. In addition, applying data augmentation strategies used in other industries such as oversampling, undersampling, and generative adversarial networks (GAN) could be implemented when detecting infrequent events in medicine. OMFS data generation from EHRs and Computerized Physician Order Entry (CPOE) has had limited exploration for AI/ML model development. OMFS hospital services treat a diverse patient population ranging from pediatrics to geriatrics that require a wide range of procedures. This also includes a wide range of patient demographics, primary diagnosis, and patient comorbidities. Patient turnover is typically high with many patients hospitalized for less than a week at a time. In addition, depending on the institution type, OMFS providers range from the surgeon, trainees, advanced practice providers, and hospitalists. The natural variability within OMFS practices creates the potential for medication errors that can lead to increased patient morbidity and cost. Being able to precisely predict medication order errors in OMFS patients using AI/ML could decrease patient morbidity, cost, and alert fatigue.\u003c/p\u003e \u003cp\u003eThe purpose of this study was to determine if supervised machine learning algorithms can predict intercepted medication ordering errors within OMFS inpatients. Retrospective data including patient demographics, medications, comorbidities, surgeries, vital signs, lab values, and providers were collected over a 5-year span from a single OMFS hospital inpatient service. Supervised machine learning algorithms were then trained, validated, tested, and analyzed for diagnostic performance. The investigators hypothesize that supervised machine learning can accurately predict intercepted medication orders.\u003c/p\u003e \u003cp\u003eStudy aims:\u003c/p\u003e \u003cp\u003e \u003col\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eCharacterize factors associated with intercepted medication orders\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eDetermine if supervised machine learning algorithms can accurately predict intercepted medication orders\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eCompare performance of machine learning algorithms predicting intercepted medication orders\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003c/ol\u003e \u003c/p\u003e "},{"header":"Methods","content":" \u003cp\u003eA TRIPOD checklist was used for development of the prediction model. See S1 for detailed list.\u003c/p\u003e \u003cp\u003eSetting\u003c/p\u003e \u003cp\u003e The Mayo Clinic Institutional Review Board approved this study (22-007742). Mayo Clinic Hospital in Rochester is a tertiary level I trauma center and teaching institution. CPOE and patient data entry was performed using EPIC. Mayo Clinic implemented EPIC EHR in May of 2018. Data was collected from the Mayo Clinic Unified Data Platform (UDP) using custom queries.\u003c/p\u003e \u003cp\u003eData\u003c/p\u003e \u003cp\u003eMedication order history was retrospectively collected from all patients admitted to the OMFS service at the Mayo Clinic in Rochester, MN from 5/1/2018 to 12/31/2022.\u003c/p\u003e \u003cp\u003eData was cleaned, formatted, and analyzed using Microsoft Excel and Python version 3.12.\u003c/p\u003e \u003cp\u003eFor this study, only medication orders placed during an admission to service floor were used. Intra-operative and discharge medication orders were excluded. Medication orders entered by non-OMFS providers were not included in this study. At this institution, laboratory work is not indicated or routinely ordered for a significant number of the OMFS patients. For this study, missing laboratory values were imputed with the cohort median value.\u003c/p\u003e \u003cp\u003eVariables\u003c/p\u003e \u003cp\u003eIndependent variables included patient age, gender, race, weight, height, comorbidities problem list, vital signs, lab values, type of medication ordered, medication frequency, medication dose, medication form, procedures underwent during admission, time of order placement, day of order placement, provider placing the order, and admitting attending. Dependent variable of interest is if a medication order was cancelled or not. In addition, medication orders discontinued within 2 hours of being ordered, without being completed, were considered a cancelled order for this study. Distribution of time to order discontinuation was analyzed and orders discontinued within 2 hours were determined to likely represent an error. In addition, previously published medication error rates were similar and thus the 2 hour mark was used as the cutoff for this study [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eMedication order, form, route, units, frequency, and dose were all recorded for each admission. Active problem list comorbidities were collected from all admissions. Vital signs including heart rate, respiratory rate, blood pressure, temperature, and oxygen saturation were collected from each hospital admission. Lab values from admissions including potassium, blood glucose, sodium, creatinine, magnesium, phosphorous, hemoglobin, white blood cell count, and platelets were collected. Vital sign and lab data was winsorized and inspected for outliers likely caused by entry error. Outliers thought to be caused by entry error were removed. Provider placing the order was categorized into resident and advanced practice provider. Medication doses were z-scored using unique combination of medication, unit, and route.\u003c/p\u003e \u003cp\u003eProcedures underwent during the admission were also collected. Procedures underwent during admission were categorized into dentoalveolar, orthognathic/craniofacial, pathology/reconstruction, temporomandibular joint surgery, infection related, and other surgeries. Problem list comorbidities were binned into ICD-9 categories of infection, neoplasm, autoimmune related, endocrine, psychiatric, neurological, ophthalmologic, ear, circulatory, gastrointestinal, dermatology, musculoskeletal, urinary, pregnancy related, and trauma related. The maximum and minimum value for vital signs and lab values during an admission were used for modeling. Time of order placement was sub-categorized into the hour of the day and day of the week.\u003c/p\u003e \u003cdiv id=\"Sec2\" class=\"Section2\"\u003e \u003ch2\u003eStatistical Analysis\u003c/h2\u003e \u003cp\u003eDescriptive statistics and univariable regression were performed where indicated to analyze the dataset and determine which independent variables were most closely associated with self-intercepted medication orders.\u003c/p\u003e \u003cp\u003eAlgorithms were validated using stratified k-fold cross validation to account for class imbalance. The dataset was divided into 5 folds. One of the folds was held out for testing while the other folds were used for training. Stratified k-fold cross validation splits the data into k subsets of equal size and maintains the overall class distribution for each fold. Synthetic Minority Oversampling Technique (SMOTE) was also used on the training data folds to address class imbalance. SMOTE synthetically creates data that closely mirrors the minority class using k-nearest neighbor to create a more balanced dataset. Scikit-Learn was used to train and test Gradient Boosted Decision Trees (GBDT), Random Forest (RF), k-Nearest Neighbors (kNN), and Na\u0026iuml;ve Bayes (NB). Hyperparameters that were associated with the greatest area under the receiver operating characteristic curve (AUROC) during training and validation were used for algorithm testing. Performance of machine learning algorithms including AUROC, Area under Precision-Recall Curve (AUPRC), precision, recall, F1-score, and accuracy was compared using the unseen k-fold dataset.\u003c/p\u003e \u003cp\u003eParameter grid search was used for to search for hyperparameters that resulted in the highest algorithm performance. For GBDT, best hyperparameters were learning_rate: 0.1, max_depth: 7, and n_estimators: 50. For RF, best hyperparameters were max_depth: 20, min_samples_leaf: 4, min_samples_leaf: 4, n_estimators: 100, and random_state: 42. For kNN, best hyperparameters were n_neighbors: 3, weights: distance, and algorithm: auto.\u003c/p\u003e \u003c/div\u003e"},{"header":"Results","content":"\u003cp\u003eAfter data processing and cleaning, 37,493 medication orders from 1,047 patients accounting for 1,204 admissions were used for analysis in this study. 3,892 (10.4%) medication orders were cancelled or voided within 2 hours of order placement without administration. 15.99% of medication orders placed on a weekend were cancelled compared to 9.97% of medication orders placed on a weekday. 11.87% of orders placed by residents were cancelled compared to 6.21% of orders placed by APPs. See Table 1 for medication order characteristics.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eArea under the receiver operating characteristic (AUROC) for GBDT was 0.802 with 95% CI [0.787, 0.825]. Area under the precision recall curve (AUPRC) was 0.6864 with 95% CI [0.679, 0.702] (Figure 1). GBDT had a weighted precision and recall of 0.93 and 0.93, respectively (Figure 2). F1-score for GBDT was 0.93. Top 5 most important features for GBDT with F score were hour_of_day (350), age (207), weight_kg (189), height_cm (160), and Z_Score_dose (143) (Figure 2).\u003c/p\u003e\n\u003cp\u003eAUROC for RF was 0.746 with 95% CI [0.722, 0.765]. AUPRC was 0.647 with 95% CI [0.638, 0.664] (Figure 1). RF had a weighted precision and recall of 0.92 and 0.93, respectively (Figure 2). F1-score for RF was 0.92.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eAUROC for kNN was 0.685 with 95% CI [0.667, 0.699]. AUPRC was 0.429 with 95% CI [0.417, 0.434] (Figure 1). kNN had a weighted precision and recall of 0.87 and 0.82, respectively. F1-score for kNN was 0.84 (Figure 2).\u003c/p\u003e\n\u003cp\u003eAUROC for NB was 0.505 with 95% CI [0.489, 0.539]. AUPRC was 0.551 with 95% CI [0.551, 0.552] (Figure 1). NB had a weighted precision and recall of 0.89 and 0.12, respectively (Figure 2). F1-score for NB was 0.04.\u0026nbsp;\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eHealthcare systems and providers are constantly attempting to minimize medication errors, however, various strategies to reduce and catch errors have had limited success and unintended consequences. We believe that this study is the first to report on predicting intercepted CPOE using data from a single hospital service. We retrospectively collected data generated from OMFS inpatients at a single institution over approximately 5 years. The data was then used to train, validate, and test GBDT, RF, KNN, and NB. GBDT (AUROC\u0026thinsp;=\u0026thinsp;0.802) resulted in the best performance of the algorithms tested. While GBDT does not give feature coefficients like traditional regression models, feature importance can be viewed. In this study, the most important feature was the hour of day that the order was placed. This aligns with clinical factors such as increased provider fatigue and decreased trainee oversight at night.\u003c/p\u003e \u003cp\u003eThis study helps to characterize factors associated with medication orders used for OMFS inpatients at a single institution. It also demonstrates that supervised machine learning algorithms can accurately predict intercepted medication orders for OMFS inpatients. Relative algorithm performance for AUROC was similar to the results of the study by King et al when self-intercepted CPOE was predicted in a hospital wide dataset. In our study, GBDT and RF performed better at precision and recall compared to King et al. This is potentially due to using a single specialty dataset in this study. Even larger datasets from specific specialties have the potential for improved algorithm performance compared to non-specific hospital wide datasets. Using specialty specific data for algorithm training can allow for improved domain specific feature selection and decrease noise from irrelevant and redundant features [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eDomain specific features and feature selection strategies have been applied towards consumer credit risk and cybersecurity models to improve model performance [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e], Khandani et al used generalized classification and regression trees (CART) in order to recursively select independent features that improved model performance while also increasing model interpretability for consumer credit risk predictions. Datasets used for model training in cybersecurity anomaly detection are also commonly composed of high-dimensionality with many features [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e]. Dimensionality reduction strategies used in the cybersecurity industry include univariate feature selection, correlated feature elimination, and gradient boosting [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e]. Applying these various strategies to predicting intercepted CPOE will likely continue to improve model performance.\u003c/p\u003e \u003cp\u003eDue to the nature of the data, a significant amount of class imbalance exists. Various techniques including random oversampling and synthetic oversampling have been described to account for class imbalance during model training for credit card fraud detection. In credit card transaction datasets, large class imbalance exists between genuine transactions compared to fraudulent transactions. Synthetic Minority Over-sampling Technique (SMOTE) was described by Chawla et al in order to overcome dataset imbalance [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e]. Since then many credit card fraud detection models have augmented their datasets using SMOTE during model training in order to improve prediction performance on highly imbalanced datasets [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e]. SMOTE increased algorithm performance for all algorithms tested during our study.\u003c/p\u003e \u003cp\u003eWeaknesses of the study include dataset size, missing laboratory data, and using vital signs and lab values from the entire admission. At the time of data collection, the institution had implemented EPIC as its EHR and CPOE for approximately 5 years. Combining data from the previous CPOE system and EHR was not practical due to the voiding and cancellation process of the previous system. Handling of missing data is important for optimizing algorithm performance without causing overfitting or losing statistical power. Missing lab values in the setting of OMFS inpatients are a common occurrence due to the lack of clinical indication. For this study, missing laboratory data was addressed by imputing the median value for the labs. Removing medication orders associated with admissions without lab values would lead to a high percentage of the dataset being removed. The use of AI/ML to predict outcomes should rely on data that is collected before an outcome occurs. In this study, vitals and lab values from over the course of the entire admission were used. Future studies should use vitals and lab data collected from the initial admission or vitals collected prior to medication orders being placed such as during a procedure in the operating room. In addition, the use of intercepted medication orders as a surrogate for ordering errors is not perfect. However, previous studies have demonstrated that approximately 70% \u0026minus;\u0026thinsp;90% of self-intercepted orders are errors [\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e].\u003c/p\u003e"},{"header":"Conclusions","content":"\u003cp\u003eOur results demonstrated that GBDT, a supervised machine learning algorithm, can use patient factors and clinical scenario to accurately predict discontinued CPOE. Future implementation of machine learning based CDSSs at level of EMR CPOE could decrease medication errors in real time as providers place orders while also alleviating the need for rule based alert systems. This could ultimately reduce hospital costs, alert fatigue, and patient complications. Further studies and validation are still needed prior to clinical implementation of AI/ML guided CDSS.\u003c/p\u003e "},{"header":"Declarations","content":"\u003cp\u003eDeclaration of Consent\u003c/p\u003e\n\u003cp\u003eThe Mayo Clinic Institutional Review Board waived the requirement for HIPAA Authorization Waiver and patient consent.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eIRB# 22-007742\u003c/p\u003e\n\u003cp\u003eEthics Declaration\u003c/p\u003e\n\u003cp\u003eThe research was conducted in accordance with the Declaration of Helsinki.\u003c/p\u003e\n\u003cp\u003eFunding\u003c/p\u003e\n\u003cp\u003eNone\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003e\u0026ldquo;Computer Physician Order Entry: Benefits, Costs, and Issues | Annals of Internal Medicine.\u0026rdquo; Accessed: Nov. 06, 2023. [Online]. Available: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.acpjournals.org/doi/full/\u003c/span\u003e\u003cspan address=\"https://www.acpjournals.org/doi/full/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.7326/0003-4819-139-1-200307010-00010\u003c/span\u003e\u003cspan address=\"10.7326/0003-4819-139-1-200307010-00010\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003e\u0026ldquo;Applied Sciences | Free Full-Text | Applicability of Clinical Decision Support in Management among Patients Undergoing Cardiac Surgery in Intensive Care Unit: A Systematic Review.\u0026rdquo; Accessed: Nov. 06, 2023. [Online]. Available: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.mdpi.com/2076-3417/11/6/2880\u003c/span\u003e\u003cspan address=\"https://www.mdpi.com/2076-3417/11/6/2880\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJ. S. Ancker \u003cem\u003eet al.\u003c/em\u003e, \u0026ldquo;Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system,\u0026rdquo; BMC Med. Inform. Decis. Mak., vol. 17, no. 1, p. 36, Apr. 2017, doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1186/s12911-017-0430-8\u003c/span\u003e\u003cspan address=\"10.1186/s12911-017-0430-8\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eT. K. Nuckols \u003cem\u003eet al.\u003c/em\u003e, \u0026ldquo;The effectiveness of computerized order entry at reducing preventable adverse drug events and medication errors in hospital settings: a systematic review and meta-analysis,\u0026rdquo; Syst. Rev., vol. 3, no. 1, p. 56, Jun. 2014, doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1186/2046-4053-3-56\u003c/span\u003e\u003cspan address=\"10.1186/2046-4053-3-56\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003e\u0026ldquo;Reducing drug prescription errors and adverse drug events by application of a probabilistic, machine-learning based clinical decision support system in an inpatient setting | Journal of the American Medical Informatics Association | Oxford Academic.\u0026rdquo; Accessed: Nov. 06, 2023. [Online]. Available: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://academic.oup.com/jamia/article/26/12/1560/5544737\u003c/span\u003e\u003cspan address=\"https://academic.oup.com/jamia/article/26/12/1560/5544737\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003e\u0026ldquo;Predicting self-intercepted medication ordering errors using machine learning | PLOS ONE.\u0026rdquo; Accessed: Nov. 06, 2023. [Online]. Available: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://journals.plos.org/plosone/article?id=10.1371/journal.pone.0254358\u003c/span\u003e\u003cspan address=\"https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0254358\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eD. G. Krishnan, \u0026ldquo;Artificial Intelligence in Oral and Maxillofacial Surgery Education,\u0026rdquo; Oral Maxillofac. Surg. Clin., vol. 34, no. 4, pp. 585\u0026ndash;591, Nov. 2022, doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.coms.2022.03.006\u003c/span\u003e\u003cspan address=\"10.1016/j.coms.2022.03.006\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eP. Bouletreau, M. Makaremi, B. Ibrahim, A. Louvrier, and N. Sigaux, \u0026ldquo;Artificial Intelligence: Applications in orthognathic surgery,\u0026rdquo; \u003cem\u003eJ. Stomatol. Oral Maxillofac. Surg.\u003c/em\u003e, vol. 120, no. 4, pp. 347\u0026ndash;354, Sep. 2019, doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.jormas.2019.06.001\u003c/span\u003e\u003cspan address=\"10.1016/j.jormas.2019.06.001\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003e\u0026ldquo;Artificial intelligence for oral and maxillo-facial surgery: A narrative review - ScienceDirect.\u0026rdquo; Accessed: Nov. 06, 2023. [Online]. Available: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.sciencedirect.com/science/article/pii/S2468785522000271\u003c/span\u003e\u003cspan address=\"https://www.sciencedirect.com/science/article/pii/S2468785522000271\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJ. Watson \u003cem\u003eet al.\u003c/em\u003e, \u0026ldquo;Overcoming barriers to the adoption and implementation of predictive modeling and machine learning in clinical care: what can we learn from US academic medical centers?,\u0026rdquo; \u003cem\u003eJAMIA Open\u003c/em\u003e, vol. 3, no. 2, pp. 167\u0026ndash;172, Jul. 2020, doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1093/jamiaopen/ooz046\u003c/span\u003e\u003cspan address=\"10.1093/jamiaopen/ooz046\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eS. Shirgave, C. Awati, R. More, and S. Patil, \u0026ldquo;A Review On Credit Card Fraud Detection Using Machine Learning,\u0026rdquo; \u003cem\u003eInt. J. Sci. Technol. Res.\u003c/em\u003e, vol. 8, pp. 1217\u0026ndash;1220, Oct. 2019.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eT. Wuest, D. Weimer, C. Irgens, and K.-D. Thoben, \u0026ldquo;Machine learning in manufacturing: advantages, challenges, and applications,\u0026rdquo; Prod. Manuf. Res., vol. 4, no. 1, pp. 23\u0026ndash;45, Jan. 2016, doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1080/21693277.2016.1192517\u003c/span\u003e\u003cspan address=\"10.1080/21693277.2016.1192517\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eA. Handa, A. Sharma, and S. K. Shukla, \u0026ldquo;Machine learning in cybersecurity: A review,\u0026rdquo; WIREs Data Min. Knowl. Discov., vol. 9, no. 4, p. e1306, 2019, doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1002/widm.1306\u003c/span\u003e\u003cspan address=\"10.1002/widm.1306\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJ. Nanduri, Y. Jia, A. Oka, J. Beaver, and Y.-W. Liu, \u0026ldquo;Microsoft Uses Machine Learning and Optimization to Reduce E-Commerce Fraud,\u0026rdquo; \u003cem\u003eInf. J. Appl. Anal.\u003c/em\u003e, vol. 50, no. 1, pp. 64\u0026ndash;79, Jan. 2020, doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1287/inte.2019.1017\u003c/span\u003e\u003cspan address=\"10.1287/inte.2019.1017\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003e\u0026ldquo;Machine Learning for Anomaly Detection: A Systematic Review | IEEE Journals \u0026amp; Magazine | IEEE Xplore.\u0026rdquo; Accessed: Jan. 03, 2024. [Online]. Available: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://ieeexplore.ieee.org/abstract/document/9439459\u003c/span\u003e\u003cspan address=\"https://ieeexplore.ieee.org/abstract/document/9439459\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eL. Briceland and D. S. Stein, \u0026ldquo;Factors Related to Errors in Medication Prescribing\u0026rdquo;.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eL. Budach \u003cem\u003eet al.\u003c/em\u003e, \u0026ldquo;The Effects of Data Quality on Machine Learning Performance,\u0026rdquo; arXiv.org. Accessed: Jan. 03, 2024. [Online]. Available: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://arxiv.org/abs/2207.14529v4\u003c/span\u003e\u003cspan address=\"https://arxiv.org/abs/2207.14529v4\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eM. Ahsan, R. Gomes, M. M. Chowdhury, and K. E. Nygard, \u0026ldquo;Enhancing Machine Learning Prediction in Cybersecurity Using Dynamic Feature Selector,\u0026rdquo; \u003cem\u003eJ. Cybersecurity Priv.\u003c/em\u003e, vol. 1, no. 1, Art. no. 1, Mar. 2021, doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.3390/jcp1010011\u003c/span\u003e\u003cspan address=\"10.3390/jcp1010011\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eA. E. Khandani, A. J. Kim, and A. W. Lo, \u0026ldquo;Consumer credit-risk models via machine-learning algorithms,\u0026rdquo; \u003cem\u003eJ. Bank. Finance\u003c/em\u003e, vol. 34, no. 11, pp. 2767\u0026ndash;2787, Nov. 2010, doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.jbankfin.2010.06.001\u003c/span\u003e\u003cspan address=\"10.1016/j.jbankfin.2010.06.001\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eA. Alshaibi, M. Al-Ani, A. Al-Azzawi, A. Konev, and A. Shelupanov, \u0026ldquo;The Comparison of Cybersecurity Datasets,\u0026rdquo; \u003cem\u003eData\u003c/em\u003e, vol. 7, no. 2, Art. no. 2, Feb. 2022, doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.3390/data7020022\u003c/span\u003e\u003cspan address=\"10.3390/data7020022\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eN. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, \u0026ldquo;SMOTE: Synthetic Minority Over-sampling Technique,\u0026rdquo; \u003cem\u003eJ. Artif. Intell. Res.\u003c/em\u003e, vol. 16, pp. 321\u0026ndash;357, Jun. 2002, doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1613/jair.953\u003c/span\u003e\u003cspan address=\"10.1613/jair.953\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eA. Thennakoon, C. Bhagyani, S. Premadasa, S. Mihiranga, and N. Kuruwitaarachchi, Real-time Credit Card Fraud Detection Using Machine Learning. 2019. doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1109/CONFLUENCE.2019.8776942\u003c/span\u003e\u003cspan address=\"10.1109/CONFLUENCE.2019.8776942\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eD. Varmedja, M. Karanovic, S. Sladojevic, M. Arsenovic, and A. Anderla, \u0026ldquo;Credit Card Fraud Detection - Machine Learning methods,\u0026rdquo; in \u003cem\u003e2019 18th International Symposium INFOTEH-JAHORINA (INFOTEH)\u003c/em\u003e, East Sarajevo, Bosnia and Herzegovina: IEEE, Mar. 2019, pp. 1\u0026ndash;5. doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1109/INFOTEH.2019.8717766\u003c/span\u003e\u003cspan address=\"10.1109/INFOTEH.2019.8717766\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eT. G. Kannampallil \u003cem\u003eet al.\u003c/em\u003e, \u0026ldquo;Learning from errors: analysis of medication order voiding in CPOE systems,\u0026rdquo; \u003cem\u003eJ. Am. Med. Inform. Assoc.\u003c/em\u003e, vol. 24, no. 4, pp. 762\u0026ndash;768, Jul. 2017, doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1093/jamia/ocw187\u003c/span\u003e\u003cspan address=\"10.1093/jamia/ocw187\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJ. Abraham \u003cem\u003eet al.\u003c/em\u003e, \u0026ldquo;Reasons for computerised provider order entry (CPOE)-based inpatient medication ordering errors: an observational study of voided orders,\u0026rdquo; BMJ Qual. Saf., vol. 27, no. 4, pp. 299\u0026ndash;307, Apr. 2018, doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1136/bmjqs-2017-006606\u003c/span\u003e\u003cspan address=\"10.1136/bmjqs-2017-006606\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"},{"header":"Tables","content":"\u003ctable border=\"0\" cellspacing=\"0\" cellpadding=\"0\" width=\"461\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd width=\"47.071583514099785%\" valign=\"bottom\"\u003e\n \u003cp\u003e\u003cstrong\u003eVariable\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"29.718004338394795%\" valign=\"bottom\"\u003e\n \u003cp\u003e\u003cstrong\u003eNon-Voided Order\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"23.210412147505423%\" valign=\"bottom\"\u003e\n \u003cp\u003e\u003cstrong\u003eVoided Order\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"47.071583514099785%\" valign=\"bottom\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd width=\"29.718004338394795%\" valign=\"bottom\"\u003e\n \u003cp\u003e33601 (89.6%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"23.210412147505423%\" valign=\"bottom\"\u003e\n \u003cp\u003e3892 (10.4%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"47.071583514099785%\" valign=\"bottom\"\u003e\n \u003cp\u003e\u003cstrong\u003eAge\u003c/strong\u003e (mean, SD)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"29.718004338394795%\" valign=\"bottom\"\u003e\n \u003cp\u003e48.7 (21.4)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"23.210412147505423%\" valign=\"bottom\"\u003e\n \u003cp\u003e51.7 (20.8)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"47.071583514099785%\" valign=\"bottom\"\u003e\n \u003cp\u003e\u003cstrong\u003eSex\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"29.718004338394795%\" valign=\"bottom\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd width=\"23.210412147505423%\" valign=\"bottom\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"47.071583514099785%\" valign=\"bottom\"\u003e\n \u003cp\u003eFemale\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"29.718004338394795%\" valign=\"bottom\"\u003e\n \u003cp\u003e19646 (58%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"23.210412147505423%\" valign=\"bottom\"\u003e\n \u003cp\u003e2330 (60%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"47.071583514099785%\" valign=\"bottom\"\u003e\n \u003cp\u003eMale\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"29.718004338394795%\" valign=\"bottom\"\u003e\n \u003cp\u003e13955 (42%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"23.210412147505423%\" valign=\"bottom\"\u003e\n \u003cp\u003e1562 (40%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"47.071583514099785%\" valign=\"bottom\"\u003e\n \u003cp\u003e\u003cstrong\u003eRace/Ethnicity\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"29.718004338394795%\" valign=\"bottom\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd width=\"23.210412147505423%\" valign=\"bottom\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"47.071583514099785%\" valign=\"bottom\"\u003e\n \u003cp\u003eCaucasian\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"29.718004338394795%\" valign=\"bottom\"\u003e\n \u003cp\u003e30635 (91%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"23.210412147505423%\" valign=\"bottom\"\u003e\n \u003cp\u003e3579 (92%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"47.071583514099785%\" valign=\"bottom\"\u003e\n \u003cp\u003eAsian\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"29.718004338394795%\" valign=\"bottom\"\u003e\n \u003cp\u003e932 (3%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"23.210412147505423%\" valign=\"bottom\"\u003e\n \u003cp\u003e61 (2%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"47.071583514099785%\" valign=\"bottom\"\u003e\n \u003cp\u003eBlack\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"29.718004338394795%\" valign=\"bottom\"\u003e\n \u003cp\u003e709 (2%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"23.210412147505423%\" valign=\"bottom\"\u003e\n \u003cp\u003e96 (2%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"47.071583514099785%\" valign=\"bottom\"\u003e\n \u003cp\u003eOther\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"29.718004338394795%\" valign=\"bottom\"\u003e\n \u003cp\u003e1325 (4%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"23.210412147505423%\" valign=\"bottom\"\u003e\n \u003cp\u003e156 (4%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"47.071583514099785%\" valign=\"bottom\"\u003e\n \u003cp\u003e\u003cstrong\u003eDay of Week\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"29.718004338394795%\" valign=\"bottom\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd width=\"23.210412147505423%\" valign=\"bottom\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"47.071583514099785%\" valign=\"bottom\"\u003e\n \u003cp\u003eWeekend\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"29.718004338394795%\" valign=\"bottom\"\u003e\n \u003cp\u003e2139 (6%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"23.210412147505423%\" valign=\"bottom\"\u003e\n \u003cp\u003e407 (10%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"47.071583514099785%\" valign=\"bottom\"\u003e\n \u003cp\u003eWeekday\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"29.718004338394795%\" valign=\"bottom\"\u003e\n \u003cp\u003e31462 (94%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"23.210412147505423%\" valign=\"bottom\"\u003e\n \u003cp\u003e3485 (90%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"47.071583514099785%\" valign=\"bottom\"\u003e\n \u003cp\u003e\u003cstrong\u003eShift\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"29.718004338394795%\" valign=\"bottom\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd width=\"23.210412147505423%\" valign=\"bottom\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"47.071583514099785%\" valign=\"bottom\"\u003e\n \u003cp\u003eDay\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"29.718004338394795%\" valign=\"bottom\"\u003e\n \u003cp\u003e24811 (74%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"23.210412147505423%\" valign=\"bottom\"\u003e\n \u003cp\u003e2470 (63%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"47.071583514099785%\" valign=\"bottom\"\u003e\n \u003cp\u003eNight\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"29.718004338394795%\" valign=\"bottom\"\u003e\n \u003cp\u003e8790 (26%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"23.210412147505423%\" valign=\"bottom\"\u003e\n \u003cp\u003e1422 (37%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"47.071583514099785%\" valign=\"bottom\"\u003e\n \u003cp\u003e\u003cstrong\u003eOrdering Provider\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"29.718004338394795%\" valign=\"bottom\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd width=\"23.210412147505423%\" valign=\"bottom\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"47.071583514099785%\" valign=\"bottom\"\u003e\n \u003cp\u003eResident\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"29.718004338394795%\" valign=\"bottom\"\u003e\n \u003cp\u003e24351 (72%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"23.210412147505423%\" valign=\"bottom\"\u003e\n \u003cp\u003e3280 (84%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"47.071583514099785%\" valign=\"bottom\"\u003e\n \u003cp\u003eAPP\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"29.718004338394795%\" valign=\"bottom\"\u003e\n \u003cp\u003e9250 (28%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"23.210412147505423%\" valign=\"bottom\"\u003e\n \u003cp\u003e612 (16%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"47.071583514099785%\" valign=\"bottom\"\u003e\n \u003cp\u003e\u003cstrong\u003eMedication Type\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"29.718004338394795%\" valign=\"bottom\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd width=\"23.210412147505423%\" valign=\"bottom\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"47.071583514099785%\" valign=\"bottom\"\u003e\n \u003cp\u003eAntimicrobial\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"29.718004338394795%\" valign=\"bottom\"\u003e\n \u003cp\u003e4362 (12%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"23.210412147505423%\" valign=\"bottom\"\u003e\n \u003cp\u003e435 (11%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"47.071583514099785%\" valign=\"bottom\"\u003e\n \u003cp\u003eAnalgesic\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"29.718004338394795%\" valign=\"bottom\"\u003e\n \u003cp\u003e9716 (29%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"23.210412147505423%\" valign=\"bottom\"\u003e\n \u003cp\u003e1391 (36%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"47.071583514099785%\" valign=\"bottom\"\u003e\n \u003cp\u003eCirculatory\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"29.718004338394795%\" valign=\"bottom\"\u003e\n \u003cp\u003e5592 (17%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"23.210412147505423%\" valign=\"bottom\"\u003e\n \u003cp\u003e723 (19%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"47.071583514099785%\" valign=\"bottom\"\u003e\n \u003cp\u003eRespiratory\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"29.718004338394795%\" valign=\"bottom\"\u003e\n \u003cp\u003e898 (3%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"23.210412147505423%\" valign=\"bottom\"\u003e\n \u003cp\u003e29 (1%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"47.071583514099785%\" valign=\"bottom\"\u003e\n \u003cp\u003eGI/Endo\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"29.718004338394795%\" valign=\"bottom\"\u003e\n \u003cp\u003e3374 (10%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"23.210412147505423%\" valign=\"bottom\"\u003e\n \u003cp\u003e195 (5%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"47.071583514099785%\" valign=\"bottom\"\u003e\n \u003cp\u003eOther\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"29.718004338394795%\" valign=\"bottom\"\u003e\n \u003cp\u003e9627 (29%)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"23.210412147505423%\" valign=\"bottom\"\u003e\n \u003cp\u003e1117 (28%)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"47.071583514099785%\" valign=\"bottom\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd width=\"29.718004338394795%\" valign=\"bottom\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd width=\"23.210412147505423%\" valign=\"bottom\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"47.071583514099785%\" valign=\"bottom\"\u003e\n \u003cp\u003eSD = standard deviation\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"29.718004338394795%\" valign=\"bottom\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd width=\"23.210412147505423%\" valign=\"bottom\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"47.071583514099785%\" valign=\"bottom\"\u003e\n \u003cp\u003eAPP = advanced practice provider\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"29.718004338394795%\" valign=\"bottom\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd width=\"23.210412147505423%\" valign=\"bottom\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eTable 1 Medication order cohort characteristics.\u0026nbsp;\u003c/p\u003e\n\u003ctable border=\"0\" cellspacing=\"0\" cellpadding=\"0\" title=\"Table 2\" summary=\"Classificatin report for GBDT, RF, kNN, and NB.\" width=\"704\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd width=\"55.25568181818182%\" colspan=\"4\" valign=\"bottom\"\u003e\n \u003cp\u003e\u003cstrong\u003eGBDT\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"44.74431818181818%\" colspan=\"4\" valign=\"bottom\"\u003e\n \u003cp\u003e\u003cstrong\u003eRF\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"17.613636363636363%\" valign=\"bottom\"\u003e\n \u003cp\u003eOutcome\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.795454545454545%\" valign=\"bottom\"\u003e\n \u003cp\u003ePrecision\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"13.494318181818182%\" valign=\"bottom\"\u003e\n \u003cp\u003eRecall\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"13.494318181818182%\" valign=\"bottom\"\u003e\n \u003cp\u003eF1-score\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.795454545454545%\" valign=\"bottom\"\u003e\n \u003cp\u003eOutcome\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"11.931818181818182%\" valign=\"bottom\"\u003e\n \u003cp\u003ePrecision\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.9375%\" valign=\"bottom\"\u003e\n \u003cp\u003eRecall\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.9375%\" valign=\"bottom\"\u003e\n \u003cp\u003eF1-score\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"17.613636363636363%\" valign=\"bottom\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.795454545454545%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.96\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"13.494318181818182%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.97\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"13.494318181818182%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.96\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.795454545454545%\" valign=\"bottom\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"11.931818181818182%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.95\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.9375%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.98\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.9375%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.96\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"17.613636363636363%\" valign=\"bottom\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.795454545454545%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.69\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"13.494318181818182%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.64\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"13.494318181818182%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.66\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.795454545454545%\" valign=\"bottom\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"11.931818181818182%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.73\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.9375%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.51\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.9375%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.6\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"17.613636363636363%\" valign=\"bottom\"\u003e\n \u003cp\u003eMacro Average\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.795454545454545%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.83\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"13.494318181818182%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.8\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"13.494318181818182%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.81\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.795454545454545%\" valign=\"bottom\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd width=\"11.931818181818182%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.84\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.9375%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.75\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.9375%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.78\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"17.613636363636363%\" valign=\"bottom\"\u003e\n \u003cp\u003eWeighted Average\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.795454545454545%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.93\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"13.494318181818182%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.93\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"13.494318181818182%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.93\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.795454545454545%\" valign=\"bottom\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd width=\"11.931818181818182%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.92\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.9375%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.93\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.9375%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.92\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"17.613636363636363%\" valign=\"bottom\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd width=\"10.795454545454545%\" valign=\"bottom\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd width=\"13.494318181818182%\" valign=\"bottom\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd width=\"13.494318181818182%\" valign=\"bottom\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd width=\"10.795454545454545%\" valign=\"bottom\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd width=\"11.931818181818182%\" valign=\"bottom\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd width=\"10.9375%\" valign=\"bottom\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd width=\"10.9375%\" valign=\"bottom\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"55.25568181818182%\" colspan=\"4\" valign=\"bottom\"\u003e\n \u003cp\u003e\u003cstrong\u003ekNN\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"44.74431818181818%\" colspan=\"4\" valign=\"bottom\"\u003e\n \u003cp\u003e\u003cstrong\u003eNB\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"17.613636363636363%\" valign=\"bottom\"\u003e\n \u003cp\u003eOutcome\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.795454545454545%\" valign=\"bottom\"\u003e\n \u003cp\u003ePrecision\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"13.494318181818182%\" valign=\"bottom\"\u003e\n \u003cp\u003eRecall\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"13.494318181818182%\" valign=\"bottom\"\u003e\n \u003cp\u003eF1-score\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.795454545454545%\" valign=\"bottom\"\u003e\n \u003cp\u003eOutcome\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"11.931818181818182%\" valign=\"bottom\"\u003e\n \u003cp\u003ePrecision\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.9375%\" valign=\"bottom\"\u003e\n \u003cp\u003eRecall\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.9375%\" valign=\"bottom\"\u003e\n \u003cp\u003eF1-score\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"17.613636363636363%\" valign=\"bottom\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.795454545454545%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.94\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"13.494318181818182%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.85\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"13.494318181818182%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.89\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.795454545454545%\" valign=\"bottom\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"11.931818181818182%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.98\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.9375%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.01\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.9375%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.03\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"17.613636363636363%\" valign=\"bottom\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.795454545454545%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.29\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"13.494318181818182%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.52\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"13.494318181818182%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.37\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.795454545454545%\" valign=\"bottom\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"11.931818181818182%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.9375%\" valign=\"bottom\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.9375%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.19\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"17.613636363636363%\" valign=\"bottom\"\u003e\n \u003cp\u003eMacro Average\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.795454545454545%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.61\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"13.494318181818182%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.69\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"13.494318181818182%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.63\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.795454545454545%\" valign=\"bottom\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"11.931818181818182%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.54\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.9375%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.51\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.9375%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.11\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"17.613636363636363%\" valign=\"bottom\"\u003e\n \u003cp\u003eWeighted Average\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.795454545454545%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.87\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"13.494318181818182%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.82\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"13.494318181818182%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.84\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.795454545454545%\" valign=\"bottom\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"11.931818181818182%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.89\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.9375%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.12\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.9375%\" valign=\"bottom\"\u003e\n \u003cp\u003e0.04\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u0026nbsp;Table 2 GBDT, RF, kNN, and NB classification report.\u003c/p\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"oral-and-maxillofacial-surgery","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"omfs","sideBox":"Learn more about [Oral and Maxillofacial Surgery](http://link.springer.com/journal/10006)","snPcode":"10006","submissionUrl":"https://submission.nature.com/new-submission/10006/3","title":"Oral and Maxillofacial Surgery","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-3868326/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-3868326/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eObjective\u003c/h2\u003e \u003cp\u003eThe aim of this study is to determine if supervised machine learning algorithms can accurately predict cancelled and self-intercepted computerized physician order entry in oral and maxillofacial surgery inpatients.\u003c/p\u003e\u003ch2\u003eMethods\u003c/h2\u003e \u003cp\u003eData from Electronic Medical Record included patient demographics, comorbidities, procedures, vital signs, laboratory values, and medication orders were retrospectively collected. Predictor variables included patient demographics, comorbidities, procedures, vital signs, and laboratory values. Outcome of interest is if a medication order was voided or not. Data was cleaned and pro1cessed using Microsoft Excel and Python v3.12. Gradient Boosted Decision Trees, Random Forest, K-Nearest Neighbor, and Na\u0026iuml;ve Bayes were trained, validated, and tested for accuracy of the prediction of voided medication orders.\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e \u003cp\u003e37,546 medication orders from 1,204 patient admissions over 5 years were used for this study included 3,892 (10.4%) medication orders that were voided. Gradient Boosted Decision Trees, Random Forest, K-Nearest Neighbor, and Na\u0026iuml;ve Bayes had an Area Under the Receiver Operating Curve of 0.802 802 with 95% CI [0.787, 0.825], 0.746 with 95% CI [0.722, 0.765], 0.685 with 95% CI [0.667, 0.699], and 0.505 with 95% CI [0.489, 0.539], respectively. Area Under the Precision Recall Curve was 0.684 with 95% CI [0.679, 0.702], 0.647 with 95% CI [0.638, 0.664], 0.429 with 95% CI [0.417, 0.434], and 0.551 with 95% CI [0.551, 0.552], respectively.\u003c/p\u003e\u003ch2\u003eConclusion\u003c/h2\u003e \u003cp\u003eGradient Boosted Decision Trees is the best model of the supervised machine learning algorithms with satisfactory performance in validation cohort for predicting voided Computerized Physician Order Entry in Oral and Maxillofacial Surgery inpatients.\u003c/p\u003e","manuscriptTitle":"Predicting voided computerized physician order entry in oral and maxillofacial surgery inpatients: development and validation of machine learning model","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-01-18 15:33:33","doi":"10.21203/rs.3.rs-3868326/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2024-05-12T08:06:53+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-04-06T12:23:44+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"adc98702-e588-4d6a-9624-50d1b0f57d08","date":"2024-04-06T11:20:25+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"0c14acfb-8314-4c01-8051-3c7fc75bcdb5","date":"2024-01-22T00:06:34+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-01-21T08:27:21+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-01-17T01:05:15+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-01-17T01:05:14+00:00","index":"","fulltext":""},{"type":"submitted","content":"Oral and Maxillofacial Surgery","date":"2024-01-16T01:41:31+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"oral-and-maxillofacial-surgery","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"omfs","sideBox":"Learn more about [Oral and Maxillofacial Surgery](http://link.springer.com/journal/10006)","snPcode":"10006","submissionUrl":"https://submission.nature.com/new-submission/10006/3","title":"Oral and Maxillofacial Surgery","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"4351f57a-7bf3-4941-a2b1-735a227b977d","owner":[],"postedDate":"January 18th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2024-06-09T06:38:30+00:00","versionOfRecord":[],"versionCreatedAt":"2024-01-18 15:33:33","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-3868326","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-3868326","identity":"rs-3868326","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}