Reliability and Validity of 2 Surgical Prioritization Systems for Reinstating Nonemergent Benign Gynecologic Surgery during the COVID-19 Pandemic.

There were 93 patients in the sample. The mean distribution of patients with the mESAS ratings were 8 (9%) level 3, 33 (35%) level 2, and 52 (56%) level 1. Of the 93 patients, 12 had been deemed highest priority by the SSP scheme, and their surgeries were performed without delay. An additional 12 patients were chosen as the next level of prioritization to be scheduled as soon as allowable. These 24 patients make up the "urgent SSP". Individual Gyn-MeNTS scores ranged from 48 to 70 points. Average scores that fell into the most urgent quartile ranged from 49 to 55. The distributions of Gyn-MeNTS score assignments for the 3 reviewers can be seen in Figs. 2 to 4 . Fig. 2 Reviewer 1 distribution of Gyn-MeNTS score. Gyn-MeNTS = Gynecologic Medically-Necessary Time-Sensitive. Fig 2 Fig. 3 Reviewer 2 distribution of Gyn-MeNTS score. Gyn-MeNTS =  Gynecologic Medically-Necessary Time-Sensitive. Fig 3 Fig. 4 Reviewer 3 distribution of Gyn-MeNTS score. Gyn-MeNTS =  Gynecologic Medically-Necessary Time-Sensitive. Fig 4 Reviewer 1 distribution of Gyn-MeNTS score. Gyn-MeNTS = Gynecologic Medically-Necessary Time-Sensitive. Reviewer 2 distribution of Gyn-MeNTS score. Gyn-MeNTS =  Gynecologic Medically-Necessary Time-Sensitive. Reviewer 3 distribution of Gyn-MeNTS score. Gyn-MeNTS =  Gynecologic Medically-Necessary Time-Sensitive. The mean ± standard deviation Gyn-MeNTS scores for raters 1, 2, and 3 were 58.0 ± 4.8, 59.2 ± 4.3, and 58.0 ± 5.0, respectively. The Gyn-MeNTS scores for raters 1 and 3 did not differ (mean difference 0.03, p = .91), whereas the scores for rater 2 differed significantly from both rater 1 (mean difference −1.2, p <.001) and rater 3 (mean difference 1.2, p 0.80 (0.84 for 1 vs 2, 0.82 for 1 vs 3, and 0.82 for 2 vs 3, all p <.001) indicating strong agreement in the patient rankings, with rater 2 having a constant added to each patient's score. The Gyn-MeNTS items with the worst interrater agreement were operating room time, anticipated estimated blood loss, trial of alternative therapy, and pain severity ( Table 3 ). Table 3 Percent of patients with perfect interrater agreement on Gyn-MeNTS items between raters 1, 2, and 3 Table 3 Variable 1 vs 2 1 vs 3 2 vs 3 Average OR time 42 53 62 52 Estimated LOS 95 99 98 97 Postoperative ICU need 96 96 100 97 Anticipated EBL 72 74 73 73 Surgical team size 99 89 87 92 Anesthesia requirement 97 99 98 98 Surgical site 95 89 92 92 Trial of alternative therapy 28 41 32 34 Pain 58 63 49 57 Anemia 100 100 100 100 Impact on desired immediate fertility 97 97 97 97 Disease impact on GI/GU morbidity 96 94 95 95 Impact of 6-week delay in disease outcome 87 84 80 84 Impact of 6-week delay in surgical difficulty/risk 99 99 100 99 Age 100 100 100 100 Lung disease 100 100 100 100 Obesity/OSA 100 100 100 100 CV disease 100 100 100 100 Diabetes 100 100 100 100 Immunocompromised 100 100 100 100 Exposure to known COVID + person in past 14 days 100 100 100 100 COVID = coronavirus disease; CV = cardiovascular; EBL = estimated blood loss; GI = gastrointestinal; GU = genitourinary; Gyn-MeNTS = Gynecologic Medically-Necessary Time-Sensitive; ICU = intensive care unit; LOS = length of stay; OSA = obstructive sleep apnea; OR = operating room. Percent of patients with perfect interrater agreement on Gyn-MeNTS items between raters 1, 2, and 3 COVID = coronavirus disease; CV = cardiovascular; EBL = estimated blood loss; GI = gastrointestinal; GU = genitourinary; Gyn-MeNTS = Gynecologic Medically-Necessary Time-Sensitive; ICU = intensive care unit; LOS = length of stay; OSA = obstructive sleep apnea; OR = operating room. The mean ± standard deviation mESAS ratings were 1.40 ± 0.61 for rater 4 and 1.41 ± 0.61 for rater 5 (not different, p = .84). The weighted κ for the 2 mESAS raters was 0.57 (95% confidence interval [CI], 0.40–0.73) indicating only fair agreement between raters. Specifically, there was 73% agreement in those categorized as level 1, 38% agreement in level 2, and 50% agreement in level 3. Gyn-MeNTS scores showed fair correlation with mESAS (Spearman r −0.31, p = .003), and a moderate correlation with urgent SSP (Spearman r −0.46, p <.001). However, mESAS had a slightly stronger correlation with urgent SSP (Spearman r 0.53, p <.001). When used together to predict urgent SSP, both mESAS and Gyn-MeNTS had significant independent contributions, and the overall model had an area under the curve of 0.89, indicating excellent discrimination. For each 1-unit increase in mESAS, the odds of being labeled as urgent SSP increased by 12.78 (95% CI, 3.65–44.76; p <.001), after adjusting for the Gyn-MeNTS score. For each 1-unit increase in the Gyn-MeNTS score, the odds of being urgent SSP decreased by 0.75 (95% CI, 0.62–0.90; p = .002), after adjusting for the mESAS score. When the probabilities derived from this model were coded into quartiles, the incidence of being urgent SSP was 75% in the highest priority quartile, 14% and 12% in the middle 2 quartiles, and 0% in the lowest quartile (p <.001). The equation for calculating the probabilities is in the Supplemental Appendix . The 67th, 75th, 80th, 85th, 90th, and 95th percentiles of the Gyn-MeNTS score are shown in Table 4 . Table 4 Association of Gyn-MeNTS percentiles with SGS mESAS rank Table 4 Percentile n Gyn-MeNTS score SGS mESAS rank 1 (%) 1.5–2 (%) 2.5–3 (%) p 95th 5 ≤51 1 (20) 3 (60) 1 (20) .23 90th 11 ≤53 3 (27) 7 (64) 1 (9) .10 85th 14 ≤53.7 3 (21) 10 (71) 1 (7) .009 80th 19 ≤54.4 6 (32) 11 (58) 2 (11) .05 75th 23 ≤55 6 (26) 13 (57) 4 (17) .02 67th 33 ≤56 12 (36) 16 (48) 5 (15) .044 Gyn-MeNTS = Gynecologic Medically-Necessary Time-Sensitive; mESAS = modified Elective Surgery Acuity Scale; SGS = Society for Gynecologic Surgeons. SGS mESAS score is average score between raters 4 and 5. Gyn-MeNTS score is average score between raters 1, 2, and 3. Association of Gyn-MeNTS percentiles with SGS mESAS rank Gyn-MeNTS = Gynecologic Medically-Necessary Time-Sensitive; mESAS = modified Elective Surgery Acuity Scale; SGS = Society for Gynecologic Surgeons. SGS mESAS score is average score between raters 4 and 5. Gyn-MeNTS score is average score between raters 1, 2, and 3. To make it easier to visualize these relationships, we coded the Gyn-MeNTS into 3 levels: low, medium, and high priority, on the basis of quartiles: the lowest quartile of scores was highest priority, the middle 2 quartiles were medium priority, and the highest quartile of scores was the lowest priority. We then compared these 3 levels of Gyn-MeNTS urgency and the 3 levels of SGS priority with the urgent SSP ( Table 5 ). Table 5 Association of Gyn-MeNTS priority scores and SGS mESAS rank versus SSP rank Table 5 Urgent SSP Rating scale Lower priority Highest priority p Gyn-MeNTS .001  Top priority quartile (%) 11 (48) 12 (52)  Middle priority 2 quartiles (%) 36 (77) 11 (23)  Lowest priority quartile (%) 22 (96) 1 (4) SGS mESAS <.001  Priority 3 (top) (%) 1 (13) 7 (88)  Priority 2 (middle) (%) 21 (64) 12 (36)  Priority 1 (lowest) (%) 47 (90) 5 (10) Gyn-MeNTS = Gynecologic Medically-Necessary Time-Sensitive; mESAS = modified Elective Surgery Acuity Scale; SGS = Society for Gynecologic Surgeons; SSP = surgeon self-prioritization. Association of Gyn-MeNTS priority scores and SGS mESAS rank versus SSP rank Gyn-MeNTS = Gynecologic Medically-Necessary Time-Sensitive; mESAS = modified Elective Surgery Acuity Scale; SGS = Society for Gynecologic Surgeons; SSP = surgeon self-prioritization. Comparing Gyn-MeNTS with urgent SSP, we found that from the highest priority quartile on Gyn-MeNTS to the lowest, 52%, 23%, and 4% of the patients were urgent SSP (p = .001). For patients with mESAS levels of high, moderate, and low priority, the percentages found within the urgent SSP category were 88%, 36%, and 10%, respectively (p <.001). When looking only at the 12 of 24 patients who were deemed the highest priority by the SSP scheme and their surgeries were performed without delay, the mESAS system was able to capture 92% of these patients in its most urgent quartile, whereas the Gyn-MeNTS system captured only 67% in its most urgent quartile.

After obtaining institutional review board exemption (IRB#NCR202525), we performed a single-center retrospective cohort study evaluating the interrater reliability and validity of 2 novel prioritization systems, the SGS adaptation of the ESAS (modified ESAS [mESAS]) and a Gynecologic Medically-Necessary Time-Sensitive tool (Gyn-MeNTS). A total of 97 benign gynecologic procedures were affected between March 16, 2020, and April 30, 2020, in our tertiary academic institution in Washington, DC, including cases from 8 general obstetrician gynecologists, 2 minimally invasive gynecologic surgeons, a urogynecologist, and a gynecologic oncologist. We excluded 3 patients who had not completed their preoperative evaluations, and thus the severity of their disease was unclear. One additional patient was excluded because she suffered from a subarachnoid hemorrhage and was no longer eligible for her planned procedure. Beginning March 16, as required by our hospital system, all patients were categorized by their respective surgeons into 1 of 3 categories on the basis of their level of morbidity if delayed. Level 1 indicated no morbidity with delay, level 2 indicated some morbidity with delay, and level 3 indicated significant morbidity and/or mortality with delay. Only those deemed level 3 were initially allowed to proceed as scheduled. Levels 1 and 2 would be postponed until further notice, likely to extend 2 to 3 months. For the purpose of the validity analysis, those patients who were allowed to proceed with surgery will be referred to as urgent SSP. The ESAS scale divides elective surgeries into 3 tiers, low acuity (tier 1) defined as “nonlife-threatening illness,” intermediate acuity (tier 2) defined as “nonlife-threatening but with potential for future morbidity and mortality,” and high acuity (tier 3). In addition, each tier is further divided into subtype A (healthy patients) or subtype B (unhealthy patients) to better discriminate patient risk and hospital resource use. The ACS recommended proceeding with tier 3 surgery and postponing tier 1 and 2 surgery (or performing at an ambulatory surgery center) [4] .The joint statement published by SGS on April 28 applied the ACS–tiered system to numerous benign gynecologic procedures ( Fig. 1 ) [5] . This framework categorizes specific surgical procedures taking into consideration their indications and severity of disease symptoms. Two authors (W.A.B. and C.M.C.B.) assigned all patients into 1 of the 3 mESAS tiers by reviewing the patient's electronic medical record. These authors were queried about their awareness of the alternative MeNTS scoring system and, after finding them unfamiliar, were instructed to intentionally proceed in this manner to reduce potential bias toward another scoring system. Fig 1 SGS mESAS for benign gynecologic indications and surgeries. AMH, antimüllerian hormone; ASA, American Society of Anesthesiologists; ASC = ambulatory surgery center; AUB = abnormal uterine bleeding; CIN = cervical intraepithelial neoplasia; EIN = endometrial intraepithelial neoplasm; EMB = endometrial biopsy; GYN = gynecology; LARC = long-acting reversible contraception; mESAS = modified Elective Surgery Acuity Scale; MUS = midurethral sling; PMB = postmenopausal bleeding; SGS = Society for Gynecologic Surgeons; UTI = urinary tract infection; QoL = quality of life. Fig 1 SGS mESAS for benign gynecologic indications and surgeries. AMH, antimüllerian hormone; ASA, American Society of Anesthesiologists; ASC = ambulatory surgery center; AUB = abnormal uterine bleeding; CIN = cervical intraepithelial neoplasia; EIN = endometrial intraepithelial neoplasm; EMB = endometrial biopsy; GYN = gynecology; LARC = long-acting reversible contraception; mESAS = modified Elective Surgery Acuity Scale; MUS = midurethral sling; PMB = postmenopausal bleeding; SGS = Society for Gynecologic Surgeons; UTI = urinary tract infection; QoL = quality of life. The original MeNTS scoring criteria attempts to objectively prioritize surgeries by grading 21 factors within the broad categories of procedure variables, disease state, and comorbidities [6] . The cumulative score ranges between 21 and 105 and serves as a rank in priority, with lower numbers equating to greater priority ( Table 1 ). Higher scores equate to poorer perioperative outcomes, higher hospital resource use, an increased risk of COVID-19 transmission, and an increased ability to safely defer surgery. Table 1 Medically-Necessary Time-Sensitive (MeNTS) OR Procedure prioritization worksheet Table 1 1 2 3 4 5 Procedure OR time <30 min 31–60 min 61–120 min 121–180 min ≥181 min LOS anticipated Outpatient 23 h 24–48 h 2–3 d ≥4 d Post-Op ICU need Very unlikely 25% Bleeding risk/EBL 25% Surgical site None of the following Abdominopelvic MIS Surgery Abdominopelvic Open Surgery, Infraumbilical Abdominopelvic Open Surgery, Supraumbilical OHNS/Upper GI/Thoracic Disease Nonoperative treatment option effectiveness None available Available, <40% effective as surgery Available, 40–60% effective as surgery Available, 61–95% effective as surgery Available, equally effective Nonoperative treatment option resource use/ exposure risk Significantly worse/ not applicable Somewhat worse Equivalent Somewhat better Significantly Better Impact of 2-week delay in disease outcome Significantly worse Worse Moderately worse Slightly worse Minimally worse Impact of 2-week delay in surgical difficulty/risk Significantly worse Worse Moderately worse Slightly worse Minimally worse Impact of 6-week delay in disease outcome Significantly worse Worse Moderately worse Slightly worse Minimally worse Impact of 6-week delay in surgical difficulty/risk Significantly worse Worse Moderately worse Slightly worse Minimally worse Patient Age 65 yrs Lung disease (asthma, COPD, CF) None - - Minimal (rare inhaler) > Minimal OSA Not present - - Mild/moderate (no CPAP) On CPAP CV disease (HTN, CHF, CAD) None Minimal (no meds) Mild (1 med) Moderate (2 meds) Severe (≥3 meds) Diabetes None - Mild (no meds) Moderate (PO meds only) >Moderate (insulin) Immunocompromised No - - Moderate Severe ILI Sx's (fever, cough, sore throat, body aches, diarrhea) None (Asymptomatic) - - - Yes Exposure to COVID in past 14 d No Probably Not Possibly Probably Yes - CAD = coronary artery disease; CF = cystic fibrosis; CHF = congestive heart failure; COPD = chronic obstructive pulmonary disease; COVID = coronavirus disease; CPAP = continuous positive airway pressure; CV = cardiovascular; d = day; EBL = estimated blood loss; GI = gastrointestinal; HTN = hypertension; ICU = intensive care unit; ILI Sx = influenza-like-illness symptoms; LOS = length of stay; MIS = minimally-invasive surgery; OR = operating room; OSA = obstructive sleep apnea; OHNS = otolaryngology head and neck Surgery; PO = by mouth. Based on Prachand et al [6] . Medically-Necessary Time-Sensitive (MeNTS) OR Procedure prioritization worksheet CAD = coronary artery disease; CF = cystic fibrosis; CHF = congestive heart failure; COPD = chronic obstructive pulmonary disease; COVID = coronavirus disease; CPAP = continuous positive airway pressure; CV = cardiovascular; d = day; EBL = estimated blood loss; GI = gastrointestinal; HTN = hypertension; ICU = intensive care unit; ILI Sx = influenza-like-illness symptoms; LOS = length of stay; MIS = minimally-invasive surgery; OR = operating room; OSA = obstructive sleep apnea; OHNS = otolaryngology head and neck Surgery; PO = by mouth. Based on Prachand et al [6] . When attempting to apply the MeNTS model to our gynecologic patients, we believed adaptations could be made to improve clarity, objectivity, and validity. Specific adaptations made to the original MeNTS score included ( Table 2 ): (1) Modifying the “need for intubation” to 3 distinct planned anesthesia modalities (local/regional, monitored anesthesia care/conscious sedation, or general). This eliminated the surgeon estimating risk of intubation when no calculator exists for this. (2) Modifying surgical site to reflect gynecologic procedures and potential exposure of operating room staff to smoke plume with planned electrosurgery or laparoscopy. Despite a lack of evidence that infectious viral particles exist outside of the airway, we believed it prudent to incorporate risk of uncontained/unfiltered plume as societies began implementing warnings about the unknown risk of COVID-19 from abdominopelvic surgery, particularly with laparoscopy and electrosurgery (Society of American Gastrointestinal and Endoscopic Surgeons) [ 7 , 8 ]. At our institution, we have ultralow particulate air filters available for laparoscopy to mitigate this risk. (3) Instead of calculating “nonoperative treatment effectiveness percentage” and “exposure risk,” we transformed these 2 variables into 5 distinct categories pertinent to the gynecologic surgical patient including whether and how many alternative therapies have been tried, the presence/severity of pain, the presence/severity of anemia, the impact on desired immediate fertility, and the impact on adjacent genitourinary and gastrointestinal systems. (4) Removing the variables related to “two-week delay.” Because we could think of no examples within nonemergent benign gynecologic surgery in which a difference in morbidity or surgical difficulty exists between a 2-week and 6-week delay, we dropped this categorization. (5) Substituting the obstructive sleep apnea scale with an obesity scale, which included the binary question of the presence or absence of sleep apnea. Given the importance of obesity severity on postoperative outcomes, acknowledging this formally seemed prudent. In addition, given a large number of patients who are morbidly obese have undiagnosed sleep apnea and many diagnosed patients are noncompliant with therapy, we believed making obstructive sleep apnea a binary yes/no question was valid. (6) Removing the variable “influenza-like illness symptoms.” Any patient demonstrating these symptoms should continue to be delayed until resolution. (7) Limiting the options for “exposure to known COVID positive patient in the last 14 days” to improve reproducibility. At our institution, we implemented universal COVID-19 testing for all patients undergoing emergent and scheduled surgery. Table 2 Gynecologic Medically-Necessary Time-Sensitive (Gyn-MeNTS) Procedure prioritization worksheet Table 2 1 2 3 4 5 Procedure OR Time <30 min 31–60 min 61–120 min 121–180 min ≥181 min Estimated LOS Outpatient <23 h 24–48 h 2-3 d ≥ 4 d Post-Op ICU need Very Unlikely 25% Anticipated EBL ≤100 mL 101–250 mL 251–500 mL 501–750 mL ≥751 mL Surgical team size 1 2 3 4 ≥5 Anesthesia requirement Local/Regional - MAC/Conscious Sedation - General Surgical site Vaginal Surgery OR Hysteroscopy IF NO electrosurgery Hysteroscopy with electrocautery OR Abdominopelvic MIS (LSC) surgery Vaginal surgery IF electrosurgery Abdominopelvic open surgery, infraumbilical Abdominopelvic open surgery, supraumbilical Disease Trial of alternative therapy* Exhausted reasonable alternatives OR none exist Treatment requires frequent office visits 2 alternatives tried 1 alternative tried None Pain Severe (7–10), poorly controlled OR requiring ED visit Moderate to Severe (4–10), and minimally controlled, OR requiring frequent clinic visits Moderate (4–6), controlled Mild (1–3), controlled None Anemia Debilitating, Hgb 12 Impact on desired immediate fertility Significant/ Probable - Mild/ Possible - None Disease impact on GI/GU morbidity † Significant/ Progressive - Minimal/ Stable - None Impact of 6-week delay in disease outcome Significantly worse Worse Moderately worse Slightly worse Minimally worse Impact of 6-week delay in surgical difficulty/risk Significantly worse Worse Moderately worse Slightly worse Minimally worse Patient Age ≤20 yrs 21–40 yrs 41–50 yrs 51–65 yrs ≥66 yrs Lung disease (asthma, COPD, CF, smoking) None - - Minimal OR Active smoking > Minimal Obesity/ OSA Normal BMI (BMI <25) Overweight (BMI 25–29.9) Class 1 Obesity (BMI 30–34.9) Class 2 Obesity (BMI 35–39.9) Class 3 Obesity (BMI ≥40) or OSA CV disease (HTN, CHF, CAD) None Minimal (no meds) Mild (1 med) Moderate (2 meds) Severe (≥ 3 meds) Diabetes None - Mild (no meds) Moderate (PO meds only) >Moderate (insulin) OR poorly controlled Immunocompromised No - - Moderate Severe Exposure to COVID in past 14 days No - Possibly - Yes BMI = body mass index; CAD = coronary artery disease; CF = cystic fibrosis; CHF = congestive heart failure; COVID = coronavirus disease; COPD = chronic obstructive pulmonary disease; CV = cardiovascular; ED = emergency department; EBL = estimated blood loss; GI = gastrointestinal; GU = genitourinary; Hgb = hemoglobin; HTN = hypertension; IV = intravenous; ICU = intensive care unit; LOS = length of stay; LSC = laparoscopy; MAC = monitored anesthesia care; MIS = minimally invasive surgery; OR = operating room; OSA = obstructive sleep apnea; PO = by mouth. *Trial of alternative therapy examples: expectant management, OCPs, POPs, NSAIDs, TXA, LARCs, GnRH analogues, pelvic floor physical therapy, nerve blocks, pessary, UAE. † Examples include: hydroureter or hydronephrosis, urinary retention, bowel lumen narrowing. Modifying the “need for intubation” to 3 distinct planned anesthesia modalities (local/regional, monitored anesthesia care/conscious sedation, or general). This eliminated the surgeon estimating risk of intubation when no calculator exists for this. Modifying surgical site to reflect gynecologic procedures and potential exposure of operating room staff to smoke plume with planned electrosurgery or laparoscopy. Despite a lack of evidence that infectious viral particles exist outside of the airway, we believed it prudent to incorporate risk of uncontained/unfiltered plume as societies began implementing warnings about the unknown risk of COVID-19 from abdominopelvic surgery, particularly with laparoscopy and electrosurgery (Society of American Gastrointestinal and Endoscopic Surgeons) [ 7 , 8 ]. At our institution, we have ultralow particulate air filters available for laparoscopy to mitigate this risk. Instead of calculating “nonoperative treatment effectiveness percentage” and “exposure risk,” we transformed these 2 variables into 5 distinct categories pertinent to the gynecologic surgical patient including whether and how many alternative therapies have been tried, the presence/severity of pain, the presence/severity of anemia, the impact on desired immediate fertility, and the impact on adjacent genitourinary and gastrointestinal systems. Removing the variables related to “two-week delay.” Because we could think of no examples within nonemergent benign gynecologic surgery in which a difference in morbidity or surgical difficulty exists between a 2-week and 6-week delay, we dropped this categorization. Substituting the obstructive sleep apnea scale with an obesity scale, which included the binary question of the presence or absence of sleep apnea. Given the importance of obesity severity on postoperative outcomes, acknowledging this formally seemed prudent. In addition, given a large number of patients who are morbidly obese have undiagnosed sleep apnea and many diagnosed patients are noncompliant with therapy, we believed making obstructive sleep apnea a binary yes/no question was valid. Removing the variable “influenza-like illness symptoms.” Any patient demonstrating these symptoms should continue to be delayed until resolution. Limiting the options for “exposure to known COVID positive patient in the last 14 days” to improve reproducibility. At our institution, we implemented universal COVID-19 testing for all patients undergoing emergent and scheduled surgery. Gynecologic Medically-Necessary Time-Sensitive (Gyn-MeNTS) Procedure prioritization worksheet BMI = body mass index; CAD = coronary artery disease; CF = cystic fibrosis; CHF = congestive heart failure; COVID = coronavirus disease; COPD = chronic obstructive pulmonary disease; CV = cardiovascular; ED = emergency department; EBL = estimated blood loss; GI = gastrointestinal; GU = genitourinary; Hgb = hemoglobin; HTN = hypertension; IV = intravenous; ICU = intensive care unit; LOS = length of stay; LSC = laparoscopy; MAC = monitored anesthesia care; MIS = minimally invasive surgery; OR = operating room; OSA = obstructive sleep apnea; PO = by mouth. *Trial of alternative therapy examples: expectant management, OCPs, POPs, NSAIDs, TXA, LARCs, GnRH analogues, pelvic floor physical therapy, nerve blocks, pessary, UAE. Examples include: hydroureter or hydronephrosis, urinary retention, bowel lumen narrowing. Three authors (C.Q.M., J.S.K., C.Z.W.) adapted and applied the modified Gyn-MeNTS scoring system to all patients. All 5 authors were blinded to each other's scores to reduce bias. The authors could not be blinded from the urgent surgeon self-prioritization (SSP) because these surgeries were performed as scheduled and could be elicited from chart review. Gyn-MeNTS scores were calculated by 3 reviewers and analyzed as continuous variables with possible scores ranging from 21 to 105. The lower the score, the more prioritized the surgery would be. The mESAS score was calculated by 2 different reviewers and analyzed as a 3-level ordinal variable (1/2/3) with a higher score indicating more urgency/prioritization. Finally, the SSP score was made into a binary variable with those considered highest priority (urgent SSP) separated from those considered lower priority. The 3 Gyn-MeNTS raters’ scores were examined using Spearman r to determine the level of monotonic association and paired t tests to determine whether there were systematic differences between raters. A relevant systematic difference was indicated by a mean difference >0.1 along with a significant difference on the paired t test. If all Spearman r ’s were >0.80, it indicated a strong interrater reliability. To determine which Gyn-MeNTS items had the worst reliability, agreement between raters’ Gyn-MeNTS item scores were examined using percent exact agreement, rather than κ, because the raters used different numbers of categories on several items. We took the mean of the Gyn-MeNTS scores and examined the Spearman r of this score with the mean of the mESAS scores and with urgent SSP as measured by actual scheduling (a binary variable, yes/no). Concurrent validity for the Gyn-MeNTS score was indicated by strong Spearman r with both the mESAS score and urgent SSP. We also examined whether urgent SSP could be predicted independently using both the Gyn-MeNTS score and the mESAS rating in a multivariable logistic regression model. If both were significant independent predictors, we then used the log-linear equation produced by the regression model to calculate each patient's probability of being classified as urgent SSP and examined the association of probability quartile with urgent SSP status using chi-square. We examined the distribution of mESAS rating with the Gyn-MeNTS scores achieving the highest 67%, 75%, 80%, 85%, 90%, and 95% of urgency levels using chi-square. SAS (version 9.4, SAS Institute, Cary, NC) was used for data analysis, with p <.05 considered significant.

Despite finding overall high interrater reproducibility in the Gyn-MeNTS scoring system (Spearman r 0.82–0.84), it does not seem that this scoring system strongly discriminates the most urgent cases as determined by either the mESAS system (Spearman r 0.31) or when surgeons proceed with using their instinct alone, the SSP (Spearman r 0.46). The interrater reproducibility of the mESAS–tiered system was moderate (weighted κ 0.57), and it appears to perform slightly better in discerning how surgeons instinctively prioritize (Spearman r 0.53). The mESAS system identified 92% of the patients classified as most urgent SSP, whereas the Gyn-MeNTS found only 67%. However, when used together, the 2 scoring systems had high discrimination in capturing clinicians’ instinctive beliefs about urgency and each contributed independently, suggesting that (a) they capture distinct issues related to urgency and (b) their combined use may provide the optimal system for objectively rating surgical urgency. Despite seeing merit in the original scoring system, we believed that additional steps could be taken to make the MeNTS model more objective, yielding higher interobserver reliability. Our goal was to create a modified gynecologic MeNTS that would still score in comparable ranges with the original in the event our institution later decided to prioritize by this route. Most modifications were made within the disease factors category. We believed that the category should be given more weight of importance (with more scored items) and tailored to the disease burden that gynecologic patients uniquely incur. Thus, we expanded the number of scoring categories and created objective criteria for quantifying pain, anemia, fertility impact, and impact on adjacent organ systems such as the genitourinary and gastrointestinal systems. Although we agree that exposure to known COVID-19 should be considered, we recommend that, if available, all patients undergoing scheduled surgery be tested for COVID-19 within 48 hours of their planned surgery. If a patient's result is positive, the surgery should be delayed. If testing is unavailable, we recommend screening all patients undergoing elective surgery for influenza-like symptoms, and the surgery should be postponed if present. Given the reports of significantly worsened morbidity and mortality when surgery is unwittingly performed in patients who are presymptomatic for COVID-19, all attempts to identify these patients should be made [ 9 , 10 ]. In addition, universal COVID-19 testing protects healthcare workers from unnecessary exposure, as well as creates a binary personal protective equipment triaging system to “standard precautions” or “transmission-based precautions” to protect scarce resources. Despite our attempts to make the scoring system as objective as possible, many categories are open to wider interpretation than it may first seem. We recommend, before implementing the scoring system that an initial dialogue to “lay the ground rules” occurs to improve reviewer reproducibility and thus reliability across the cohort. For example: (1) Lung Disease. Attempt to define what constitutes “minimal” disease. Our system did not capture smoking history other than a binary yes/no. Quantifying risk of disease in a pack/year history calculation may prove beneficial as smoking affects risks of both respiratory disease and wound healing with a direct impact on outcomes. It is also becoming increasingly known that lung disease, such as chronic obstructive pulmonary disease) and asthma, can worsen patient outcomes in the setting of COVID-19, independent of their usual perioperative risks [11] . (2) Surgical team size. We agreed that team size would be calculated by the minimum number of surgeons needed to perform a procedure safely and efficiently. In an academic teaching institution, the size of the surgical teams can easily be twice the actual number needed. Thus, all hysteroscopies (that did not require intraoperative sonography) were scored a 1, and almost all laparoscopies a 2. When constant uterine manipulation was needed, a score of 3 was given to laparoscopy. (3) Operating room time. At our institution, we are asked to provide estimates of “wheels-in” to “wheels-out” rather than “incision-to-closure” time when posting cases. As long as consistency is applied across graders, this variable has the potential to be reliable. Given surgeon notoriety around being poor predictors of needed surgical time, it is not surprising that this variable performed among the worst in interrater agreement. Lung Disease. Attempt to define what constitutes “minimal” disease. Our system did not capture smoking history other than a binary yes/no. Quantifying risk of disease in a pack/year history calculation may prove beneficial as smoking affects risks of both respiratory disease and wound healing with a direct impact on outcomes. It is also becoming increasingly known that lung disease, such as chronic obstructive pulmonary disease) and asthma, can worsen patient outcomes in the setting of COVID-19, independent of their usual perioperative risks [11] . Surgical team size. We agreed that team size would be calculated by the minimum number of surgeons needed to perform a procedure safely and efficiently. In an academic teaching institution, the size of the surgical teams can easily be twice the actual number needed. Thus, all hysteroscopies (that did not require intraoperative sonography) were scored a 1, and almost all laparoscopies a 2. When constant uterine manipulation was needed, a score of 3 was given to laparoscopy. Operating room time. At our institution, we are asked to provide estimates of “wheels-in” to “wheels-out” rather than “incision-to-closure” time when posting cases. As long as consistency is applied across graders, this variable has the potential to be reliable. Given surgeon notoriety around being poor predictors of needed surgical time, it is not surprising that this variable performed among the worst in interrater agreement. (1) Pain. Despite our attempt to objectively define pain using a visual analog scale system and incorporating the ability to control this pain on the basis of frequency of office or emergency room visits, our interrater agreement remained fair (agreement = 57%) ( Table 3 ). This was due, in large part, to provider differences in the documentation of pain, its impact on quality of life, and the ability to control pain with medical management. (2) Alternative therapy. We created objective parameters for measuring alternative therapy that included an extensive list of examples (Table 2 footnotes) and accounted for number of therapies trialed and burden of resource utilization/exposure risk. Despite this, our interrater agreement was poor (agreement = 34%). We believe this was due to the difficulty in abstracting this information from charts. (3) Prediction of perioperative transfusion. We agree that the presence of anemia can be inconsequential when it comes to procedures with a risk of low blood loss. The counter is true as well; high blood loss procedures can be tolerated when the patient has no baseline anemia. The most important question, particularly during a time of blood shortage, is whether and to what degree the patient will need perioperative blood products. Finding an objective way to quantify this risk was difficult, a problem compounded by the surgeons’ inability to reliably predict estimates of blood loss (agreement = 73%). (4) Cardiovascular disease. While simplifying the determination of heart disease to represent the number of medications it takes to control it, care should be taken to avoid this simple assumption. Patients with untreated hypertension may carry significantly more heart disease than patients who are well controlled on 3 medications owing simply to their access to healthcare and their compliance with recommended therapy. (5) Emphasis on hospital resource use with a bias toward the young, healthy patient. Perhaps the biggest limitation to the Gyn-MeNTS scoring system is its favor toward quick procedures on patients who are young and healthy. Despite our attempts to increase the weight of disease burden by adding more graded variables, patients who are young and healthy getting quick, elective procedures (e.g., tubal ligation or polypectomy) were consistently prioritized in this grading system. Although not an invalid conclusion when a hospital system is severely limited in its capacity to do anything but the most quick and simple of procedures on patients who are healthy, it would be difficult to justify, for example, elective sterilization over treatment of debilitating pain from endometriosis, especially if patients have access to alternative contraception. In our institution, despite receiving some of the lowest Gyn-MeNTS scores, patients requesting sterilization are not being prioritized at this time. (6) We also must acknowledge that many of our changes to the original MeNTS scoring system could be specific to the study institution, which thus affects the generalizability of the Gyn-MeNTS system. Pain. Despite our attempt to objectively define pain using a visual analog scale system and incorporating the ability to control this pain on the basis of frequency of office or emergency room visits, our interrater agreement remained fair (agreement = 57%) ( Table 3 ). This was due, in large part, to provider differences in the documentation of pain, its impact on quality of life, and the ability to control pain with medical management. Alternative therapy. We created objective parameters for measuring alternative therapy that included an extensive list of examples (Table 2 footnotes) and accounted for number of therapies trialed and burden of resource utilization/exposure risk. Despite this, our interrater agreement was poor (agreement = 34%). We believe this was due to the difficulty in abstracting this information from charts. Prediction of perioperative transfusion. We agree that the presence of anemia can be inconsequential when it comes to procedures with a risk of low blood loss. The counter is true as well; high blood loss procedures can be tolerated when the patient has no baseline anemia. The most important question, particularly during a time of blood shortage, is whether and to what degree the patient will need perioperative blood products. Finding an objective way to quantify this risk was difficult, a problem compounded by the surgeons’ inability to reliably predict estimates of blood loss (agreement = 73%). Cardiovascular disease. While simplifying the determination of heart disease to represent the number of medications it takes to control it, care should be taken to avoid this simple assumption. Patients with untreated hypertension may carry significantly more heart disease than patients who are well controlled on 3 medications owing simply to their access to healthcare and their compliance with recommended therapy. Emphasis on hospital resource use with a bias toward the young, healthy patient. Perhaps the biggest limitation to the Gyn-MeNTS scoring system is its favor toward quick procedures on patients who are young and healthy. Despite our attempts to increase the weight of disease burden by adding more graded variables, patients who are young and healthy getting quick, elective procedures (e.g., tubal ligation or polypectomy) were consistently prioritized in this grading system. Although not an invalid conclusion when a hospital system is severely limited in its capacity to do anything but the most quick and simple of procedures on patients who are healthy, it would be difficult to justify, for example, elective sterilization over treatment of debilitating pain from endometriosis, especially if patients have access to alternative contraception. In our institution, despite receiving some of the lowest Gyn-MeNTS scores, patients requesting sterilization are not being prioritized at this time. We also must acknowledge that many of our changes to the original MeNTS scoring system could be specific to the study institution, which thus affects the generalizability of the Gyn-MeNTS system. Despite the extensive list of surgical examples given by major gynecologic surgical societies in the mESAS table, we found inconsistencies with application between reviewers. It is clear there is still room for interpretation of disease severity, and, thus acuity, making assignment of tiers prone to significant variation. In addition, at the end of this exercise, a high-volume institution could still have large numbers of patients within each cohort that must be prioritized further, and this system provides no guidance as to how to perform this. Finally, the mESAS system appears to contain some inconsistencies within their surgical examples. For instance, endometriosis with poorly controlled pain and desire for fertility is categorized as a level 2, whereas myomectomy for a patient who is asymptomatic and is experiencing infertility is categorized as a level 3. A hysteroscopic polypectomy in the patient who is infertile is categorized more urgently (level 3) than a hysteroscopic evaluation or polypectomy in patients older than 50 years with an inability to sample in the office (level 2) that have higher risks of malignant potential. Although these surgical assignments have been agreed on by major societal stakeholders, these discrepancies deserve attention. To a certain extent, both scoring systems depend on accurate and elaborate chart documentation. Ideally, surgeons would grade their own patients to improve accuracy and overcome this obstacle. Difficulty was encountered when reviewers graded each other's patients in categories such as efficacy of alternative therapies, immediacy of fertility desire, and severity of pain. This clearly affected the interrater reliability and could lead to the false assumption that poor interrater reliability means that a scoring system is invalid. Finally, further evaluation in a setting with clinicians who did not develop the scoring system is warranted. To our knowledge, this is the first study to assess reliability and validity of previously published surgical scoring systems. In addition, we are the first to report application of these scoring systems in gynecologic patients and have made recommendations for implementation in this arena. More robust prospective data are needed to either confirm or refute our retrospective findings. Further study should also evaluate the efficacy of using both the Gyn-MeNTS and mESAS systems together to triage nonemergent procedures. We also believe strongly that a system that provides more emphasis on the disease variables of the Gyn-MeNTS scoring system would yield an even more valid triaging system.