Improving the Resident Interview Process with Structural and Statistical Bias Correction.

Improving the Resident Interview Process with Structural and Statistical Bias Correction. | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Improving the Resident Interview Process with Structural and Statistical Bias Correction. Sultan Abdelhamid, Zaid Haddadin, Molly Casey, Emery Cuellar, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7842934/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 5 You are reading this latest preprint version Abstract Introduction : Biases in the general surgery interview selection process may exclude promising students from a residency program. There are biases inherent to both the evaluator and the structure of the interview process. Several studies have questioned the reliability of the interview process. We aimed to improve the resident interview structure through structural improvements in the interview and increasing the validity of the selection methodology. Methods: We performed a prospective comparative analysis of our general surgery residency interview process over two consecutive academic years (AY 2022 and 2023). We used descriptive statistics and a mixed-effects ordered logit model to measure bias and guide process improvement. Results: Eighty total students were interviewed. We found numerous statistical biases, including leniency bias exacerbated by the unequal distribution of evaluators, significant differences in the variance from evaluator’s scoring (p < 0.0001), and non-normal distribution (p < 0.001). Rater reliability was “good” in both years: 0.69 (C.I. 0.51–0.82). Using uncorrected means, on average, each student was statistically different from 11.5 ± 1.1 other students. Using our model, we improved the number of statistically distinct groupings as each student now differed by 30 ± 1.0 others (p < 0.0001). In AY23, we restructured the interviews so that all the evaluators scored every student, which significantly improved the interview accuracy using the same mixed model: R-squared of 0.95 versus 0.70, and a smaller percent of students had a change in their rank using the improved structure (57% versus 90%, p < 0.05), compared to AY22. Conclusions: Among a pool of students applying for general surgery residency at a single institution, our study shows that using the uncorrected evaluators’ impressions results in a minimal distinction between any of the candidates except at the extremes of the score range, statistically demonstrating that the interview, in its raw form, cannot be a valuable tool in the resident selection. Due to numerous statistical biases, there is little differentiation between students and thus little validity in scoring students using raw mean scores. This can be overcome by developing a structured interview and correcting for statistical biases. We suggest that interviews be independently performed (not in a group), and evaluators should be blinded to others’ impressions. Figures Figure 1 Figure 2 Introduction Residency selection relies heavily on application metrics and interviews, but the latter remains a subjective and error-prone process. Improving interview reliability and fairness is essential to support equitable access and optimal trainee selection. Biases in the selection process may exclude promising students from a residency program. There are biases inherent to both the evaluator and the structure of the interview process. Statistical bias is an intrinsic tendency that produces differences between results and actual facts. Examples of evaluator bias include stereotyping, halo effect, leniency, central tendency, and conformity bias ( 2 ). The structure can be biased with group interviews, group discussions of candidates before scoring, or some evaluators not present for every interview. We undertook this study measuring the biases in our interview process and explored statistical and structural methods to overcome these biases to achieve a reflective and fair interview process. Methods We received applications through the Electronic Residency Application Services of the Association of American Medical Colleges. Applications were reviewed by the program director and core faculty. Reviewers had access to full ERAS data, including transcripts, test scores, personal statements, and recommendation letters. Decisions for interview invitations were made in a non blinded, consensus based process. Interview scores were used as the primary ranking tool post interview In AY 22, our interview schedule consisted of four interview days, and each student met individually (virtual meetings) with six evaluators for 15 minutes each. There was a pool of 8 evaluators to accommodate scheduling conflicts to ensure each student had six interviews. Each evaluator was instructed to rate (score) each student on a 1 to 10 scale (1 being the worst and 10 being the best). Each evaluator had access to the student’s complete application before the interview, including medical or osteopathic school transcripts, United States Medical Licensing Examination (USMLE) scores (step 1, 2), letters of recommendation, personal statement, and curriculum vitae. The interviews were free-form, and there were no specific scoring instructions. After the interviews, the evaluators discussed all the students in the groups before the scores were submitted. USMLE 1 scores and the “prestige” of a medical school were hypothesized to be a potential source of evaluator bias, and we incorporated these variables in our initial modeling. School prestige was a continuous variable defined as the ranking of medical schools by the U.S. News and World Report 2021. A score of 1 was the highest, and 100 was the lowest possible. USMLE 1 was a continuous variable (211 to 264 in our sample). Candidates were given anonymous IDs each year of 1–40. We used a multilevel mixed-effects ordered logistic regression with an unstructured covariance matrix and robust error correction with maximal likelihood estimation to analyze the interview score. The aim was to isolate evaluator-specific variance and estimate adjusted candidate rankings while accounting for missing data and scoring tendencies. Adjusted scores were then standardized for comparison and ranking purposes. We used the student as a random effect. The fit of non-nested models was examined by comparing their Akaike information criterion (AIC). Generalized linear latency was used to measure significant differences in variance between the interviews. We used the coefficient for each candidate from our mixed model to create an adjusted score. We estimated the number of statistically distinct categories by counting the non-overlapping confidence intervals. In AY 23, we attempted to minimize some of the statistical biases we identified in AY 22. In the second part of our study, we mandated a balanced set of four evaluators (every evaluator interviewed every student) and developed a set of consistent, structured interview questions. The scoring sheet now listed standardized interpretations of each score (i.e., 10 is the best student ever seen. 5 is a typical average student) to improve distinctions in scores and decrease clustering at the high scores range. The interviewers did not discuss applicants until after submitting the scoring to prevent post-interview biases. We then repeated the above modeling. The Interrater Reliability (IRR) was calculated in AY22 using a one-way random effects model between average measurements. In AY23, as we had the same raters for all students, we used a two-way mixed effects model. Shapiro Wilk test was used to test normality. IRB exempt status was obtained. All calculations were performed using Stata 17.0 (College Station, TX). All candidate scores were de-identified prior to analysis. The anonymization and data linkage were performed by the program director, and the first author only received coded data. No resident had access to identifiable interview results of their peers. Data archiving is not mandated but data will be made available on reasonable request. Results Table 1 compares the interview structure between AY22 and AY23. Both distributions of scores between AY22 and AY23 were right censored (that is, there was clustering of scores towards the high end) and skewed (median and mean scores differed). Thus, the data is not normally distributed, invalidating many statistical tools (ANOVA, t-tests, linear regression). Our attempts to improve normality were unsuccessful in AY23 as scores still demonstrated clustered in the higher registry to a similar degree. Table 1 Characteristic of the General Surgery Interview. AY22 AY23 Applicant Pool 2300 1794 Applicants offered interviews 43 45 Applicants declined interview 3 5 Applicants Interviewed 40 40 Size of Evaluator Pool 8 4 Median # Interviews (range) 6 ( 4 – 6 ) 4 Structured Interview No Yes Scoring Guide No Yes Range of interviews performed by each evaluator 10–40 (mean 30) 40 Average score 7.14 ± 2.3(C.I. 6.84–7.43) 6.84 ± 1.9 (C.I. 6.6–7.1) Median Score 7.0 7.0 Normal Distribution No (p < 0.0001) No (p = 0.01) When examining the mean raw scores, there was minimal differentiation between applicants for either of the academic years due to the wide statistical error (Fig. 1 ). While each student differed from 11 ± 1.1 other students, that difference varied depending on the score. Students' raw scores were most different at the extremes compared to the large indistinguishable middle group. In fact, there were only three groups of students, with most in the middle group. This poor differentiation in raw scores was no different between AY22 and AY23, even with the institution of balanced and structured interviews. Figure 1 . The interrater reliability (IRR) was at 0.69 (C.I. 0.50–0.82), suggesting good consistency between the judges. Correcting using Mixed Effects Model For AY22, examining our mixed effects logit model, two evaluators scored significantly more leniently than the others; one only interviewed ten candidates. Only two evaluators interviewed all candidates (Table 2 ). All the evaluators had significantly different variances (p < 0.0001) in their scoring. Table 2 Results from Mixed Ordered Logistic Model Adjusted Parameters AY22 AY23 Coefficient ± S.E. P-value Coefficient ± S.E. P-value Gender of Candidate (M v F) 0.56 ± 0.56 0.99 -0.69 0.13 USMLE1 -0.014 0.51 0.002 0.90 Evaluator as independent predictor of score (#of Interviews AY22,AY23) Evaluator 1 (10,40) 1 1 Evaluator 2 (30,40) 0.51 ± 0.59 0.38 -0.21 ± 0.36 0.56 Evaluator 3 (40) 1.18 ± 0.52 0.03 0.15 ± 0.38 0.69 Evaluator 4 (14,40) 2.28 ± 0.80 0.005 -0.41 ± 0.47 0.38 Evaluator 5 (40,40) -0.09 ± 0.53 0.87 Evaluator 6 (30) -0.66 ± 0.88 0.45 Evaluator 7 (39) -0.08 ± 0.56 0.89 Evaluator 8 (35,40) -0.49 ± 0.58 0.405 Lenient vs total evaluators 2/8 0/4 Variance Between Evaluators YES P < 0.0001 NO P = 0.57 Identifying the leniency and unbalanced interviews as a source of bias, we altered our interview structure as described in Methods. Unlike in AY22, there was no statistical difference in the measurement variance between any evaluators, and no evaluator was statistically harsher (or more lenient) than the others. In both years, our mixed model substantially improved the differentiation of students by decreasing error and controlling for interviewer bias. Prior to modeling, each student's score was statistically distinct from 11.1 +-1.1 other candidates on average. After adjustment, this increased to 30.1+-1.0, reflecting improved discrimination between candidates. While there was still more differentiation at the extremes of score, this was much less pronounced than the raw mean scores (Fig. 2 ). When we ranked the scores, our model substantially altered the rank order. Comparing student ranking using the raw and corrected interview score in AY22, corrected scores varied between − 8 to + 10 positions, with 90% (36/40 ) of candidates having a change in their rank order after adjustment. Interestingly, the difference was less pronounced in AY23, with ranking only varying between − 3 to + 3 positions, with 57% (23/40) of candidates having a change in their rank order after adjustment. Gender, School prestige, and USMLE1 score were not significant predictors of interview scores. The R-squared for the AY22 mixed model was 0.70, and the R-squared for the AY23 mixed model was 0.95. This suggests that the process improvement we undertook based on the analysis of AY22 resulted in a substantially more valid scoring for our students. Discussion Among a pool of students applying for general surgery residency at a single institution, our study shows that using the uncorrected evaluators’ impressions results in a minimal distinction between any of the candidates except at the extremes of the score range, statistically demonstrating that the interview, in its raw form, cannot be a valuable tool in the resident selection. If the evaluators in a rating system cannot reliably distinguish subjects, there is no value in the process (except, perhaps, to attract candidates or eliminate far outliers). Using a statistical model, we were able to distinguish candidates much better. The shortcomings of our initial interview process became apparent to us only after analysis. Substantial sources of scoring error occurred because some evaluators had intrinsically harsher or lenient scoring systems, not all evaluators evaluated all candidates, each evaluator had a different variance in their scoring, the scoring was clustered, and there were differences in the number of interviews per student (Table 3 ). Table 3 Bias encountered Sources of Statistical Bias Encountered - Non-Normality - Skewed (Significant deviation from Median and Mean) - Clustering (grouping at highest scores) - Heteroskedasticity (lower scores had greater discord between evaluators) Unbalanced Data - Unbalanced Data - Not every evaluator interviewed every student - Each evaluator had their own variances in score - Some evaluators score significantly more harsh or lenient than others - Not all students had the same number of interviews Repeated measures - Standard deviation around mean - Difficult to maintain consistency over prolonged interview season - Interviewer fatigue results in changes in scoring over the day Understanding how bias is introduced in an interview is essential to appropriately utilizing this costly, time-intensive, and anxiety-producing process ( 2 ). While non-normality and clustering are not biases in a traditional epidemiological sense, they present challenges when using raw means to differentiate similarly scoring candidates. Our use of statistical bias here refers to measurement errors that reduce score discrimination and violate assumptions of many common parametric statistical tools. We showed that one substantial factor in our processes’ structure was due to unbalanced measurements. This is where the students have a differing number of interviews, or, more commonly, not every evaluator rates every student. In the former case, differences in measurement variance would result. In the latter case, evaluator biases will not be evenly distributed amongst the entire group of students. This is especially problematic with leniency bias. Leniency (or harshness) bias will unfairly elevate or depress a student’s rating. Statistical modeling using mixed effects models is ideal for controlling individual evaluator variances. It can partially correct for other sources of statistical error (as they account for random effect and missing data.) Mixed models separate the error (variance) between the independent variables (each interviewer) and the random effect (each student). By controlling for each interviewer (who each has their tendencies for scoring), we can better measure distinguishing factors from the student. We also observed the right censoring (or clustering of scores at the high end), which we suspect is common in most interviews. We attempted to improve this clustering in AY23 by defining each score (1–10) with guides, although this was only slightly successful. How a program weighs any part of the student application, including the interview, in developing a resident rank order list should be commensurate with its accuracy in predicting a successful resident. If a program does not measure the IRR of the evaluators, then the interview might be unduly weighted and generate unreliable results. Interviews in medical education and other employment interviews have previously been shown not to be predictive of achievement ( 3 ). A striking example occurred at the University of Texas Medical School at Houston. In 1987, Texas state legislators realized that the state was short on physicians. To fix the problem, the legislature required the school to increase the class size from 150 to 200 after the admissions committee had already chosen its preferred 150 students. The 50 students brought into the class had previously received the lowest ranking from the admissions committee. The performance at graduation of initially accepted and initially rejected students was the same ( 4 ). We have learned from this study how to design the interview process in our institution better. We suggest that interviews be independently performed (not in a group), and evaluators should be blinded to others’ impressions. All applicants should have an equal number of interviews and the same evaluators. Additionally, evaluators should undergo pre-interview training to self-identify types of bias and standardize the scoring system. Design alone is not enough to overcome the substantial lack of differentiation between candidates; thoughtful statistical modeling was also needed. An interview process might be poor at differentiating candidates because the candidates performed equally well in an interview session or the interviews were poorly designed to measure important differences. Despite the statistical errors that seemingly lead to arbitrary ranking decisions, the current rank system works well from the standpoint of most programs. This might be because the students we select are already highly functioning, educated, motivated, and have already overcome substantial educational barriers. We perceive our current selection process to be excellent because any random group of applicants would likely be excellent. Perhaps there is little additional valuable information that can be obtained from an interview process that is predictive of resident success, and the focus should be on truthfully marketing a program’s attributes to better match the student’s needs. The most important limitation in this study should be highlighted: We (nor any others to our knowledge) know what predictive attributes predict who will be an exceptional resident. Those personality characteristics are much more challenging to measure than medical school grades or test scores ( 5 ). This is also a small study from a single institution, but our interview process is, in essence, like most programs, and thus, the conclusions are generalizable. Conclusion A typical residency interview structure is statistically biased and unable to easily differentiate between students. Through process improvement, guided by statistical modeling, we managed many of these biases and substantively improved the validity of the interview process. As the career stakes for each applicant are high and with every program aiming to recruit the best applicant, designing a fair system should be a priority. References NRMP, 2022, Advanced Data Report. https://www.nrmp.org/wp-content/uploads/2022/03/Advance-Data-Tables-2022-FINAL.pdf Laszlo Kiraly, Elizabeth Dewey, Karen Brasel, Hawks and Doves: Adjusting for Bias in Residency Interview Scoring, Journal of Surgical Education, Volume 77, Issue 6, 2020, Pages e132-e137, McDaniel, M. A., Whetzel, D. L., Schmidt, F. L., & Maurer, S. D. (1994). The validity of employment interviews: A comprehensive review and meta-analysis. Journal of Applied Psychology , 79 (4), 599–616. Smith, S R. Medical school and residency performances of students admitted with and without an admission interview. Academic Medicine 66(8):p 474–6, August 1991. Stephenson-Famy A, Houmard BS, Oberoi S, Manyak A, Chiang S, Kim S. Use of the Interview in Resident Candidate Selection: A Review of the Literature. J Grad Med Educ. 2015;7(4):539–48 Cite Share Download PDF Status: Under Review Version 1 posted Reviewers agreed at journal 11 Nov, 2025 Reviewers invited by journal 11 Nov, 2025 Editor invited by journal 10 Nov, 2025 Editor assigned by journal 10 Nov, 2025 First submitted to journal 12 Oct, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7842934","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":543485618,"identity":"aad82cf7-1081-40e4-a6cd-ac4ac7fa38fd","order_by":0,"name":"Sultan Abdelhamid","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA8ElEQVRIiWNgGAWjYJCCAx94bJgZmA8+gAkYENLBeHCGTBozA1syXCVBLcyHeWwOMxCvRb7/dMIBnpzz7PxszIyPK/7Y5DGwN2+TwKfF4EbuhgMSZ24zS7YxMxuebUsrZuA5VoZfiwTvhgOGPbeZDe73H5NsbDic2CCRY4ZXi3z/2Q0HEv+dY7Y/xsz+s+EPUIv8G/xaGA4AHXaA5wCzARszG2MDG8gWHvxaQH452MCTzCxxjJlZsrEtLbGNJ63YgoDDNn/+w2OXzN/GzPix4Y9NYj/74Y038DoMCpLhLDZilIOAHbEKR8EoGAWjYAQCAF4ZSqj8YmjkAAAAAElFTkSuQmCC","orcid":"https://orcid.org/0000-0001-8395-2637","institution":"Jefferson Einstein Philadelphia Hospital","correspondingAuthor":true,"prefix":"","firstName":"Sultan","middleName":"","lastName":"Abdelhamid","suffix":""},{"id":543485619,"identity":"8c9cf57a-c5a1-421a-987c-28fbb45ffa99","order_by":1,"name":"Zaid Haddadin","email":"","orcid":"","institution":"Jefferson Einstein Philadelphia Hospital","correspondingAuthor":false,"prefix":"","firstName":"Zaid","middleName":"","lastName":"Haddadin","suffix":""},{"id":543485620,"identity":"94918c58-49ca-4bea-a044-a81f588d1733","order_by":2,"name":"Molly Casey","email":"","orcid":"","institution":"Jefferson Einstein Philadelphia Hospital","correspondingAuthor":false,"prefix":"","firstName":"Molly","middleName":"","lastName":"Casey","suffix":""},{"id":543485621,"identity":"def84d22-5238-4298-bcbc-113c8a3d9394","order_by":3,"name":"Emery Cuellar","email":"","orcid":"","institution":"Jefferson Einstein Philadelphia Hospital","correspondingAuthor":false,"prefix":"","firstName":"Emery","middleName":"","lastName":"Cuellar","suffix":""},{"id":543485622,"identity":"d13068c7-0401-4ed2-a6ae-fc08a1b12c17","order_by":4,"name":"Moshumi Godbole","email":"","orcid":"","institution":"Jefferson Einstein Philadelphia Hospital","correspondingAuthor":false,"prefix":"","firstName":"Moshumi","middleName":"","lastName":"Godbole","suffix":""},{"id":543485623,"identity":"d9e618f6-295a-403a-aeed-72f13a54e492","order_by":5,"name":"Ramsey Dallal","email":"","orcid":"","institution":"Jefferson Einstein Philadelphia Hospital","correspondingAuthor":false,"prefix":"","firstName":"Ramsey","middleName":"","lastName":"Dallal","suffix":""}],"badges":[],"createdAt":"2025-10-12 20:58:47","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7842934/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7842934/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":96556264,"identity":"1c972f95-346c-4142-85e9-c7dadac5a5ed","added_by":"auto","created_at":"2025-11-23 11:45:40","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":175195,"visible":true,"origin":"","legend":"","description":"","filename":"Fig.docx","url":"https://assets-eu.researchsquare.com/files/rs-7842934/v1/e96c26074719e9b8177b7e74.docx"},{"id":96556262,"identity":"ee7308da-92e0-4b42-b971-a36b4f98fbc5","added_by":"auto","created_at":"2025-11-23 11:45:40","extension":"docx","order_by":2,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":21044,"visible":true,"origin":"","legend":"","description":"","filename":"Tables.docx","url":"https://assets-eu.researchsquare.com/files/rs-7842934/v1/2637a3e81b81379816a8ce0e.docx"},{"id":96556265,"identity":"cb6ab701-130e-4ab4-bbdb-3cc050ccf870","added_by":"auto","created_at":"2025-11-23 11:45:40","extension":"xml","order_by":3,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":9743,"visible":true,"origin":"","legend":"","description":"","filename":"gsedGSEDD2500174.xml","url":"https://assets-eu.researchsquare.com/files/rs-7842934/v1/a8cc768bfaebff5ad95de889.xml"},{"id":96556267,"identity":"bbf61a26-aa25-44ba-8806-dc12bc9f2d3c","added_by":"auto","created_at":"2025-11-23 11:45:40","extension":"xml","order_by":4,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":996,"visible":true,"origin":"","legend":"","description":"","filename":"GSEDD25001743058.go.xml","url":"https://assets-eu.researchsquare.com/files/rs-7842934/v1/dc3e74001c896165b7ff6db1.xml"},{"id":96605268,"identity":"1d7b4a2d-00c3-479f-9576-125b0d4519cd","added_by":"auto","created_at":"2025-11-24 09:21:54","extension":"xml","order_by":5,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":864,"visible":true,"origin":"","legend":"","description":"","filename":"GSEDD2500174Import.xml","url":"https://assets-eu.researchsquare.com/files/rs-7842934/v1/de16b41898865a6f451e577c.xml"},{"id":96556268,"identity":"5104275c-7283-4019-be16-8dc0f216d7cc","added_by":"auto","created_at":"2025-11-23 11:45:40","extension":"xml","order_by":6,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":42110,"visible":true,"origin":"","legend":"","description":"","filename":"GSEDD25001740enriched.xml","url":"https://assets-eu.researchsquare.com/files/rs-7842934/v1/e2b317860ac2e3bf6e208271.xml"},{"id":96556270,"identity":"c899a890-f48b-45b3-9626-f8cd664f0e22","added_by":"auto","created_at":"2025-11-23 11:45:40","extension":"png","order_by":9,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":34291,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-7842934/v1/11d90d9a6cc2b5b6ba1f5da9.png"},{"id":96556269,"identity":"fca42354-93fd-4b34-9b26-386a2df188cc","added_by":"auto","created_at":"2025-11-23 11:45:40","extension":"xml","order_by":10,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":40504,"visible":true,"origin":"","legend":"","description":"","filename":"GSEDD25001740structuring.xml","url":"https://assets-eu.researchsquare.com/files/rs-7842934/v1/26e13bed8c0864e81d474307.xml"},{"id":96556266,"identity":"bc4f742b-29f4-47d8-8266-1d4b26e3cac9","added_by":"auto","created_at":"2025-11-23 11:45:40","extension":"html","order_by":11,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":45196,"visible":true,"origin":"","legend":"","description":"","filename":"earlyproof.html","url":"https://assets-eu.researchsquare.com/files/rs-7842934/v1/3104edba36a12db46bc2c8ae.html"},{"id":96604975,"identity":"5f1261ec-d97d-4df6-8fae-c5035df3b1bb","added_by":"auto","created_at":"2025-11-24 09:17:04","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":138529,"visible":true,"origin":"","legend":"\u003cp\u003eAdjusted and raw interview scores are shown for each student. Non-overlapping CI is statistically significant. Note the smaller CI in the mixed model. The X-axis is ordered from lowest adjusted score to highest.\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-7842934/v1/d83b7ff00a2507918b9f5b66.png"},{"id":96556261,"identity":"a3a5b4fc-b4a6-418c-a4b7-e8f6e057211d","added_by":"auto","created_at":"2025-11-23 11:45:39","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":80761,"visible":true,"origin":"","legend":"\u003cp\u003eRaw mean scores resulted in fewer statistically distinct students (average 11.1) from any one student compared to using scores generated from our mixed model (average 30.1). The number of distinct students increases at the extremes of the score range. Using the Mixed Model to generate a score, students are far more distinct from each other compared to using the raw score.\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-7842934/v1/5caa7777b935634d4b3260be.png"},{"id":96607895,"identity":"63a65e78-d283-4005-8d8b-8aaf191a5886","added_by":"auto","created_at":"2025-11-24 09:28:05","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":667292,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7842934/v1/a67a5f25-c407-48c0-b9ce-602410cad440.pdf"}],"financialInterests":"","formattedTitle":"Improving the Resident Interview Process with Structural and Statistical Bias Correction.","fulltext":[{"header":"Introduction","content":"\u003cp\u003eResidency selection relies heavily on application metrics and interviews, but the latter remains a subjective and error-prone process. Improving interview reliability and fairness is essential to support equitable access and optimal trainee selection.\u003c/p\u003e\u003cp\u003eBiases in the selection process may exclude promising students from a residency program. There are biases inherent to both the evaluator and the structure of the interview process. Statistical bias is an intrinsic tendency that produces differences between results and actual facts. Examples of evaluator bias include stereotyping, halo effect, leniency, central tendency, and conformity bias (\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e). The structure can be biased with group interviews, group discussions of candidates before scoring, or some evaluators not present for every interview.\u003c/p\u003e\u003cp\u003eWe undertook this study measuring the biases in our interview process and explored statistical and structural methods to overcome these biases to achieve a reflective and fair interview process.\u003c/p\u003e"},{"header":"Methods","content":"\u003cp\u003eWe received applications through the Electronic Residency Application Services of the Association of American Medical Colleges. Applications were reviewed by the program director and core faculty. Reviewers had access to full ERAS data, including transcripts, test scores, personal statements, and recommendation letters. Decisions for interview invitations were made in a non blinded, consensus based process. Interview scores were used as the primary ranking tool post interview\u003c/p\u003e\u003cp\u003eIn AY 22, our interview schedule consisted of four interview days, and each student met individually (virtual meetings) with six evaluators for 15 minutes each. There was a pool of 8 evaluators to accommodate scheduling conflicts to ensure each student had six interviews. Each evaluator was instructed to rate (score) each student on a 1 to 10 scale (1 being the worst and 10 being the best). Each evaluator had access to the student\u0026rsquo;s complete application before the interview, including medical or osteopathic school transcripts, United States Medical Licensing Examination (USMLE) scores (step 1, 2), letters of recommendation, personal statement, and curriculum vitae. The interviews were free-form, and there were no specific scoring instructions. After the interviews, the evaluators discussed all the students in the groups before the scores were submitted.\u003c/p\u003e\u003cp\u003eUSMLE 1 scores and the \u0026ldquo;prestige\u0026rdquo; of a medical school were hypothesized to be a potential source of evaluator bias, and we incorporated these variables in our initial modeling. School prestige was a continuous variable defined as the ranking of medical schools by the U.S. News and World Report 2021. A score of 1 was the highest, and 100 was the lowest possible. USMLE 1 was a continuous variable (211 to 264 in our sample). Candidates were given anonymous IDs each year of 1\u0026ndash;40.\u003c/p\u003e\u003cp\u003eWe used a multilevel mixed-effects ordered logistic regression with an unstructured covariance matrix and robust error correction with maximal likelihood estimation to analyze the interview score. The aim was to isolate evaluator-specific variance and estimate adjusted candidate rankings while accounting for missing data and scoring tendencies. Adjusted scores were then standardized for comparison and ranking purposes. We used the student as a random effect. The fit of non-nested models was examined by comparing their Akaike information criterion (AIC). Generalized linear latency was used to measure significant differences in variance between the interviews.\u003c/p\u003e\u003cp\u003eWe used the coefficient for each candidate from our mixed model to create an adjusted score. We estimated the number of statistically distinct categories by counting the non-overlapping confidence intervals.\u003c/p\u003e\u003cp\u003eIn AY 23, we attempted to minimize some of the statistical biases we identified in AY 22. In the second part of our study, we mandated a balanced set of four evaluators (every evaluator interviewed every student) and developed a set of consistent, structured interview questions. The scoring sheet now listed standardized interpretations of each score (i.e., 10 is the best student ever seen. 5 is a typical average student) to improve distinctions in scores and decrease clustering at the high scores range. The interviewers did not discuss applicants until after submitting the scoring to prevent post-interview biases. We then repeated the above modeling.\u003c/p\u003e\u003cp\u003eThe Interrater Reliability (IRR) was calculated in AY22 using a one-way random effects model between average measurements. In AY23, as we had the same raters for all students, we used a two-way mixed effects model. Shapiro Wilk test was used to test normality. IRB exempt status was obtained.\u003c/p\u003e\u003cp\u003eAll calculations were performed using Stata 17.0 (College Station, TX). All candidate scores were de-identified prior to analysis. The anonymization and data linkage were performed by the program director, and the first author only received coded data. No resident had access to identifiable interview results of their peers. Data archiving is not mandated but data will be made available on reasonable request.\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e compares the interview structure between AY22 and AY23. Both distributions of scores between AY22 and AY23 were right censored (that is, there was clustering of scores towards the high end) and skewed (median and mean scores differed). Thus, the data is not normally distributed, invalidating many statistical tools (ANOVA, t-tests, linear regression). Our attempts to improve normality were unsuccessful in AY23 as scores still demonstrated clustered in the higher registry to a similar degree.\u003c/p\u003e\u003cp\u003e\u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e\u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e\u003cdiv class=\"CaptionContent\"\u003e\u003cp\u003eCharacteristic of the General Surgery Interview.\u003c/p\u003e\u003c/div\u003e\u003c/caption\u003e\u003ccolgroup cols=\"3\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e\u003cp\u003eAY22\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c3\"\u003e\u003cp\u003eAY23\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eApplicant Pool\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e2300\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e1794\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eApplicants offered interviews\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e43\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e45\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eApplicants declined interview\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e3\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e5\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eApplicants Interviewed\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e40\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e40\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eSize of Evaluator Pool\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e8\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e4\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eMedian # Interviews (range)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e6 (\u003cspan additionalcitationids=\"CR5\" citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e4\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eStructured Interview\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eNo\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003eYes\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eScoring Guide\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eNo\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003eYes\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eRange of interviews performed by each evaluator\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e10\u0026ndash;40 (mean 30)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e40\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eAverage score\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e7.14\u0026thinsp;\u0026plusmn;\u0026thinsp;2.3(C.I. 6.84\u0026ndash;7.43)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e6.84\u0026thinsp;\u0026plusmn;\u0026thinsp;1.9 (C.I. 6.6\u0026ndash;7.1)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eMedian Score\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e7.0\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e7.0\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eNormal Distribution\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eNo (p\u0026thinsp;\u0026lt;\u0026thinsp;0.0001)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003eNo (p\u0026thinsp;=\u0026thinsp;0.01)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/colgroup\u003e\u003c/table\u003e\u003c/div\u003e\u003c/p\u003e\u003cp\u003eWhen examining the mean raw scores, there was minimal differentiation between applicants for either of the academic years due to the wide statistical error (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). While each student differed from 11\u0026thinsp;\u0026plusmn;\u0026thinsp;1.1 other students, that difference varied depending on the score. Students' raw scores were most different at the extremes compared to the large indistinguishable middle group. In fact, there were only three groups of students, with most in the middle group. This poor differentiation in raw scores was no different between AY22 and AY23, even with the institution of balanced and structured interviews. Figure\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e. The interrater reliability (IRR) was at 0.69 (C.I. 0.50\u0026ndash;0.82), suggesting good consistency between the judges.\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\n\u003ch3\u003eCorrecting using Mixed Effects Model\u003c/h3\u003e\n\u003cp\u003eFor AY22, examining our mixed effects logit model, two evaluators scored significantly more leniently than the others; one only interviewed ten candidates. Only two evaluators interviewed all candidates (Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e). All the evaluators had significantly different variances (p\u0026thinsp;\u0026lt;\u0026thinsp;0.0001) in their scoring.\u003c/p\u003e\u003cp\u003e\u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e\u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e\u003cdiv class=\"CaptionContent\"\u003e\u003cp\u003eResults from Mixed Ordered Logistic Model\u003c/p\u003e\u003c/div\u003e\u003c/caption\u003e\u003ccolgroup cols=\"5\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e\u003cp\u003eAdjusted Parameters\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colspan=\"2\" nameend=\"c3\" namest=\"c2\"\u003e\u003cp\u003eAY22\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colspan=\"2\" nameend=\"c5\" namest=\"c4\"\u003e\u003cp\u003eAY23\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e\u003cp\u003eCoefficient\u0026thinsp;\u0026plusmn;\u0026thinsp;S.E.\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c3\"\u003e\u003cp\u003eP-value\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c4\"\u003e\u003cp\u003eCoefficient\u0026thinsp;\u0026plusmn;\u0026thinsp;S.E.\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c5\"\u003e\u003cp\u003eP-value\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eGender of Candidate (M v F)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e0.56\u0026thinsp;\u0026plusmn;\u0026thinsp;0.56\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e0.99\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e-0.69\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e0.13\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eUSMLE1\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e-0.014\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e0.51\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e0.002\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e0.90\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colspan=\"5\" nameend=\"c5\" namest=\"c1\"\u003e\u003cp\u003e\u003cb\u003eEvaluator as independent predictor of score (#of Interviews AY22,AY23)\u003c/b\u003e\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eEvaluator 1 (10,40)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e1\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e1\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eEvaluator 2 (30,40)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e0.51\u0026thinsp;\u0026plusmn;\u0026thinsp;0.59\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e0.38\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e-0.21\u0026thinsp;\u0026plusmn;\u0026thinsp;0.36\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e0.56\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eEvaluator 3 (40)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e1.18\u0026thinsp;\u0026plusmn;\u0026thinsp;0.52\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e0.03\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e0.15\u0026thinsp;\u0026plusmn;\u0026thinsp;0.38\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e0.69\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eEvaluator 4 (14,40)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e2.28\u0026thinsp;\u0026plusmn;\u0026thinsp;0.80\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e0.005\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e-0.41\u0026thinsp;\u0026plusmn;\u0026thinsp;0.47\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003e0.38\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eEvaluator 5 (40,40)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e-0.09\u0026thinsp;\u0026plusmn;\u0026thinsp;0.53\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e0.87\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eEvaluator 6 (30)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e-0.66\u0026thinsp;\u0026plusmn;\u0026thinsp;0.88\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e0.45\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eEvaluator 7 (39)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e-0.08\u0026thinsp;\u0026plusmn;\u0026thinsp;0.56\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e0.89\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eEvaluator 8 (35,40)\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e-0.49\u0026thinsp;\u0026plusmn;\u0026thinsp;0.58\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e0.405\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eLenient vs total evaluators\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003e2/8\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003e0/4\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eVariance Between Evaluators\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e\u003cp\u003eYES\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003eP\u0026thinsp;\u0026lt;\u0026thinsp;0.0001\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e\u003cp\u003eNO\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e\u003cp\u003eP\u0026thinsp;=\u0026thinsp;0.57\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/colgroup\u003e\u003c/table\u003e\u003c/div\u003e\u003c/p\u003e\u003cp\u003eIdentifying the leniency and unbalanced interviews as a source of bias, we altered our interview structure as described in Methods.\u003c/p\u003e\u003cp\u003eUnlike in AY22, there was no statistical difference in the measurement variance between any evaluators, and no evaluator was statistically harsher (or more lenient) than the others.\u003c/p\u003e\u003cp\u003eIn both years, our mixed model substantially improved the differentiation of students by decreasing error and controlling for interviewer bias. Prior to modeling, each student's score was statistically distinct from 11.1 +-1.1 other candidates on average. After adjustment, this increased to 30.1+-1.0, reflecting improved discrimination between candidates. While there was still more differentiation at the extremes of score, this was much less pronounced than the raw mean scores (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e). When we ranked the scores, our model substantially altered the rank order. Comparing student ranking using the raw and corrected interview score in AY22, corrected scores varied between \u0026minus;\u0026thinsp;8 to +\u0026thinsp;10 positions, with 90% (36/40\u003cem\u003e)\u003c/em\u003e of candidates having a change in their rank order after adjustment. Interestingly, the difference was less pronounced in AY23, with ranking only varying between \u0026minus;\u0026thinsp;3 to +\u0026thinsp;3 positions, with 57% (23/40) of candidates having a change in their rank order after adjustment. Gender, School prestige, and USMLE1 score were not significant predictors of interview scores.\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003eThe R-squared for the AY22 mixed model was 0.70, and the R-squared for the AY23 mixed model was 0.95. This suggests that the process improvement we undertook based on the analysis of AY22 resulted in a substantially more valid scoring for our students.\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eAmong a pool of students applying for general surgery residency at a single institution, our study shows that using the uncorrected evaluators\u0026rsquo; impressions results in a minimal distinction between any of the candidates except at the extremes of the score range, statistically demonstrating that the interview, in its raw form, cannot be a valuable tool in the resident selection. If the evaluators in a rating system cannot reliably distinguish subjects, there is no value in the process (except, perhaps, to attract candidates or eliminate far outliers). Using a statistical model, we were able to distinguish candidates much better.\u003c/p\u003e\u003cp\u003eThe shortcomings of our initial interview process became apparent to us only after analysis. Substantial sources of scoring error occurred because some evaluators had intrinsically harsher or lenient scoring systems, not all evaluators evaluated all candidates, each evaluator had a different variance in their scoring, the scoring was clustered, and there were differences in the number of interviews per student (Table \u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e).\u003c/p\u003e\u003cp\u003e\u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e\u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e\u003cdiv class=\"CaptionContent\"\u003e\u003cp\u003eBias encountered\u003c/p\u003e\u003c/div\u003e\u003c/caption\u003e\u003ccolgroup cols=\"1\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e\u003cp\u003eSources of Statistical Bias Encountered\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e- Non-Normality\u003c/p\u003e\u003cp\u003e- Skewed (Significant deviation from Median and Mean)\u003c/p\u003e\u003cp\u003e- Clustering (grouping at highest scores)\u003c/p\u003e\u003cp\u003e- Heteroskedasticity (lower scores had greater discord between evaluators)\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eUnbalanced Data\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e- Unbalanced Data\u003c/p\u003e\u003cp\u003e- Not every evaluator interviewed every student\u003c/p\u003e\u003cp\u003e- Each evaluator had their own variances in score\u003c/p\u003e\u003cp\u003e- Some evaluators score significantly more harsh or lenient than others\u003c/p\u003e\u003cp\u003e- Not all students had the same number of interviews\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eRepeated measures\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003e- Standard deviation around mean\u003c/p\u003e\u003cp\u003e- Difficult to maintain consistency over prolonged interview season\u003c/p\u003e\u003cp\u003e- Interviewer fatigue results in changes in scoring over the day\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/colgroup\u003e\u003c/table\u003e\u003c/div\u003e\u003c/p\u003e\u003cp\u003eUnderstanding how bias is introduced in an interview is essential to appropriately utilizing this costly, time-intensive, and anxiety-producing process (\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e). While non-normality and clustering are not biases in a traditional epidemiological sense, they present challenges when using raw means to differentiate similarly scoring candidates. Our use of statistical bias here refers to measurement errors that reduce score discrimination and violate assumptions of many common parametric statistical tools. We showed that one substantial factor in our processes\u0026rsquo; structure was due to unbalanced measurements. This is where the students have a differing number of interviews, or, more commonly, not every evaluator rates every student. In the former case, differences in measurement variance would result. In the latter case, evaluator biases will not be evenly distributed amongst the entire group of students. This is especially problematic with leniency bias. Leniency (or harshness) bias will unfairly elevate or depress a student\u0026rsquo;s rating.\u003c/p\u003e\u003cp\u003eStatistical modeling using mixed effects models is ideal for controlling individual evaluator variances. It can partially correct for other sources of statistical error (as they account for random effect and missing data.) Mixed models separate the error (variance) between the independent variables (each interviewer) and the random effect (each student). By controlling for each interviewer (who each has their tendencies for scoring), we can better measure distinguishing factors from the student. We also observed the right censoring (or clustering of scores at the high end), which we suspect is common in most interviews. We attempted to improve this clustering in AY23 by defining each score (1\u0026ndash;10) with guides, although this was only slightly successful.\u003c/p\u003e\u003cp\u003eHow a program weighs any part of the student application, including the interview, in developing a resident rank order list should be commensurate with its accuracy in predicting a successful resident. If a program does not measure the IRR of the evaluators, then the interview might be unduly weighted and generate unreliable results.\u003c/p\u003e\u003cp\u003eInterviews in medical education and other employment interviews have previously been shown not to be predictive of achievement (\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e). A striking example occurred at the University of Texas Medical School at Houston. In 1987, Texas state legislators realized that the state was short on physicians. To fix the problem, the legislature required the school to increase the class size from 150 to 200 after the admissions committee had already chosen its preferred 150 students. The 50 students brought into the class had previously received the lowest ranking from the admissions committee. The performance at graduation of initially accepted and initially rejected students was the same (\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e).\u003c/p\u003e\u003cp\u003eWe have learned from this study how to design the interview process in our institution better. We suggest that interviews be independently performed (not in a group), and evaluators should be blinded to others\u0026rsquo; impressions. All applicants should have an equal number of interviews and the same evaluators. Additionally, evaluators should undergo pre-interview training to self-identify types of bias and standardize the scoring system. Design alone is not enough to overcome the substantial lack of differentiation between candidates; thoughtful statistical modeling was also needed.\u003c/p\u003e\u003cp\u003eAn interview process might be poor at differentiating candidates because the candidates performed equally well in an interview session or the interviews were poorly designed to measure important differences. Despite the statistical errors that seemingly lead to arbitrary ranking decisions, the current rank system works well from the standpoint of most programs. This might be because the students we select are already highly functioning, educated, motivated, and have already overcome substantial educational barriers. We perceive our current selection process to be excellent because any random group of applicants would likely be excellent. Perhaps there is little additional valuable information that can be obtained from an interview process that is predictive of resident success, and the focus should be on truthfully marketing a program\u0026rsquo;s attributes to better match the student\u0026rsquo;s needs.\u003c/p\u003e\u003cp\u003eThe most important limitation in this study should be highlighted: We (nor any others to our knowledge) know what predictive attributes predict who will be an exceptional resident. Those personality characteristics are much more challenging to measure than medical school grades or test scores (\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e). This is also a small study from a single institution, but our interview process is, in essence, like most programs, and thus, the conclusions are generalizable.\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eA typical residency interview structure is statistically biased and unable to easily differentiate between students. Through process improvement, guided by statistical modeling, we managed many of these biases and substantively improved the validity of the interview process. As the career stakes for each applicant are high and with every program aiming to recruit the best applicant, designing a fair system should be a priority.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eNRMP, 2022, Advanced Data Report. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.nrmp.org/wp-content/uploads/2022/03/Advance-Data-Tables-2022-FINAL.pdf\u003c/span\u003e\u003cspan address=\"https://www.nrmp.org/wp-content/uploads/2022/03/Advance-Data-Tables-2022-FINAL.pdf\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eLaszlo Kiraly, Elizabeth Dewey, Karen Brasel,\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eHawks and Doves: Adjusting for Bias in Residency Interview Scoring, Journal of Surgical Education, Volume 77, Issue 6, 2020, Pages e132-e137,\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eMcDaniel, M. A., Whetzel, D. L., Schmidt, F. L., \u0026amp; Maurer, S. D. (1994). The validity of employment interviews: A comprehensive review and meta-analysis. \u003cem\u003eJournal of Applied Psychology\u003c/em\u003e, \u003cem\u003e79\u003c/em\u003e(4), 599\u0026ndash;616.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eSmith, S R. Medical school and residency performances of students admitted with and without an admission interview. Academic Medicine 66(8):p 474\u0026ndash;6, August 1991.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eStephenson-Famy A, Houmard BS, Oberoi S, Manyak A, Chiang S, Kim S. Use of the Interview in Resident Candidate Selection: A Review of the Literature. J Grad Med Educ. 2015;7(4):539\u0026ndash;48\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":true,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"global-surgical-education-journal-of-the-association-for-surgical-education","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"GSED","sideBox":"Learn more about [Global Surgical Education - Journal of the Association for Surgical Education](https://link.springer.com/journal/44186)","snPcode":"44186","submissionUrl":"https://www.editorialmanager.com/gsed/default1.aspx","title":"Global Surgical Education - Journal of the Association for Surgical Education","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-7842934/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7842934/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003eIntroduction\u003c/strong\u003e: Biases in the general surgery interview selection process may exclude promising students from a residency program. There are biases inherent to both the evaluator and the structure of the interview process. Several studies have questioned the reliability of the interview process. We aimed to improve the resident interview structure through structural improvements in the interview and increasing the validity of the selection methodology.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMethods: \u003c/strong\u003eWe performed a prospective comparative analysis of our general surgery residency interview process over two consecutive academic years (AY 2022 and 2023). We used descriptive statistics and a mixed-effects ordered logit model to measure bias and guide process improvement.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResults: \u003c/strong\u003eEighty total students were interviewed. We found numerous statistical biases, including leniency bias exacerbated by the unequal distribution of evaluators, significant differences in the variance from evaluator’s scoring (p \u0026lt; 0.0001), and non-normal distribution (p \u0026lt; 0.001). Rater reliability was “good” in both years: 0.69 (C.I. 0.51–0.82). Using uncorrected means, on average, each student was statistically different from 11.5 ± 1.1 other students. Using our model, we improved the number of statistically distinct groupings as each student now differed by 30 ± 1.0 others (p \u0026lt; 0.0001). In AY23, we restructured the interviews so that all the evaluators scored every student, which significantly improved the interview accuracy using the same mixed model: R-squared of 0.95 versus 0.70, and a smaller percent of students had a change in their rank using the improved structure (57% versus 90%, p \u0026lt; 0.05), compared to AY22.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConclusions: \u003c/strong\u003eAmong a pool of students applying for general surgery residency at a single institution, our study shows that using the uncorrected evaluators’ impressions results in a minimal distinction between any of the candidates except at the extremes of the score range, statistically demonstrating that the interview, in its raw form, cannot be a valuable tool in the resident selection. Due to numerous statistical biases, there is little differentiation between students and thus little validity in scoring students using raw mean scores. This can be overcome by developing a structured interview and correcting for statistical biases. We suggest that interviews be independently performed (not in a group), and evaluators should be blinded to others’ impressions.\u003c/p\u003e","manuscriptTitle":"Improving the Resident Interview Process with Structural and Statistical Bias Correction.","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-11-23 11:45:35","doi":"10.21203/rs.3.rs-7842934/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"reviewerAgreed","content":"","date":"2025-11-11T18:15:35+00:00","index":0,"fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-11-11T17:33:45+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"Global Surgical Education - Journal of the Association for Surgical Education","date":"2025-11-11T03:51:16+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-11-10T14:14:17+00:00","index":"","fulltext":""},{"type":"submitted","content":"Global Surgical Education - Journal of the Association for Surgical Education","date":"2025-10-12T16:57:17+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"global-surgical-education-journal-of-the-association-for-surgical-education","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"GSED","sideBox":"Learn more about [Global Surgical Education - Journal of the Association for Surgical Education](https://link.springer.com/journal/44186)","snPcode":"44186","submissionUrl":"https://www.editorialmanager.com/gsed/default1.aspx","title":"Global Surgical Education - Journal of the Association for Surgical Education","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"eb8141cb-e54e-465b-9192-8d16ded9c8b4","owner":[],"postedDate":"November 23rd, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2026-01-06T12:02:52+00:00","versionOfRecord":[],"versionCreatedAt":"2025-11-23 11:45:35","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7842934","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7842934","identity":"rs-7842934","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}