Unique performance problems in surgical residency as novel predictors of negative post-residency outcomes

Unique performance problems in surgical residency as novel predictors of negative post-residency outcomes | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Unique performance problems in surgical residency as novel predictors of negative post-residency outcomes Nicole Roberts, Margaret L Boehler, V. Prasad Poola This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6968543/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Purpose: Surgical educators have an interest in ensuring graduates of residency programs are able to pass board examinations and build a surgical practice. We sought to learn if there are indicators in residency that can be used to predict propensity for difficulty, in passing boards or in later practice. Methods We reviewed the residency and post-residency (board results) files of 104 residents in our general surgery program, cataloguing performance problems using a checklist we had developed for a previous study. We then used online resources including state licensing boards to assess non-board post-residency outcomes. We used ROC analysis to determine if there were cutoff scores for ABSITE exams, seriousness of problems, and number of unique problems identified in the file. Results The 44 (42%) of residents who had documented residency performance problems were at statistically significant risk for first-time board failure, negative mentions online, and overall negative post-residency outcomes. Residents who had five or more unique problems in our check list were at statistically significant higher risk of first-time board failure and negative post-residency outcomes than residents with zero documented problems. They were also at statistically significantly higher risk than residents who had 1–4 unique performance problems documented in their files. Residents who had 1–4 documented performance problems were at statistically significantly higher risk than residents who had zero documented performance problems only for negative online mentions. Conclusion Capturing unique performance problems during residency is a novel method to assess risk for post residency problems, both in passing board exams and in mounting a successful practice. Figures Figure 1 Figure 2 Figure 3 Introduction Characterizing and describing surgery resident performance problems has occupied a conspicuous place in the medical education literature. [ 1 – 5 ] Educators work to recognize when a learner’s performance problem is simply part of the learning process and when the learner’s problem is severe enough to require extra attention. They also seek to understand when a learner’s problems are severe enough to preclude them from continuing in the program. Programs use various measures to ensure their residents’ progress, including the ACGME Milestones, various evaluations, and now Entrustable Professional Activities ratings. They also use measures technically intended to assure programmatic quality, like the ABSITEs, to help predict likelihood of individual resident success. There is always a balance between the learners and their desires to complete training and enter a career in medicine and the future patients of the learner and their potential safety. Educators are sensitive to the costs learners have borne and to the fact that any educational program must be prepared to teach—that learners cannot be expected to know everything. Most programs work towards remediating those who have problems with an aim to keep them in the program to graduate.[ 6 – 9 ] They trust that learners whose problems have come to their attention will straighten out, whether in their current program, in the next step of their training, or with the help of a mentor in practice. In addition, the general public, as well as certifying boards, consider physician board certification as an assurance of both program quality and of the individual physician’s qualifications. We sought to understand if there were other indicators that residencies could use to predict resident performance on boards and post residency. The relationship between trouble in medical school and trouble in residency and in practice has begun to be solidified by the Krupat[ 10 ] study. The authors found that students who appeared before their school’s review boards were 5 times more likely to undergo disciplinary review in residency, and almost 4 times more likely to require remediation or counseling than the control group. Further, they were more likely to have malpractice or disciplinary notices in practice, however, the authors note that there are other, perhaps more important predictors. Our study is focused on the correlation between residency performance and post-residency success. Our research question was: what indicators in residency can be used to signal potential negative post-residency performance indicators. Method We performed a retrospective review of all SIU general surgery categorical residents’ records between 1977 and 2014. This study was deemed non human-subjects research by the institutional review board ( IRB Number : 22–256 Reference Number : 02996). We had previously characterized the performance of residents who were in the program 1978–2006 in a prior study. [ 11 ] We repeated that characterization for residents who followed by constructing a database, kept in Redcap. We used the behavior indicators elaborated in our previous study to characterize in-residency performance, and to determine which of our new cohort of residents had performance problems. We added to the description of both cohort’s percentile scores on in-training exams, and post-residency outcomes. Residents with performance problems were considered our target residents. Those who had no documented performance problems served as our control group. Residents were categorized as having performance problems if a particular problem was noted more than once in the resident’s record or if an unusual, consequential problem was documented. In our experience, it is unusual for a problem to be documented in the resident record unless it is significant. If it is documented more than once, it represents a real, and validated problem. We also recorded the type of problems encountered, remediation chosen and final resolution for trainees. We used the behavioral checklist from our previous study [ 11 ] (See appendix A). with the understanding that if additional categories of problems in performance were noted in our new data set, we would add to the checklist. Items on the checklist were derived from experience of the investigators and discussions with program faculty. The checklist arose from ad-hoc comments added to official evaluations, or from memos or other notices of performance problems. Seriousness was rated by the authors on a scale of 1 (least serious) to 5 (most serious). We did not count the number of citations in a resident’s file. Instead, we counted the unique problems noted in the resident’s file. We then characterized the residents’ post-residency performance as having a good or a bad outcome. A bad outcome is defined as failing board exams, not becoming board certified in a surgical specialty, failure to sustain a practice, or developing a record of difficulty in practice or in public realms in a way that would reflect badly on the person or the surgical profession. Good outcome is defined as passing boards on the first attempt, being board certified in a surgical specialty, being in a surgical practice, sustaining a practice for at least 15 years for those out of residency that long, maintaining licensure, and not having any public reports or notices on licensure of inappropriate behavior or disciplinary action. Data were analyzed using SPSS (version 29; IBM) To ensure the ratings were consistent among the raters, we reviewed a subset of 17 cases. Interrater reliability was 94% (one disagreement). We discussed and came to consensus on the one disagreed upon case. We used an ROC analysis to determine which indicators could be used to predict potential problems in practice. We used MedCalc software (version 23.0.6) to calculate the relative risk of various outcomes, using cohorts of residents with 0 identified performance problems, 1–4 identified performance problems, and 5 or more identified performance problems. Results We extracted the records of 104 residents. Eighty-eight graduated with no documented performance problems. Thirteen graduated with documented unresolved performance problems and three left the program voluntarily. Forty-four (42%) had documented performance problems. Number of unique problems ranged from 0 to 16. Among those who were classified as having performance problems, the mean number of unique problems was 5.36, and the median was 4. We performed ROC analyses on various within residency indicators. We assessed ABSITE scores for each year and for average ABSITE scores. We assessed seriousness of problems as rated by the authors, and we assessed the number of unique problems identified in the resident record. The model quality for the Year 5 ABSITE score was the best of the ABSITE predictors for negative post residency outcomes including qualifying exam failure, however, because of the wide range of scores, the sensitivity and specificity of a cutoff score is questionable. Seriousness of problems as identified by the researchers was a reasonable predictor of board failure and of any of the identified difficulties in practice, with a 3/5 seriousness being a good cutoff. We don’t consider this to be as useful as other indicators because seriousness was judged by three non-surgeon researchers, and thus may be too subjective and dependent on their perceptions. Number of unique problems turned out to be the best predictor of both board failure and difficulty in practice. The ROC analysis for number of unique problems identified a predictive model for failure of ABS qualifying and certifying exam and for negative post-residency outcomes (Figs. 1 – 3 ). Five or more unique problems documented during residency indicated a higher risk of negative post-residency indicators. The ROC allowed us to narrow the number of residents who were considered at high risk from 44 to 20. Of the 20 residents over the 37 years we assessed who had 5 + unique problems, 17 had post residency negative outcomes and three did not. We did not find any commonality among the three who did not have post-residency negative outcomes We assessed relative risk for board failure and negative post residency outcomes for three cohorts: those who had 0 documented performance problems during residency, those who had 1–4 unique documented performance problems, and those who had 5 or more unique performance problems documented (Table 1). Residents who had 5 or more problems were at a statistically significant higher risk for all negative outcomes when compared to those with 0 documented problems. Those with 1–4 unique documented problems were at a statistically significant higher risk only for negative online mentions than those with 0 documented problems. Those with 5 or more unique documented problems were at a statistically significant higher risk for all non-board negative outcomes and for qualifying exam failure. Discussion Our study shows that the resident who had a documented unique problem from our check list is at a higher risk for post residency failure compared to a resident who had none. Similarly, a resident who had five or more problems identified during residency is at a significantly higher risk in terms of passing the ABS boards, and maintaining the practice in good standing. Many residents will have at least one problem documented during the five years of residency training (42% in our study). Most residents in this category ultimately perform adequately—passing board exams and maintaining a surgical practice without disciplinary action. However, they may not fare as well as those who completed residency without any significant concerns. We did not identify a single, specific issue during residency that reliably predicted post-residency failure. This is challenging for two primary reasons: First, problems tend to be highly individualized, influenced by personal factors and the unique structure and culture of each residency program. Second, our study was not designed to isolate such predictors; it spans several decades, encompassing generational shifts and variations in training. Nonetheless, if a specific recurring issue can be identified, it would be worthwhile to initiate a conversation about implementing targeted remediation strategies. In cases where concerns do not rise to the level of disciplinary action and the resident is permitted to graduate—as was the case for most of the residents in our program—it may still be beneficial to counsel the resident about their relatively higher risk of post-residency challenges, especially compared to peers who completed training without issues. It is not surprising that residents with five or more distinct problems documented on our checklist had poor post-residency outcomes. Many failed to pass the boards on the first attempt, and some never passed. Even among those who did pass, ongoing professional difficulties prevented some of them from sustaining a surgical practice in good standing. Several continued to face disciplinary actions and accumulated negative online reviews. Residents in this category would benefit from a thorough “goodness of fit” for surgical practice evaluation. While we are not suggesting that all such residents require disciplinary action or termination, graduating them without serious consideration of their long-term viability does a disservice both to them and to the public. Limitations This is a single institution stud which spans over many decades which has inherent bias. We suspect that the core finding that residents with a breadth of problems are more likely to be flagged as problematic even at graduation and are also less likely to maintain a future surgical practice is generalizable. We did find similar results in the same institution’s orthopaedic surgery residency program. Though using the post-residency indicators that we used means that any program could replicate this study, the indicators are limited and do not capture all potential problems a practicing physician might encounter. We were purposefully liberal in our application of the problem checklist, due to our observation that faculty are loathe to document performance problems, so if something does show up in a resident file, it is likely to have been noticed repeatedly. In addition, our checklist may be institution specific, having been developed based on residency files and faculty experience within our school. Conclusion Our study finds that identifying unique problems from our checklist during residency is helpful in predicting post residency performance. Residency leadership can use this information to augment other evaluation processes to assist in making difficult decisions about a resident’s progression. Though a resident who does not have problems identified by our list is not guaranteed to be successful in all of the measures we identified, one who has 5 or more identified problems is at substantial risk of having negative post-residency outcome. Declarations Conflict of Interest: On behalf of all authors, the corresponding author states that there is no conflict of interest. Data Availability: De-identified data will be made available upon reasonable request. References J. Beard and H. Sanfey, "Managing Underperformance in Trainees," in Advancing Surgical Education: Theory, Evidence and Practice , vol. 17, D. Nestel, K. Dalrymple, J. T. Paige, and R. Aggarwal Eds., (Innovation and Change in Professional Education. Berlin: Springer-Verlag Berlin, 2019, pp. 313–326. W. C. Crannell and K. J. Brasel, "Dealing with the struggling learner," (in English), Surgery , Article vol. 167, no. 3, pp. 523–527, Mar 2020, doi: 10.1016/j.surg.2019.06.013 . K. Hagelsteen, B. M. Johansson, A. Bergenfelz, and C. Mathieu, "Identification of Warning Signs During Selection of Surgical Trainees," (in English), J. Surg. Educ. , Article vol. 76, no. 3, pp. 684–693, May-Jun 2019, doi: 10.1016/j.jsurg.2018.12.002 . R. M. Minter, G. L. Dunnington, R. Sudan, K. P. Terhune, D. L. Dent, and A. K. Lentz, "Can This Resident Be Saved? Identification and Early Intervention for Struggling Residents," (in English), J. Am. Coll. Surg. , Article vol. 219, no. 5, pp. 1088–1095, Nov 2014, doi: 10.1016/j.jamcollsurg.2014.06.013 . H. S. Raman et al. , "Prevalence, management, and outcome of problem residents among neurosurgical training programs in the United States," (in English), J. Neurosurg. , Article vol. 130, no. 1, pp. 322–326, Jan 2019, doi: 10.3171/2017.8.Jns171719 . A. Yaghoubian et al. , "General Surgery Resident Remediation and Attrition A Multi-institutional Study," (in English), Arch. Surg. , Article vol. 147, no. 9, pp. 829–833, Sep 2012, doi: 10.1001/archsurg.2012.1676 . Q. Yan, R. N. Treffalls, T. R. Li, S. Prasla, and M. G. Davies, "Graduate Medical Education "Trainee in difficulty" current remediation practices and outcomes," (in English), Am. J. Surg. , Article vol. 224, no. 2, pp. 796–808, Aug 2022, doi: 10.1016/j.amjsurg.2021.12.031 . K. B. Santosa et al. , "Identifying Strategies for Struggling Surgery Residents," (in English), J. Surg. Res. , Article vol. 273, pp. 147–154, May 2022, doi: 10.1016/j.jss.2021.12.026 . F. Shweikeh, A. C. Schwed, C. H. Hsu, and V. N. Nfonsam, "Status of Resident Attrition From Surgical Residency in the Past, Present, and Future Outlook," (in English), J. Surg. Educ. , Article vol. 75, no. 2, pp. 254–262, Mar-Apr 2018, doi: 10.1016/j.jsurg.2017.07.015 . E. Krupat et al. , "Do Professionalism Lapses in Medical School Predict Problems in Residency and Clinical Practice?," Acad Med , vol. 95, no. 6, pp. 888–895, Jun 2020, doi: 10.1097/ACM.0000000000003145 . R. G. Williams, N. K. Roberts, C. J. Schwind, and G. L. Dunnington, "The nature of general surgery resident performance problems," Surgery , vol. 145, no. 6, pp. 651-8, Jun 2009, doi: 10.1016/j.surg.2009.01.019 . Tables Table 1 is available in the Supplementary Files section. Supplementary Files AppendixA.docx Table1.docx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6968543","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":483548047,"identity":"4a7f4f2b-9c33-42e2-81c3-5f701286970a","order_by":0,"name":"Nicole Roberts","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA5ElEQVRIie3OsQrCMBCA4ZNAptiuLYK+wpWCUhz6KimCvoGrg6CLiKMFH8LJyeEkoIvoKnRRBCcHRRAHB1tdpbWbQ/7hSMJ9EACd7j/j72kyRiCTEwFYPxAEu89lToJbgZ97FqmV1HJ/nT99VOJ2OfRmZYPYIhIpxBs2W874hEGoijMr6EWuTbxRTyO4FtWSIJRGTCAmwZSSlyzyJPRBieMlJp0pmfdsAoSFkRKQfEwiCZ5BeMsZkBuEXV615CZyQsVdb5JK2HL/oLJvmup4fbSjirHqHnbnFPIllm9dp9PpdF96ATP2T0xRPtUcAAAAAElFTkSuQmCC","orcid":"https://orcid.org/0000-0002-1880-2859","institution":"Southern Illinois University School of Medicine","correspondingAuthor":true,"prefix":"","firstName":"Nicole","middleName":"","lastName":"Roberts","suffix":""},{"id":483548048,"identity":"202615c5-bd1a-4c18-935e-2d9303a151aa","order_by":1,"name":"Margaret L Boehler","email":"","orcid":"","institution":"Southern Illinois University School of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Margaret","middleName":"L","lastName":"Boehler","suffix":""},{"id":483548049,"identity":"26f11f11-f871-4150-8d84-198084de9e10","order_by":2,"name":"V. Prasad Poola","email":"","orcid":"","institution":"Southern Illinois University School of Medicine","correspondingAuthor":false,"prefix":"","firstName":"V.","middleName":"Prasad","lastName":"Poola","suffix":""}],"badges":[],"createdAt":"2025-06-24 19:26:29","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6968543/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6968543/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":86765390,"identity":"ac7107b1-493f-4106-a8f2-fc2c721e57b5","added_by":"auto","created_at":"2025-07-15 10:57:30","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":65847,"visible":true,"origin":"","legend":"\u003cp\u003eReceiver Operator Characteristic Unique Problems vs. Qualifying Exams\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-6968543/v1/dcacdc866fcb9a72cbc44db2.png"},{"id":86765392,"identity":"9cce4efe-d794-4a38-81cc-d25d3de11796","added_by":"auto","created_at":"2025-07-15 10:57:30","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":260328,"visible":true,"origin":"","legend":"\u003cp\u003eReceiver operator curve for Number of unique problems vs. certifying exam failure\u003c/p\u003e","description":"","filename":"floatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-6968543/v1/b4b01bfe92a707aee29aa9b0.png"},{"id":86766017,"identity":"ed692f58-6ce9-46d0-9c62-aaa325a27630","added_by":"auto","created_at":"2025-07-15 11:05:30","extension":"jpeg","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":449646,"visible":true,"origin":"","legend":"\u003cp\u003eNumber of unique problems vs. negative post residency indicators other than board exams\u003c/p\u003e","description":"","filename":"floatimage4.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-6968543/v1/4a5f79715d70ddddd647aabd.jpeg"},{"id":88650847,"identity":"7e5c8f92-9083-4a6a-a3b4-653ecb6f9f7a","added_by":"auto","created_at":"2025-08-08 17:32:18","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1096529,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6968543/v1/84e91222-c250-4a79-9be7-7fed2e05e6a4.pdf"},{"id":86766012,"identity":"e3ee0b88-d5a2-44c3-9996-cb6ba6474b8d","added_by":"auto","created_at":"2025-07-15 11:05:30","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":15577,"visible":true,"origin":"","legend":"","description":"","filename":"AppendixA.docx","url":"https://assets-eu.researchsquare.com/files/rs-6968543/v1/8ed0fcbef184e69c70b8ae8e.docx"},{"id":86765394,"identity":"1030d8fb-3438-4041-b75b-81494a71e934","added_by":"auto","created_at":"2025-07-15 10:57:30","extension":"docx","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":15166,"visible":true,"origin":"","legend":"","description":"","filename":"Table1.docx","url":"https://assets-eu.researchsquare.com/files/rs-6968543/v1/baf5b48796dbdc266edd4cf5.docx"}],"financialInterests":"","formattedTitle":"Unique performance problems in surgical residency as novel predictors of negative post-residency outcomes","fulltext":[{"header":"Introduction","content":"\u003cp\u003eCharacterizing and describing surgery resident performance problems has occupied a conspicuous place in the medical education literature. [\u003cspan additionalcitationids=\"CR2 CR3 CR4\" citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e–\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e] Educators work to recognize when a learner’s performance problem is simply part of the learning process and when the learner’s problem is severe enough to require extra attention. They also seek to understand when a learner’s problems are severe enough to preclude them from continuing in the program. Programs use various measures to ensure their residents’ progress, including the ACGME Milestones, various evaluations, and now Entrustable Professional Activities ratings. They also use measures technically intended to assure programmatic quality, like the ABSITEs, to help predict likelihood of individual resident success.\u003c/p\u003e\u003cp\u003eThere is always a balance between the learners and their desires to complete training and enter a career in medicine and the future patients of the learner and their potential safety. Educators are sensitive to the costs learners have borne and to the fact that any educational program must be prepared to teach—that learners cannot be expected to know everything. Most programs work towards remediating those who have problems with an aim to keep them in the program to graduate.[\u003cspan additionalcitationids=\"CR7 CR8\" citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e–\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e] They trust that learners whose problems have come to their attention will straighten out, whether in their current program, in the next step of their training, or with the help of a mentor in practice. In addition, the general public, as well as certifying boards, consider physician board certification as an assurance of both program quality and of the individual physician’s qualifications. We sought to understand if there were other indicators that residencies could use to predict resident performance on boards and post residency.\u003c/p\u003e\u003cp\u003eThe relationship between trouble in medical school and trouble in residency and in practice has begun to be solidified by the Krupat[\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e] study. The authors found that students who appeared before their school’s review boards were 5 times more likely to undergo disciplinary review in residency, and almost 4 times more likely to require remediation or counseling than the control group. Further, they were more likely to have malpractice or disciplinary notices in practice, however, the authors note that there are other, perhaps more important predictors. Our study is focused on the correlation between residency performance and post-residency success. Our research question was: what indicators in residency can be used to signal potential negative post-residency performance indicators.\u003c/p\u003e"},{"header":"Method","content":"\u003cp\u003eWe performed a retrospective review of all SIU general surgery categorical residents’ records between 1977 and 2014. This study was deemed non human-subjects research by the institutional review board (\u003cb\u003eIRB Number\u003c/b\u003e: 22–256 \u003cb\u003eReference Number\u003c/b\u003e: 02996).\u003c/p\u003e\u003cp\u003eWe had previously characterized the performance of residents who were in the program 1978–2006 in a prior study. [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e] We repeated that characterization for residents who followed by constructing a database, kept in Redcap. We used the behavior indicators elaborated in our previous study to characterize in-residency performance, and to determine which of our new cohort of residents had performance problems. We added to the description of both cohort’s percentile scores on in-training exams, and post-residency outcomes.\u003c/p\u003e\u003cp\u003eResidents with performance problems were considered our target residents. Those who had no documented performance problems served as our control group. Residents were categorized as having performance problems if a particular problem was noted more than once in the resident’s record or if an unusual, consequential problem was documented. In our experience, it is unusual for a problem to be documented in the resident record unless it is significant. If it is documented more than once, it represents a real, and validated problem. We also recorded the type of problems encountered, remediation chosen and final resolution for trainees.\u003c/p\u003e\u003cp\u003eWe used the behavioral checklist from our previous study [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e] (See appendix A). with the understanding that if additional categories of problems in performance were noted in our new data set, we would add to the checklist. Items on the checklist were derived from experience of the investigators and discussions with program faculty. The checklist arose from ad-hoc comments added to official evaluations, or from memos or other notices of performance problems. Seriousness was rated by the authors on a scale of 1 (least serious) to 5 (most serious). We did not count the \u003cem\u003enumber\u003c/em\u003e of citations in a resident’s file. Instead, we counted the \u003cem\u003eunique problems\u003c/em\u003e noted in the resident’s file.\u003c/p\u003e\u003cp\u003eWe then characterized the residents’ post-residency performance as having a good or a bad outcome. A bad outcome is defined as failing board exams, not becoming board certified in a surgical specialty, failure to sustain a practice, or developing a record of difficulty in practice or in public realms in a way that would reflect badly on the person or the surgical profession. Good outcome is defined as passing boards on the first attempt, being board certified in a surgical specialty, being in a surgical practice, sustaining a practice for at least 15 years for those out of residency that long, maintaining licensure, and not having any public reports or notices on licensure of inappropriate behavior or disciplinary action.\u003c/p\u003e\u003cp\u003eData were analyzed using SPSS (version 29; IBM) To ensure the ratings were consistent among the raters, we reviewed a subset of 17 cases. Interrater reliability was 94% (one disagreement). We discussed and came to consensus on the one disagreed upon case. We used an ROC analysis to determine which indicators could be used to predict potential problems in practice. We used MedCalc software (version 23.0.6) to calculate the relative risk of various outcomes, using cohorts of residents with 0 identified performance problems, 1–4 identified performance problems, and 5 or more identified performance problems.\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003eWe extracted the records of 104 residents. Eighty-eight graduated with no documented performance problems. Thirteen graduated with documented unresolved performance problems and three left the program voluntarily. Forty-four (42%) had documented performance problems. Number of unique problems ranged from 0 to 16. Among those who were classified as having performance problems, the mean number of unique problems was 5.36, and the median was 4.\u003c/p\u003e\u003cp\u003eWe performed ROC analyses on various within residency indicators. We assessed ABSITE scores for each year and for average ABSITE scores. We assessed seriousness of problems as rated by the authors, and we assessed the number of unique problems identified in the resident record.\u003c/p\u003e\u003cp\u003eThe model quality for the Year 5 ABSITE score was the best of the ABSITE predictors for negative post residency outcomes including qualifying exam failure, however, because of the wide range of scores, the sensitivity and specificity of a cutoff score is questionable.\u003c/p\u003e\u003cp\u003eSeriousness of problems as identified by the researchers was a reasonable predictor of board failure and of any of the identified difficulties in practice, with a 3/5 seriousness being a good cutoff. We don\u0026rsquo;t consider this to be as useful as other indicators because seriousness was judged by three non-surgeon researchers, and thus may be too subjective and dependent on their perceptions.\u003c/p\u003e\u003cp\u003eNumber of unique problems turned out to be the best predictor of both board failure and difficulty in practice. The ROC analysis for number of unique problems identified a predictive model for failure of ABS qualifying and certifying exam and for negative post-residency outcomes (Figs.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e\u0026ndash;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e). Five or more unique problems documented during residency indicated a higher risk of negative post-residency indicators. The ROC allowed us to narrow the number of residents who were considered at high risk from 44 to 20. Of the 20 residents over the 37 years we assessed who had 5\u0026thinsp;+\u0026thinsp;unique problems, 17 had post residency negative outcomes and three did not. We did not find any commonality among the three who did not have post-residency negative outcomes\u003c/p\u003e\u003cp\u003eWe assessed relative risk for board failure and negative post residency outcomes for three cohorts: those who had 0 documented performance problems during residency, those who had 1\u0026ndash;4 unique documented performance problems, and those who had 5 or more unique performance problems documented (Table\u0026nbsp;1). Residents who had 5 or more problems were at a statistically significant higher risk for all negative outcomes when compared to those with 0 documented problems. Those with 1\u0026ndash;4 unique documented problems were at a statistically significant higher risk only for negative online mentions than those with 0 documented problems. Those with 5 or more unique documented problems were at a statistically significant higher risk for all non-board negative outcomes and for qualifying exam failure.\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eOur study shows that the resident who had a documented unique problem from our check list is at a higher risk for post residency failure compared to a resident who had none. Similarly, a resident who had five or more problems identified during residency is at a significantly higher risk in terms of passing the ABS boards, and maintaining the practice in good standing.\u003c/p\u003e\u003cp\u003eMany residents will have at least one problem documented during the five years of residency training (42% in our study). Most residents in this category ultimately perform adequately\u0026mdash;passing board exams and maintaining a surgical practice without disciplinary action. However, they may not fare as well as those who completed residency without any significant concerns. We did not identify a single, specific issue during residency that reliably predicted post-residency failure. This is challenging for two primary reasons: First, problems tend to be highly individualized, influenced by personal factors and the unique structure and culture of each residency program. Second, our study was not designed to isolate such predictors; it spans several decades, encompassing generational shifts and variations in training.\u003c/p\u003e\u003cp\u003eNonetheless, if a specific recurring issue can be identified, it would be worthwhile to initiate a conversation about implementing targeted remediation strategies. In cases where concerns do not rise to the level of disciplinary action and the resident is permitted to graduate\u0026mdash;as was the case for most of the residents in our program\u0026mdash;it may still be beneficial to counsel the resident about their relatively higher risk of post-residency challenges, especially compared to peers who completed training without issues.\u003c/p\u003e\u003cp\u003eIt is not surprising that residents with five or more distinct problems documented on our checklist had poor post-residency outcomes. Many failed to pass the boards on the first attempt, and some never passed. Even among those who did pass, ongoing professional difficulties prevented some of them from sustaining a surgical practice in good standing. Several continued to face disciplinary actions and accumulated negative online reviews.\u003c/p\u003e\u003cp\u003eResidents in this category would benefit from a thorough \u0026ldquo;goodness of fit\u0026rdquo; for surgical practice evaluation. While we are not suggesting that all such residents require disciplinary action or termination, graduating them without serious consideration of their long-term viability does a disservice both to them and to the public.\u003c/p\u003e\u003cp\u003eLimitations\u003c/p\u003e\u003cp\u003eThis is a single institution stud which spans over many decades which has inherent bias. We suspect that the core finding that residents with a breadth of problems are more likely to be flagged as problematic even at graduation and are also less likely to maintain a future surgical practice is generalizable. We did find similar results in the same institution\u0026rsquo;s orthopaedic surgery residency program. Though using the post-residency indicators that we used means that any program could replicate this study, the indicators are limited and do not capture all potential problems a practicing physician might encounter.\u003c/p\u003e\u003cp\u003eWe were purposefully liberal in our application of the problem checklist, due to our observation that faculty are loathe to document performance problems, so if something does show up in a resident file, it is likely to have been noticed repeatedly. In addition, our checklist may be institution specific, having been developed based on residency files and faculty experience within our school.\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eOur study finds that identifying unique problems from our checklist during residency is helpful in predicting post residency performance. Residency leadership can use this information to augment other evaluation processes to assist in making difficult decisions about a resident\u0026rsquo;s progression. Though a resident who does not have problems identified by our list is not guaranteed to be successful in all of the measures we identified, one who has 5 or more identified problems is at substantial risk of having negative post-residency outcome.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003ch2\u003eConflict of Interest:\u003c/h2\u003e\u003cp\u003eOn behalf of all authors, the corresponding author states that there is no conflict of interest.\u003c/p\u003e\u003c/p\u003e\u003ch2\u003eData Availability:\u003c/h2\u003e\u003cp\u003eDe-identified data will be made available upon reasonable request.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eJ. Beard and H. Sanfey, \"Managing Underperformance in Trainees,\" in \u003cem\u003eAdvancing Surgical Education: Theory, Evidence and Practice\u003c/em\u003e, vol. 17, D. Nestel, K. Dalrymple, J. T. Paige, and R. Aggarwal Eds., (Innovation and Change in Professional Education. Berlin: Springer-Verlag Berlin, 2019, pp. 313\u0026ndash;326.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eW. C. Crannell and K. J. Brasel, \"Dealing with the struggling learner,\" (in English), \u003cem\u003eSurgery\u003c/em\u003e, Article vol. 167, no. 3, pp. 523\u0026ndash;527, Mar 2020, doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.surg.2019.06.013\u003c/span\u003e\u003cspan address=\"10.1016/j.surg.2019.06.013\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eK. Hagelsteen, B. M. Johansson, A. Bergenfelz, and C. Mathieu, \"Identification of Warning Signs During Selection of Surgical Trainees,\" (in English), \u003cem\u003eJ. Surg. Educ.\u003c/em\u003e, Article vol. 76, no. 3, pp. 684\u0026ndash;693, May-Jun 2019, doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.jsurg.2018.12.002\u003c/span\u003e\u003cspan address=\"10.1016/j.jsurg.2018.12.002\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eR. M. Minter, G. L. Dunnington, R. Sudan, K. P. Terhune, D. L. Dent, and A. K. Lentz, \"Can This Resident Be Saved? Identification and Early Intervention for Struggling Residents,\" (in English), \u003cem\u003eJ. Am. Coll. Surg.\u003c/em\u003e, Article vol. 219, no. 5, pp. 1088\u0026ndash;1095, Nov 2014, doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.jamcollsurg.2014.06.013\u003c/span\u003e\u003cspan address=\"10.1016/j.jamcollsurg.2014.06.013\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eH. S. Raman \u003cem\u003eet al.\u003c/em\u003e, \"Prevalence, management, and outcome of problem residents among neurosurgical training programs in the United States,\" (in English), \u003cem\u003eJ. Neurosurg.\u003c/em\u003e, Article vol. 130, no. 1, pp. 322\u0026ndash;326, Jan 2019, doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.3171/2017.8.Jns171719\u003c/span\u003e\u003cspan address=\"10.3171/2017.8.Jns171719\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eA. Yaghoubian \u003cem\u003eet al.\u003c/em\u003e, \"General Surgery Resident Remediation and Attrition A Multi-institutional Study,\" (in English), \u003cem\u003eArch. Surg.\u003c/em\u003e, Article vol. 147, no. 9, pp. 829\u0026ndash;833, Sep 2012, doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1001/archsurg.2012.1676\u003c/span\u003e\u003cspan address=\"10.1001/archsurg.2012.1676\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eQ. Yan, R. N. Treffalls, T. R. Li, S. Prasla, and M. G. Davies, \"Graduate Medical Education \"Trainee in difficulty\" current remediation practices and outcomes,\" (in English), \u003cem\u003eAm. J. Surg.\u003c/em\u003e, Article vol. 224, no. 2, pp. 796\u0026ndash;808, Aug 2022, doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.amjsurg.2021.12.031\u003c/span\u003e\u003cspan address=\"10.1016/j.amjsurg.2021.12.031\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eK. B. Santosa \u003cem\u003eet al.\u003c/em\u003e, \"Identifying Strategies for Struggling Surgery Residents,\" (in English), \u003cem\u003eJ. Surg. Res.\u003c/em\u003e, Article vol. 273, pp. 147\u0026ndash;154, May 2022, doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.jss.2021.12.026\u003c/span\u003e\u003cspan address=\"10.1016/j.jss.2021.12.026\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eF. Shweikeh, A. C. Schwed, C. H. Hsu, and V. N. Nfonsam, \"Status of Resident Attrition From Surgical Residency in the Past, Present, and Future Outlook,\" (in English), \u003cem\u003eJ. Surg. Educ.\u003c/em\u003e, Article vol. 75, no. 2, pp. 254\u0026ndash;262, Mar-Apr 2018, doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.jsurg.2017.07.015\u003c/span\u003e\u003cspan address=\"10.1016/j.jsurg.2017.07.015\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eE. Krupat \u003cem\u003eet al.\u003c/em\u003e, \"Do Professionalism Lapses in Medical School Predict Problems in Residency and Clinical Practice?,\" \u003cem\u003eAcad Med\u003c/em\u003e, vol. 95, no. 6, pp. 888\u0026ndash;895, Jun 2020, doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1097/ACM.0000000000003145\u003c/span\u003e\u003cspan address=\"10.1097/ACM.0000000000003145\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003eR. G. Williams, N. K. Roberts, C. J. Schwind, and G. L. Dunnington, \"The nature of general surgery resident performance problems,\" \u003cem\u003eSurgery\u003c/em\u003e, vol. 145, no. 6, pp. 651-8, Jun 2009, doi: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.surg.2009.01.019\u003c/span\u003e\u003cspan address=\"10.1016/j.surg.2009.01.019\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"},{"header":"Tables","content":"\u003cp\u003eTable 1 is available in the Supplementary Files section.\u003c/p\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-6968543/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6968543/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003ePurpose:\u003c/h2\u003e\u003cp\u003eSurgical educators have an interest in ensuring graduates of residency programs are able to pass board examinations and build a surgical practice. We sought to learn if there are indicators in residency that can be used to predict propensity for difficulty, in passing boards or in later practice.\u003c/p\u003e\u003ch2\u003eMethods\u003c/h2\u003e\u003cp\u003eWe reviewed the residency and post-residency (board results) files of 104 residents in our general surgery program, cataloguing performance problems using a checklist we had developed for a previous study. We then used online resources including state licensing boards to assess non-board post-residency outcomes. We used ROC analysis to determine if there were cutoff scores for ABSITE exams, seriousness of problems, and number of unique problems identified in the file.\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e\u003cp\u003eThe 44 (42%) of residents who had documented residency performance problems were at statistically significant risk for first-time board failure, negative mentions online, and overall negative post-residency outcomes. Residents who had five or more unique problems in our check list were at statistically significant higher risk of first-time board failure and negative post-residency outcomes than residents with zero documented problems. They were also at statistically significantly higher risk than residents who had 1\u0026ndash;4 unique performance problems documented in their files. Residents who had 1\u0026ndash;4 documented performance problems were at statistically significantly higher risk than residents who had zero documented performance problems only for negative online mentions.\u003c/p\u003e\u003ch2\u003eConclusion\u003c/h2\u003e\u003cp\u003eCapturing unique performance problems during residency is a novel method to assess risk for post residency problems, both in passing board exams and in mounting a successful practice.\u003c/p\u003e","manuscriptTitle":"Unique performance problems in surgical residency as novel predictors of negative post-residency outcomes","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-07-15 10:57:26","doi":"10.21203/rs.3.rs-6968543/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"e08a07dc-2cf3-4f2c-93e1-2240835bfcd3","owner":[],"postedDate":"July 15th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-08-08T17:24:12+00:00","versionOfRecord":[],"versionCreatedAt":"2025-07-15 10:57:26","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-6968543","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6968543","identity":"rs-6968543","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}