Technical Performance Assessment of Robotic Surgery: A Systematic Scoping Review

Technical Performance Assessment of Robotic Surgery: A Systematic Scoping Review | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Short Report Technical Performance Assessment of Robotic Surgery: A Systematic Scoping Review Seyed Mohammad Kalantar Motamedi, Sahar Fatima, Qian Zhang, Margaret J. Foster, and 11 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7080528/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 04 Sep, 2025 Read the published version in Journal of Robotic Surgery → Version 1 posted 10 You are reading this latest preprint version Abstract Background : Defining performance errors in robotic surgery is critical for the assessment of robotic surgery skill. Our goal was to identify and categorize explicitly defined intraoperative technical errors in robotic surgery, how skill assessment was performed, and how ratings were conducted either manually by experts or via automated ratings. Study Design : This scoping review included studies involving general, urologic, obstetrics/gynecologic, and thoracic surgery, and general skills as practiced in inanimate, virtual reality, in vivo / ex vivo animal, cadaver, and human operations. Primary empirical and consensus-building studies were included if they addressed intraoperative performance assessment or error definition and identification. MEDLINE (Ovid), Embase (Ovid), and Compendex were queried for results from 2012 to May 19, 2022. Results : Of 2,642 studies screened, 185 were included. The majority (n=109, 60%) were US-based and involved either simulated surgical procedures using inanimate models (n=88), virtual reality (n=72), or intraoperative performance assessments of robotic surgeries in humans (n=44); 36 studies combined two or more of these settings. Performance errors were explicitly defined in 104 articles (56%), and 64 used previously defined performance rating scales. The method of rating was split between manual (n=137) and automated ratings (n=85). Conclusion : Measures of performance vary considerably. More conceptual work is warranted to explicitly define errors that can inform robotic skill assessment. This is important given the growing interest in developing efficient and reliable objective measures of performance which are likely to rely on automated assessment methods. Robotic Surgical Procedure Surgical Error Task Performance Clinical Competence Intraoperative Period Figures Figure 1 INTRODUCTION Robotic surgery has been rapidly adopted worldwide. In a report by Intuitive,[1] the use of robotic surgery for various surgeries has reached above 2.2 million procedures in 2023 which was 22% growth compared with 2022 . Moreover, data shows that there is a migration from laparoscopic and open procedures towards robotic surgery.[2] Robotic approaches are now reported across multiple countries as the dominant method for procedures such as radical prostatectomy.[3–5] This rapid increase in robotic surgery has led to the need for training programs and ways to assess performance. Concerns have been raised about the quality of robotic training of surgeons, leading to calls for more robust skill assessments.[6] Requirements for credentialing in robotic surgical procedures are largely left to institutional medical staff offices. These requirements vary considerably across institutions.[7] While the minimum number of robotic cases is commonly used as a key requirement for credentialing, research has shown that surgical volume is not correlated to patient outcomes.[8] The current state of robotic surgery skill assessment can be classified into two categories, manual or automated.[9] Manual rating involves real-time observation or post hoc video review by a surgical expert, researcher, or crowd-sourced group using a standardized scoring rubric as a guide, which can be global, procedure-specific, or error-based assessments.[10–19] Manual methods are time-consuming[20] and especially costly as an expenditure of expert surgeons’ time[21], concerns have also been raised that manual assessment can be cognitively taxing, and prone to bias and inconsistencies.[22, 23] Rating scales are also not designed to identify errors; a low score versus high score can represent trainee progress and benchmarking by various methods,[24] but scalar metrics by themselves do not indicate the critical error or mistake, making them valuable but insufficient for coaching inexperienced surgeons. Automated assessments, typically derived from robotic kinematic tracking data, systems events, and surgical video data, promise to conserve cost, time and effort.[9] While these assessments are more efficient, they currently analyze individual tasks and subtasks in terms of kinematic efficiency (e.g., instrument movement, force applied), creating automated performance metrics (APMs)[11] , [14] or Objective Performance Indicators (OPIs).[25–27] These metrics may also lack the transparency needed for purposes of formative feedback, especially if implemented into an artificial intelligence algorithm.[20] Explicitly defined errors would serve as a complementary means of informing both manual and automated scoring systems. The use of automated assessments to distinguish between expertise levels is ongoing, and attention is starting to focus on identifying erroneous motions that contribute to suboptimal performance and errors.[22] However, prior reviews have not attempted to comprehensively identify errors or offer a repository that might bridge the gap between motion- or score-based assessment and error-based assessment. Our goal was to identify, collate, and categorize explicitly defined technical errors, which can provide a basis for informing and advancing scoring systems. METHODS This review is reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR)[28] with additional reference to the PRISMA 2020 update, as relevant.[29] Additional protocol details and data sheets are available upon reasonable request. Study Eligibility We included primary empirical studies broadly related to robotic surgical technical skill and performance assessment to capture any concepts related (implicitly or explicitly) to error definition, identification, evaluation, and prediction. Studies in inanimate, simulated or virtual reality (VR), in vivo / ex vivo animal, cadaver, and human intraoperative environments were included. By exception, we also included consensus-setting papers (e.g., Delphi panels) if the topic of consensus related directly to robotic surgical error definition. Any error definition, severity, evaluation, and prediction related to intraoperative technical skills were included. Surgical domains of interest included those where robotics were most established, i.e., general, abdominal, urologic, obstetrics/gynecologic, and thoracic surgery, as well as core surgical skills (e.g., suturing techniques). All other surgical types were excluded given the more nascent introduction of robotics in these domains, e.g., cardiovascular, orthopedic, cranio-facial, spinal, and dental surgeries, whereas the included domains have a more established history of robotic use. For inclusion, articles must have focused on the intraoperative, technical errors or skill assessment of the surgeon or trainee proceduralist; if the study only included postoperatively detectable outcomes or complications (such as positive surgical margins or anastomotic leak), non-procedural or non-surgeon-centric performance (e.g., pure ergonomics, team communication, or the calibration or precision of the robotic device itself), the paper was excluded. We similarly excluded studies focused only on global efficiency outcomes (such as mean operating time or total bleeding), which, though suggestive of skill level, does not specify individual errors. Articles lacking robotics, such as laparoscopic-only skills, were excluded. Only journal articles with research results were considered; case reports, robotic prototype development papers, protocols, conference papers, and abstracts were excluded. We included preprints only if a peer reviewed version was not available to replace the record by the time of data extraction. Due to limited resources, only English-language articles were included. Studies from 2012 to the search date were included; older studies were excluded to avoid outdated robotic technology and focus on only the latest robotic surgery capabilities in the past decade coupled with advances in machine-readable gesture recognition technology. Search Strategy Our search strategy was refined using paradigm example texts of interest and comments from our team coordinating advisors (see author contributions and acknowledgements). A specialized medical systematic review librarian (MF) then queried MEDLINE (Ovid), Embase (Ovid), and Compendex on May 19, 2022; see Appendix A for all search strings. Selection Process Results were imported into Covidence systematic review software for de-duplication.[30] After confirming inclusion and exclusion heuristics with senior coordinating advisors, authors in the working group used Covidence to screen titles and abstracts (first stage) and assess full texts for inclusion. Covidence progresses a record through each stage after two screeners concur on inclusion or exclusion (and reason for full text exclusion); conflicts were discussed and resolved in working group consensus meetings. Data Charting The working group in consultation with the coordinating advisors drafted a data extraction form in Microsoft Excel, which was tested on a sample of 3-5 included articles per round. After six rounds of refinement, the finalized data charting form was implemented in Covidence where two authors from the working group charted data from included articles, resolving charting discrepancies via consensus meetings while logging heuristics for future article charting. We recorded bibliographic information (date, title, DOI / PMID); country(ies) of study setting; study aim and design (e.g., randomized or nonrandomized interventional; observational; qualitative); surgical domain (general, urology, gynecology; thoracic; and/or core skills if only fundamental inanimate skills were studied); surgical setting (e.g., intraoperative human, VR, inanimate, etc.); type of surgical procedure; participant level of training and any further definition of baseline expertise; number of proceduralists; method of skill assessment (e.g., video recording, live observation, on-board automated simulator metrics); type of rater (e.g., expert, crowd-sourcing); blinding and reported inter-rater reliability, if applicable; any standardized scoring metrics used; presence of any kinematic assessment; specification of errors; and whether specific steps of a surgery were delineated (exhaustively or partially; n/a for basic core skills such as suturing); and the study’s conclusion. Data Synthesis The authors grouped and cross-tabulated results by key charted elements related to our objectives, including surgical setting, type of rater, and method of skill assessment, and met to evaluate meaningful trends and data presentation. Studies were grouped by those explicitly defining errors (in addition to or distinct from the use of performance scoring scales) versus those using scales only; studies using similar cognate terms (such as ‘mistake’, ‘critical failure’) were interpreted as defining errors. Of the studies defining errors, two authors (RS, SF) summarized and categorized the errors by type using thematic analysis, meeting with additional authors (SMKM, JK, QZ) to refine categories. An emergent theme approach was utilized (i.e., in which content was grouped without a priori categorizations in mind).[31] Due to the heterogeneity and breadth of included results, we did not conduct a critical appraisal. RESULTS A total of 3078 titles were screened. As shown in the PRISMA chart (Figure 1) after removing the duplicates, and screening by the titles and abstracts, 568 full-text articles were reviewed and assessed based on inclusion criteria. Finally 185 titles were included (for full list, see Appendix B ). Over the course of the last decade, there was an increasing trend in the number of publications with 11 in 2012, and 24 in 2021. The majority of included studies (n=110, 59%) had US affiliations followed by the United Kingdom (n=14, 8%), Canada (n=9, 5%), The Netherlands and Japan (each n=6, 3%). Studies were performed in a variety of clinical or skills lab settings (Table 1). The most common robotic settings were simulated surgical procedures using inanimate models (48%), virtual reality (39%), and intraoperative assessments (24%). One fifth of the studies combined two or more surgical settings (e.g., comparing VR to intraoperative or training in both inanimate and VR models). After reviewing the study designs, 98 out of 185 (53%) were non-randomized interventional, 38 (21%) were randomized controlled trials and 36 (19%) were observational. Table 1. Frequency of Various Surgical Settings Surgical Setting Numbers of Publications (%) Inanimate models 88 (48) Virtual Reality 72 (39) Intraoperative human 44 (24) Ex vivo Animal 8 (4) In vivo Animal 11 (6) Cadaver 5 (3) Not applicable 3 (2) Note: Total is > 100% of articles due to studies using multiple settings (36 articles). Percentage based on the total number of included articles (N = 185). Types of Rater We compared the types of performance assessment raters across the three most frequently used surgical settings. The types of raters included human raters (n=137, 74%) and automated computer algorithms (n=85, 46%; see Table 2). Some studies (n=36, 19%) incorporated more than one type of rater. Considering the surgical settings of the studies, automated ratings were used more in inanimate vs intraoperative settings, vs expert raters. Table 2. Rater Type by Surgical Setting Type of rater used for skill assessment Number of articles (n, %) Articles with an inanimate setting (n, %) Articles involving virtual reality (n, %) Articles with intraoperative setting (n, %) Human rating 137 (74) 61 (33) 32 (17) 44 (24) Expert rater 92 (49) 40 (22) 20 (11) 32 (17) Crowdsourcing 17 (9) 7 (4) 3 (2) 7 (4） Non-Expert rater 10 (5) 2 (1) 4 (2) 4 (2) Not reported in detail* 18 (11) 12 (6) 5 (3) 1 (1) Automated rating 85 (46) 18 (10) 61 (33) 6 (3) Other † 5 (3) 3 (2) 0 (0) 2 (1) Not applicable ‡ 4 (2) 4 (2) 0 (0) 4 (2) Sum 231 (125) 86 (46) 93 (50) 56 (30) Note: Total is > 100% due to studies using multiple rating types (36 articles). Percentage was based on the total number of included articles (N = 185). *Human raters, non-specific to expertise † Involved alterations of automated data (e.g., author-supplied kinematics equation) ‡ Article contains error definitions but no rating or direct assessment, e.g. Delphi study Explicitly Defined Errors Explicitly defined errors were identified and labeled as errors by the manuscript authors independently of whether they were listed in previously-defined rating scales. From the total of 185, 104 (56%) articles explicitly defined performance errors (Table 3). Sixty-four studies exclusively used predefined scales (Table 4), the majority of which used GEARS followed by OSATS. We categorized author-defined explicit errors into seven themes (Table 3). The most frequent categories of explicitly defined errors were suture / needle placement errors (33%), tissue injury due to excessive force (32%), drops (23%), and instrument-movement related errors (23%). While these categories were mostly distinct, which made errors easy to attribute to a category, arguably some categories could have been merged, for instance suture and needle breakage and tissue injury are both suggestive of excessive force. Also, as shown in table 3, the majority of the articles which defined error categories used inanimate and VR settings, rather than intraoperative. However, the most common error category studied within each setting was not the same. We found “Suturing and needle placement” errors under inanimate settings (29.4%), “Instrument movement-related errors” under VR (28.2%), and “Tissue injury due to excessive force” in intraoperative setting (50%), were the most common error categories within each of those settings. Table 3 . Frequency of articles that explicitly defined errors and their settings Error category Number of articles (n) Articles with an inanimate setting (n) Articles involving virtual reality (n) Articles with intraoperative setting (n) Suturing and needle placement errors (including knot tying) 31 25 5 1 Tissue injury due to excessive force (including damage, tearing, and injuries) 30 14 8 8 Drops (objects and needles) 24 19 5 0 Instrument movement-related errors (including collisions, missed targets, depth perception) 24 13 11 0 Robot control/camera control/instruments out of view 15 6 6 3 Judgment error (including autonomy) 10 6 2 2 Use of rating scale (combined with explicitly defined errors) 6 2 2 2 Total 140 85 39 16 Note: Total exceeds 104 articles due to articles containing more than one explicitly defined error or surgical setting (i.e., 140 instances of explicitly defined errors were found in 104 of the included articles). We found 64 articles that used performance rating scales. The details are shown in Table 4. The most common rating scales were GEARS and OSATS, used in 75% and 50% of the articles, respectively. GEARS was equally utilized to rate intraoperative and inanimate settings while OSATS was mostly used for inanimate surgical settings. Table 4. Performance Rating Scales Used in Robotic Studies Rating scale Total articles (n, %) Articles with an inanimate setting (n, %) Articles involving virtual reality (n, %) Articles with intraoperative setting (n, %) GEARS (Global Evaluative Assessment of Robotic Skills) 48 (75) 8 (13) 15 (23) 25 (39) OSATS (Objective Structured Assessment of Technical Skills) 32 (50) 16 (25) 6 (9) 2 (3) GOALS (Global Objective Assessment of Laparoscopic Skills) 8 (13) 2 (3) 4 (6) 2 (3) RACE (Robotic Anastomosis Competency Evaluation) 7 (11) 2 (3) 2 (3) 3 (5) GERT (Generic Error Rating Tool) 2 (3) 0 (0) 0 (0) 2 (3) PACE (Prostatectomy Assessment and Competence Evaluation) 3 (5) 2 (3) 0 (0) 1 (2) RHAS (robotic hysterectomy assessment score) 1 (2) 0 (0) 0 (0) 1 (2) RARP (Robot-assisted Radical Prostatectomy) Assessment Score 2 (3) 1 (2) 0 (0) 1 (2) APMs (Automated performance metrics) 2 (3) 0 (0) 0 (0) 2 (3) Note: Percentage out of 64 articles that included rating scales (total exceeds 100% due to some studies involving multiple scales) DISCUSSION Key Findings Our scoping review included studies of technical skill assessment during human, animal, inanimate and virtual robotic surgery, with a focus on identifying defined intraoperative technical performance errors. Approximately half of the included studies explicitly defined errors, while the remaining studies used rating scales or automated scoring without explicitly defining performance errors. We categorized the explicitly defined errors into seven themes. Although each of the category labels appear in various studies, the use of these seven categories as distinct themes for all the explicitly defined errors we encountered is novel. Categorizing explicit errors into a comprehensive set of themes may facilitate automated assessment of errors with emerging technology and help advance this technology. The method of rating was split between human raters (74%) and automated ratings (46%), inclusive of studies employing multiple rating methods. As expected, automated ratings predominated in VR-based assessments, while human raters (specifically experts) predominated in inanimate and intraoperative settings. Comparison with the Literature Several recently published systematic reviews identified measures of performance in robotic assisted surgery,[20, 32–34] commonly grouping assessment tools into manual and automated categories similar to our approach.[20, 33, 34] These reviews included studies using error-based assessments. However, none of these systematic reviews attempted to comprehensively list and categorize each of the errors specified within their included studies, in some cases only counting the number of studies using errors in the assessment method with no further synthesis on that topic. [33] Furthermore, none distinguished intraoperative errors from errors diagnosed postoperatively. For instance, Chen et al. included as “technical errors” both tissue trauma, attesting to a theme we also synthesized, and what we regarded as postoperative findings such as positive surgical margins.[20] We included only intraoperative errors in order to focus on errors available for detection by automated systems during the procedure rather than post hoc findings such as positive surgical margins or incontinence. Breadth of inclusion also depends on clear nomenclature which has by no means been standardized; Younes et al. for instance, included only nine studies versus our 185 based on narrower search criteria (clinically relevant performance metrics, or CRPM) and excluded APMs, unlike other reviews [20, 20, 33] including our own, despite potential relevance of APMs to technical assessment. Our synthesized list of error themes could be compared to others. Of the manual rating tools noted in Table 4, one was explicitly labeled as a “generic error rating tool” (GERT), developed for laparoscopic hysterectomy. It contained four error modes: too much force or distance, too little force or distance, inadequate visualization and wrong orientation of the instrument. Each error was counted regardless of sequelae such as bleeding or tissue damage, which deviates from the semantics used by most robotic rating scales. In a study focused on automated detection of suturing and needle-passing errors, Hutchinson et al.[22] proposed classifying errors into three broad categories–efficiency, safety, and other task metrics (e.g., camera movement). Though grouped differently, their prime examples (multiple attempts, needle drop, needle orientation, and out of view), also fit within our categorization, though we did find the distinction between efficiency (i.e., what is optimal versus non-optimal) and error (what is a mistake versus acceptable) a difficult line to draw in practice. Although many authors equate time to completion (efficiency) as a measure of skill, we specifically excluded time to completion as a skill assessment because it excludes performance quality and a threshold is not easily defined for competency. Operative time has “traditionally been used as a proxy for surgical skill” [35]; however, studies evaluating surgical skill show little correlation between skill ratings and operative time.[35] Of note, task-specific time penalties were encountered in some reviews; however, we elected not to classify time cut-offs as errors as none of the study authors explicitly defined them as such. The State of Performance Assessment The cited reviews concluded similarly that universally accepted robotic skills’ assessments [9] or performance metrics[32] do not currently exist, are not well validated[33], and that a standardized objective metric of technical performance is required.[34] We designed this review with an intent to scope the literature to find the state of technical skill assessment and error definition and likewise conclude that there is no consensus on error definition in literature. However, the future for automated scoring systems is very promising. Other reviews argued that more objectivity was needed in assessments, with some specifically mentioning APMs as offering promise, but needing further work. Limitations We elected to scope nine years of literature (2012-2021) which was intended to capture work on APMs which peaked 2012 onwards. While our focus on intraoperative skill assessment can be seen as a significant limitation this was intentional, as it serves as a foundational basis for future work using computer vision. We did not include studies looking at short and long term clinical outcomes such as postoperative anastomotic leak, urinary incontinence and positive surgical margins as these outcomes are not detectable by contemporaneous automated assessments. Likewise, we excluded nontechnical performance (e.g., cognitive performance and mental workload; workflow interruptions, team communication and coordination metrics), which is recognized as an important component in overall performance (as in the counterpoint argument for more holistic, multi-level measurement by El-Sayed 2024 noted above). No attempt was made to assess bias or quality as our focus was on identifying gaps in the literature rather than demonstrating strength of conclusions. Lastly, we only included English-language sources; however, only three non-English records were excluded at screening based upon language, so the potential for bias based on language criteria is minimal. Conclusion and Proposed Future Work Our review provides a snapshot of the current state of technical skill assessment in robotic surgery and helps to identify the gaps in the literature that can inform skill assessment methods in the future. We found considerable variability in performance assessment methods and error identification, with little conceptual correlation. Intraoperative performance assessment is dominated by manual assessments, with a paucity of automated assessments developed for use in the operating room. We believe the future of robotic surgical skill assessment will incorporate automated skill assessment because of its objectivity: It is more reproducible, reliable and faster than manual assessments. Future work should be directed to aligning errors with APMs, while incorporating computer vision, machine learning and deep learning methods. Automated techniques may not provide surgeons with real-time feedback as consistently as manual assessment; however, consensus definitions of intraoperative technical errors may provide context for such feedback and potentially allow immediate correction. Nonetheless, our review showed that additional work is needed to validate and standardize robotic skill assessment. Declarations Author Contribution D.S, G.L., G.S. and R.S conceptualized the study. SM.KM., M.F and J.K. provided the methodology. SM. KM.,S.F., Q.Z.,M.F., J.K., R.B., J.L., P.L and R.S., did the literature search and review and Data extraction. SM. KM.,S.F., Q.Z.,M.F., J.K. and R.S., did the critical analysis, interpreted the results and drafted the manuscript. All authors reviewed and edited the manuscript. Acknowledgement We thank Dr. Ajit Sachdeva of the American College of Surgeons for valuable feedback on a draft of this manuscript and for inspiring us to pursue this topic. References Intuitive Announces Preliminary Fourth Quarter and Full Year 2023 Results | Intuitive Surgical. https://isrg.intuitive.com/news-releases/news-release-details/intuitive-announces-preliminary-fourth-quarter-and-full-year-3/. Accessed 26 Jun 2024 Sheetz KH, Claflin J, Dimick JB (2020) Trends in the adoption of robotic surgery for common surgical procedures. JAMA Netw Open 3:e1918911. https://doi.org/10.1001/jamanetworkopen.2019.18911 Basto M, Sathianathen N, Te Marvelde L, Ryan S, Goad J, Lawrentschuk N, Costello AJ, Moon DA, Heriot AG, Butler J, Murphy DG (2016) Patterns-of-care and health economic analysis of robot-assisted radical prostatectomy in the Australian public health system. BJU Int 117:930–939. https://doi.org/10.1111/bju.13317 Chae J, Choi Y, Cho S-J (2021) Changes in Patterns of Radical Prostatectomy due to Diffusion of Robotic Surgical System: A Nationwide Study Using Health Insurance Claims Data. Yonsei Med J 62:1155–1161. https://doi.org/10.3349/ymj.2021.62.12.1155 Parsons JK, Messer K, Palazzi K, Stroup SP, Chang D (2014) Diffusion of surgical innovations, patient safety, and minimally invasive radical prostatectomy. JAMA Surg 149:845–851. https://doi.org/10.1001/jamasurg.2014.31 Lee JY, Mucksavage P, Sundaram CP, McDougall EM (2011) Best practices for robotic surgery training and credentialing. J Urol 185:1191–1197. https://doi.org/10.1016/j.juro.2010.11.067 Huffman EM, Rosen SA, Levy JS, Martino MA, Stefanidis D (2021) Are current credentialing requirements for robotic surgery adequate to ensure surgeon proficiency? Surg Endosc 35:2104–2109. https://doi.org/10.1007/s00464-020-07608-2 Larson DW, Marcello PW, Larach SW, Wexner SD, Park A, Marks J, Senagore AJ, Thorson AG, Young-Fadok TM, Green E, Sargent DJ, Nelson H (2008) Surgeon volume does not predict outcomes in the setting of technical credentialing: results from a randomized trial in colon cancer. Ann Surg 248:746–750. https://doi.org/10.1097/SLA.0b013e31818a157d Chen J, Cheng N, Cacciamani G, Oh P, Lin-Brande M, Remulla D, Gill IS, Hung AJ (2019) Objective Assessment of Robotic Surgical Technical Skill: A Systematic Review. J Urol 201:461–469. https://doi.org/10.1016/j.juro.2018.06.078 Bonrath EM, Zevin B, Dedy NJ, Grantcharov TP (2013) Error rating tool to identify and analyse technical errors and events in laparoscopic surgery. Br J Surg 100:1080–1088. https://doi.org/10.1002/bjs.9168 Chen J, Oh PJ, Cheng N, Shah A, Montez J, Jarc A, Guo L, Gill IS, Hung AJ (2018) Use of automated performance metrics to measure surgeon performance during robotic vesicourethral anastomosis and methodical development of a training tutorial. J Urol 200:895–902. https://doi.org/10.1016/j.juro.2018.05.080 Frederick PJ, Szender JB, Hussein AA, Kesterson JP, Shelton JA, Anderson TL, Barnabei VM, Guru K (2017) Surgical Competency for Robot-Assisted Hysterectomy: Development and Validation of a Robotic Hysterectomy Assessment Score (RHAS). J Minim Invasive Gynecol 24:55–61. https://doi.org/10.1016/j.jmig.2016.10.004 Goh AC, Goldfarb DW, Sander JC, Miles BJ, Dunkin BJ (2012) Global evaluative assessment of robotic skills: validation of a clinical assessment tool to measure robotic surgical skills. J Urol 187:247–252. https://doi.org/10.1016/j.juro.2011.09.032 Hung AJ, Chen J, Che Z, Nilanon T, Jarc A, Titus M, Oh PJ, Gill IS, Liu Y (2018) Utilizing Machine Learning and Automated Performance Metrics to Evaluate Robot-Assisted Radical Prostatectomy Performance and Predict Outcomes. J Endourol 32:438–444. https://doi.org/10.1089/end.2018.0035 Hussein AA, Ghani KR, Peabody J, Sarle R, Abaza R, Eun D, Hu J, Fumo M, Lane B, Montgomery JS, Hinata N, Rooney D, Comstock B, Chan HK, Mane SS, Mohler JL, Wilding G, Miller D, Guru KA, Michigan Urological Surgery Improvement Collaborative and Applied Technology Laboratory for Advanced Surgery Program (2017) Development and Validation of an Objective Scoring Tool for Robot-Assisted Radical Prostatectomy: Prostatectomy Assessment and Competency Evaluation. J Urol 197:1237–1244. https://doi.org/10.1016/j.juro.2016.11.100 Lovegrove C, Novara G, Mottrie A, Guru KA, Brown M, Challacombe B, Popert R, Raza J, Van Der Poel H, Peabody J, Dasgupta P, Ahmed K (2016) Structured and Modular Training Pathway for Robot-assisted Radical Prostatectomy (RARP): Validation of the RARP Assessment Score and Learning Curve Assessment. Eur Urol 69:526–535. https://doi.org/10.1016/j.eururo.2015.10.048 Martin JA, Regehr G, Reznick R, Macrae H, Murnaghan J, Hutchison C, Brown M (1997) Objective structured assessment of technical skill (OSATS) for surgical residents: OBJECTIVE STRUCTURED ASSESSMENT OF TECHNICAL SKILL. Br J Surg 84:273–278. https://doi.org/10.1046/j.1365-2168.1997.02502.x Raza SJ, Field E, Jay C, Eun D, Fumo M, Hu JC, Lee D, Mehboob Z, Nyquist J, Peabody JO, Sarle R, Stricker H, Yang Z, Wilding G, Mohler JL, Guru KA (2015) Surgical Competency for Urethrovesical Anastomosis During Robot-assisted Radical Prostatectomy: Development and Validation of the Robotic Anastomosis Competency Evaluation. Urology 85:27–32. https://doi.org/10.1016/j.urology.2014.09.017 Vassiliou MC, Feldman LS, Andrew CG, Bergman S, Leffondré K, Stanbridge D, Fried GM (2005) A global assessment tool for evaluation of intraoperative laparoscopic skills. Am J Surg 190:107–113. https://doi.org/10.1016/j.amjsurg.2005.04.004 Chen J, Cheng N, Cacciamani G, Oh P, Lin-Brande M, Remulla D, Gill IS, Hung AJ (2019) Objective assessment of robotic surgical technical skill: a systematic review. J Urol 201:461–469. https://doi.org/10.1016/j.juro.2018.06.078 Ghani KR, Miller DC, Linsell S, Brachulis A, Lane B, Sarle R, Dalela D, Menon M, Comstock B, Lendvay TS, Montie J, Peabody JO, Michigan Urological Surgery Improvement Collaborative (2016) Measuring to Improve: Peer and Crowd-sourced Assessments of Technical Skill with Robot-assisted Radical Prostatectomy. Eur Urol 69:547–550. https://doi.org/10.1016/j.eururo.2015.11.028 Hutchinson K, Li Z, Cantrell LA, Schenkman NS, Alemzadeh H (2022) Analysis of executional and procedural errors in dry-lab robotic surgery experiments. Int J Med Robot Comput Assist Surg MRCAS 18:e2375. https://doi.org/10.1002/rcs.2375 Oh D, Brown K, Yousaf S, Nesbitt J, Feins R, Sancheti M, Lin J, Yang S, D’Souza D, Jarc A (2023) Differences Between Attending and Trainee Surgeon Performance Using Objective Performance Indicators During Robot-Assisted Lobectomy. Innov Phila Pa 18:479–488. https://doi.org/10.1177/15569845231204607 Goldenberg MG, Garbens A, Szasz P, Hauer T, Grantcharov TP (2017) Systematic review to establish absolute standards for technical performance in surgery. Br J Surg 104:13–21. https://doi.org/10.1002/bjs.10313 Lazar JF, Brown K, Yousaf S, Jarc A, Metchik A, Henderson H, Feins RH, Sancheti MS, Lin J, Yang S, Nesbitt J, D’Souza D, Oh DS (2023) Objective performance indicators of cardiothoracic residents are associated with vascular injury during robotic-assisted lobectomy on porcine models. J Robot Surg 17:669–676. https://doi.org/10.1007/s11701-022-01476-9 Devin CL, Gillani M, Shields MC, Eldredge K, Kucera W, Rupji M, Purvis LA, Paul Olson TJ, Liu Y, Jarc A, Rosen SA (2023) Ratio of Economy of Motion: A New Objective Performance Indicator to Assign Consoles During Dual-Console Robotic Proctectomy. Am Surg 89:3416–3422. https://doi.org/10.1177/00031348231161767 Kaoukabani G, Gokcal F, Fanta A, Liu X, Shields M, Stricklin C, Friedman A, Kudsi OY (2023) A multifactorial evaluation of objective performance indicators and video analysis in the context of case complexity and clinical outcomes in robotic-assisted cholecystectomy. Surg Endosc 37:8540–8551. https://doi.org/10.1007/s00464-023-10432-z Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, Moher D, Peters MDJ, Horsley T, Weeks L, Hempel S, Akl EA, Chang C, McGowan J, Stewart L, Hartling L, Aldcroft A, Wilson MG, Garritty C, Lewin S, Godfrey CM, Macdonald MT, Langlois EV, Soares-Weiser K, Moriarty J, Clifford T, Tunçalp Ö, Straus SE (2018) PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med 169:467–473. https://doi.org/10.7326/M18-0850 Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hróbjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, Stewart LA, Thomas J, Tricco AC, Welch VA, Whiting P, Moher D (2021) The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ n71. https://doi.org/10.1136/bmj.n71 Covidence systematic review software, Veritas Health Innovation, Melbourne, Australia. Available at www.covidence.org. https://support.covidence.org/help/how-can-i-cite-covidence. Accessed 26 Jun 2024 Braun V, Clarke V (2006) Using thematic analysis in psychology. Qual Res Psychol 3:77–101. https://doi.org/10.1191/1478088706qp063oa Younes MM, Larkins K, To G, Burke G, Heriot A, Warrier S, Mohan H (2022) What are clinically relevant performance metrics in robotic surgery? A systematic review of the literature. J Robot Surg 17:335–350. https://doi.org/10.1007/s11701-022-01457-y Boal MWE, Anastasiou D, Tesfai F, Ghamrawi W, Mazomenos E, Curtis N, Collins JW, Sridhar A, Kelly J, Stoyanov D, Francis NK (2024) Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review. Br J Surg 111:znad331. https://doi.org/10.1093/bjs/znad331 El-Sayed C, Yiu A, Burke J, Vaughan-Shaw P, Todd J, Lin P, Kasmani Z, Munsch C, Rooshenas L, Campbell M, Bach SP (2024) Measures of performance and proficiency in robotic assisted surgery: a systematic review. J Robot Surg 18:16. https://doi.org/10.1007/s11701-023-01756-y Addison P, Yoo A, Duarte-Ramos J, Addy J, Dechario S, Husk G, Jarrett M, Teixeira J, Antonacci A, Filicori F (2021) Correlation between operative time and crowd-sourced skills assessment for robotic bariatric surgery. Surg Endosc 35:5303–5309. https://doi.org/10.1007/s00464-020-08019-z Additional Declarations No competing interests reported. Supplementary Files SupplementaryFile.xlsx Cite Share Download PDF Status: Published Journal Publication published 04 Sep, 2025 Read the published version in Journal of Robotic Surgery → Version 1 posted Editorial decision: Accepted 16 Aug, 2025 Reviewers agreed at journal 12 Aug, 2025 Reviews received at journal 31 Jul, 2025 Reviewers agreed at journal 17 Jul, 2025 Reviewers agreed at journal 14 Jul, 2025 Reviewers agreed at journal 12 Jul, 2025 Reviewers invited by journal 09 Jul, 2025 Editor assigned by journal 09 Jul, 2025 Submission checks completed at journal 09 Jul, 2025 First submitted to journal 09 Jul, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7080528","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Short Report","associatedPublications":[],"authors":[{"id":485383167,"identity":"29293028-ee7a-4083-9c9b-2c86af66d5f2","order_by":0,"name":"Seyed Mohammad Kalantar Motamedi","email":"","orcid":"","institution":"Indiana University School of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Seyed","middleName":"Mohammad Kalantar","lastName":"Motamedi","suffix":""},{"id":485383168,"identity":"8e49db9b-138f-4621-b344-124673dfe5df","order_by":1,"name":"Sahar Fatima","email":"","orcid":"","institution":"Houston Methodist","correspondingAuthor":false,"prefix":"","firstName":"Sahar","middleName":"","lastName":"Fatima","suffix":""},{"id":485383169,"identity":"88e685df-e661-4548-8f29-459a4b0ec6b7","order_by":2,"name":"Qian Zhang","email":"","orcid":"","institution":"Houston Methodist","correspondingAuthor":false,"prefix":"","firstName":"Qian","middleName":"","lastName":"Zhang","suffix":""},{"id":485383171,"identity":"ad75b61c-336e-41a6-a326-71a585bd851e","order_by":3,"name":"Margaret J. Foster","email":"","orcid":"","institution":"University Libraries, Texas A\u0026M University","correspondingAuthor":false,"prefix":"","firstName":"Margaret","middleName":"J.","lastName":"Foster","suffix":""},{"id":485383172,"identity":"7bf6950c-ea72-494f-8827-fdf95db70028","order_by":4,"name":"Jacob M. Kolman","email":"","orcid":"","institution":"Houston Methodist","correspondingAuthor":false,"prefix":"","firstName":"Jacob","middleName":"M.","lastName":"Kolman","suffix":""},{"id":485383174,"identity":"069eac7b-e7b9-4d52-b757-c742ae481507","order_by":5,"name":"Raaghav Bageshwar","email":"","orcid":"","institution":"Texas A\u0026M University College of Medicine, Texas A\u0026M College of Engineering and Houston Methodist Hospital","correspondingAuthor":false,"prefix":"","firstName":"Raaghav","middleName":"","lastName":"Bageshwar","suffix":""},{"id":485383175,"identity":"74a3d3f1-2e10-4fc5-966d-eaae5358c1fa","order_by":6,"name":"James L. Lilly","email":"","orcid":"","institution":"Texas A\u0026M University College of Medicine, Texas A\u0026M College of Engineering and Houston Methodist Hospital","correspondingAuthor":false,"prefix":"","firstName":"James","middleName":"L.","lastName":"Lilly","suffix":""},{"id":485383176,"identity":"ea413e82-4b49-4db6-818b-eb19ded9dac9","order_by":7,"name":"Peter K. Lin","email":"","orcid":"","institution":"Texas A\u0026M University College of Medicine, Texas A\u0026M College of Engineering and Houston Methodist Hospital","correspondingAuthor":false,"prefix":"","firstName":"Peter","middleName":"K.","lastName":"Lin","suffix":""},{"id":485383177,"identity":"d382a623-254b-46fa-86ea-5ad1839e4313","order_by":8,"name":"Adriana Lopez","email":"","orcid":"","institution":"Houston Methodist","correspondingAuthor":false,"prefix":"","firstName":"Adriana","middleName":"","lastName":"Lopez","suffix":""},{"id":485383178,"identity":"5860aa09-c19f-45eb-bcd1-1d25c37eb15d","order_by":9,"name":"Stephen L. Jones","email":"","orcid":"","institution":"Houston Methodist","correspondingAuthor":false,"prefix":"","firstName":"Stephen","middleName":"L.","lastName":"Jones","suffix":""},{"id":485383179,"identity":"b26df473-b9b1-4f99-a2a4-19c2e7ed0d36","order_by":10,"name":"Gyusung I. Lee","email":"","orcid":"","institution":"American College of Surgeons","correspondingAuthor":false,"prefix":"","firstName":"Gyusung","middleName":"I.","lastName":"Lee","suffix":""},{"id":485383180,"identity":"21f4eb59-81b5-4ac9-82f0-a2ae15ddcadc","order_by":11,"name":"Ganesh Sankaranarayanan","email":"","orcid":"","institution":"The University of Texas Southwestern Medical Center","correspondingAuthor":false,"prefix":"","firstName":"Ganesh","middleName":"","lastName":"Sankaranarayanan","suffix":""},{"id":485383181,"identity":"7ba9153f-c6d1-40d1-80c2-64efd6ab123c","order_by":12,"name":"Farzan Sasangohar","email":"","orcid":"","institution":"Houston Methodist","correspondingAuthor":false,"prefix":"","firstName":"Farzan","middleName":"","lastName":"Sasangohar","suffix":""},{"id":485383182,"identity":"b5338e95-3362-4b04-a8a5-1674f1547b40","order_by":13,"name":"Dimitrios Stefanidis","email":"","orcid":"","institution":"Indiana University School of Medicine","correspondingAuthor":false,"prefix":"","firstName":"Dimitrios","middleName":"","lastName":"Stefanidis","suffix":""},{"id":485383183,"identity":"e7a4f60f-4567-4ecc-b358-a9d696011c6b","order_by":14,"name":"Randolph H. Steadman","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABMUlEQVRIie3QsWqDQBjA8RPhspx0KwZFX+GOg0gJJc8iB+kibafgaBBuknYtSOkrOBU6ipAsNl1TbjG4pkPokhYLVSmUolKyleIfPTzwx3cKQF/fnw2W9yD43svlhbpfl78ISqsdrhbJrwn8jajOTwK6iHX9FGf72b1pDV/iHBWFYR0ny/zSHevmnSfl7nuD6Gsmk2AlyEN4zqjCMT0Jp7Z/k54hvAAyebxqEFWVoapwIUXCGWmKh+3ygfgKTxCGAA7nQZMcJYO3Dy4m0XM60lBRkYtdTUzeQQCDcjnFjtaoJLCeItUELEri7VsOxqimrwSLUoeSW04pFlMSovpbbJ/MvZaDxZvddiZOo2VKsm1hGFiw7BW544npJ/HGK9r/dFeSByR+GKk6cEpfX1/ff+wTG0BrVxQAProAAAAASUVORK5CYII=","orcid":"","institution":"Houston Methodist","correspondingAuthor":true,"prefix":"","firstName":"Randolph","middleName":"H.","lastName":"Steadman","suffix":""}],"badges":[],"createdAt":"2025-07-09 06:38:05","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7080528/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7080528/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1007/s11701-025-02689-4","type":"published","date":"2025-09-04T15:57:28+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":87028314,"identity":"843a0c2f-3e7e-4cf1-b52a-f0437edf7bd3","added_by":"auto","created_at":"2025-07-18 12:31:00","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":63160,"visible":true,"origin":"","legend":"\u003cp\u003ePRISMA flowchart\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-7080528/v1/e00c31d227e93d590790a733.png"},{"id":90828026,"identity":"40147622-3003-4832-8404-16bf75362813","added_by":"auto","created_at":"2025-09-08 16:05:07","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":962196,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7080528/v1/69c3790c-cc48-4e4a-8e31-8a797f071575.pdf"},{"id":87028317,"identity":"515c5199-c6cd-40a7-aecc-52631f90b9cd","added_by":"auto","created_at":"2025-07-18 12:31:00","extension":"xlsx","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":41596,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementaryFile.xlsx","url":"https://assets-eu.researchsquare.com/files/rs-7080528/v1/9f0fc09215c09734d95a7c93.xlsx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Technical Performance Assessment of Robotic Surgery: A Systematic Scoping Review","fulltext":[{"header":"INTRODUCTION","content":"\u003cp\u003eRobotic surgery has been rapidly adopted worldwide. In a report by Intuitive,[1] the use of robotic surgery for various surgeries has reached above 2.2 million procedures in 2023 which was 22% growth compared with 2022 . Moreover, data shows that there is a migration from laparoscopic and open procedures towards robotic surgery.[2] Robotic approaches are now reported across multiple countries as the dominant method for procedures such as radical prostatectomy.[3\u0026ndash;5]\u003c/p\u003e\n\u003cp\u003eThis rapid increase in robotic surgery has led to the need for training programs and ways to assess performance. Concerns have been raised about the quality of robotic training of surgeons, leading to calls for more robust skill assessments.[6] \u0026nbsp;Requirements for credentialing in robotic surgical procedures are largely left to institutional medical staff offices. These requirements vary considerably across institutions.[7] While the minimum number of robotic cases is commonly used as a key requirement for credentialing, research has shown that surgical volume is not correlated to patient outcomes.[8]\u003c/p\u003e\n\u003cp\u003eThe current state of robotic surgery skill assessment can be classified into two categories, manual or automated.[9] Manual rating involves real-time observation or post hoc video review by a surgical expert, researcher, or crowd-sourced group using a standardized scoring rubric as a guide, which can be global, procedure-specific, or error-based assessments.[10\u0026ndash;19] Manual methods are time-consuming[20] and especially costly as an expenditure of expert surgeons\u0026rsquo; time[21], concerns have also been raised that manual assessment can be cognitively taxing, and prone to bias and inconsistencies.[22, 23] Rating scales are also not designed to identify errors; a low score versus high score can represent trainee progress and benchmarking by various methods,[24] but scalar metrics by themselves do not indicate the critical error or mistake, making them valuable but insufficient for coaching inexperienced surgeons.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eAutomated assessments, typically derived from robotic kinematic tracking data, systems events, and surgical video data, promise to conserve cost, time and effort.[9] While these assessments are more efficient, they currently analyze individual tasks and subtasks in terms of kinematic efficiency (e.g., instrument movement, force applied), creating automated performance metrics (APMs)[11]\u003csup\u003e,\u003c/sup\u003e[14]\u0026nbsp;or Objective Performance Indicators (OPIs).[25\u0026ndash;27] These metrics may also lack the transparency needed for purposes of formative feedback, especially if implemented into an artificial intelligence algorithm.[20]\u0026nbsp;Explicitly defined errors would serve as a complementary means of informing both manual and automated scoring systems.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;The use of automated assessments to distinguish between expertise levels is ongoing, and attention is starting to focus on identifying erroneous motions that contribute to suboptimal performance and errors.[22] However, prior reviews have not attempted to comprehensively identify errors or offer a repository that might bridge the gap between motion- or score-based assessment and error-based assessment. Our goal was to identify, collate, and categorize explicitly defined technical errors, which can provide a basis for informing and advancing scoring systems.\u0026nbsp;\u003c/p\u003e"},{"header":"METHODS","content":"\u003cp\u003eThis review is reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR)[28] with additional reference to the PRISMA 2020 update, as relevant.[29] Additional protocol details and data sheets are available upon reasonable request.\u0026nbsp;\u003c/p\u003e\n\u003ch2\u003eStudy Eligibility\u003c/h2\u003e\n\u003cp\u003eWe included primary empirical studies broadly related to robotic surgical technical skill and performance assessment to capture any concepts related (implicitly or explicitly) to error definition, identification, evaluation, and prediction. Studies in inanimate, simulated or virtual reality (VR), in vivo / ex vivo animal, cadaver, and human intraoperative environments were included. By exception, we also included consensus-setting papers (e.g., Delphi panels) if the topic of consensus related directly to robotic surgical error definition. Any error definition, severity, evaluation, and prediction related to intraoperative technical skills were included.\u003c/p\u003e\n\u003cp\u003eSurgical domains of interest included those where robotics were most established, i.e., general, abdominal, urologic, obstetrics/gynecologic, and thoracic surgery, as well as core surgical skills (e.g., suturing techniques). All other surgical types were excluded given the more nascent introduction of robotics in these domains, e.g., cardiovascular, orthopedic, cranio-facial, spinal, and dental surgeries, whereas the included domains have a more established history of robotic use. For inclusion, articles must have focused on the intraoperative, technical errors or skill assessment of the surgeon or trainee proceduralist; if the study only included postoperatively detectable outcomes or complications (such as positive surgical margins or anastomotic leak), non-procedural or non-surgeon-centric performance (e.g., pure ergonomics, team communication, or the calibration or precision of the robotic device itself), the paper was excluded. We similarly excluded studies focused only on global efficiency outcomes (such as mean operating time or total bleeding), which, though suggestive of skill level, does not specify individual errors. Articles lacking robotics, such as laparoscopic-only skills, were excluded.\u003c/p\u003e\n\u003cp\u003eOnly journal articles with research results were considered; case reports, robotic prototype development papers, protocols, conference papers, and abstracts were excluded. We included preprints only if a peer reviewed version was not available to replace the record by the time of data extraction. Due to limited resources, only English-language articles were included. Studies from 2012 to the search date were included; older studies were excluded to avoid outdated robotic technology and focus on only the latest robotic surgery capabilities in the past decade coupled with advances in machine-readable gesture recognition technology.\u003c/p\u003e\n\u003ch2\u003eSearch Strategy\u003c/h2\u003e\n\u003cp\u003eOur search strategy was refined using paradigm example texts of interest and comments from our team coordinating advisors (see author contributions and acknowledgements). A specialized medical systematic review librarian (MF) then queried MEDLINE (Ovid), Embase (Ovid), and Compendex on May 19, 2022; see \u003cstrong\u003eAppendix A\u003c/strong\u003e for all search strings.\u003c/p\u003e\n\u003ch2\u003eSelection Process\u003c/h2\u003e\n\u003cp\u003eResults were imported into Covidence systematic review software for de-duplication.[30]\u003csup\u003e\u0026nbsp;\u003c/sup\u003eAfter confirming inclusion and exclusion heuristics with senior coordinating advisors, authors in the working group used Covidence to screen titles and abstracts (first stage) and assess full texts for inclusion. Covidence progresses a record through each stage after two screeners concur on inclusion or exclusion (and reason for full text exclusion); conflicts were discussed and resolved in working group consensus meetings.\u003c/p\u003e\n\u003ch2\u003eData Charting\u003c/h2\u003e\n\u003cp\u003eThe working group in consultation with the coordinating advisors drafted a data extraction form in Microsoft Excel, which was tested on a sample of 3-5 included articles per round. After six rounds of refinement, the finalized data charting form was implemented in Covidence where two authors from the working group charted data from included articles, resolving charting discrepancies via consensus meetings while logging heuristics for future article charting.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eWe recorded bibliographic information (date, title, DOI / PMID); country(ies) of study setting; study aim and design (e.g., randomized or nonrandomized interventional; observational; qualitative); surgical domain (general, urology, gynecology; thoracic; and/or core skills if only fundamental inanimate skills were studied); surgical setting (e.g., intraoperative human, VR, inanimate, etc.); type of surgical procedure; participant level of training and any further definition of baseline expertise; number of proceduralists; method of skill assessment (e.g., video recording, live observation, on-board automated simulator metrics); type of rater (e.g., expert, crowd-sourcing); blinding and reported inter-rater reliability, if applicable; any standardized scoring metrics used; presence of any kinematic assessment; specification of errors; and whether specific steps of a surgery were delineated (exhaustively or partially; n/a for basic core skills such as suturing); and the study\u0026rsquo;s conclusion.\u0026nbsp;\u003c/p\u003e\n\u003ch2\u003eData Synthesis\u003c/h2\u003e\n\u003cp\u003eThe authors grouped and cross-tabulated results by key charted elements related to our objectives, including surgical setting, type of rater, and method of skill assessment, and met to evaluate meaningful trends and data presentation. Studies were grouped by those explicitly defining errors (in addition to or distinct from the use of performance scoring scales) versus those using scales only; studies using similar cognate terms (such as \u0026lsquo;mistake\u0026rsquo;, \u0026lsquo;critical failure\u0026rsquo;) were interpreted as defining errors. Of the studies defining errors, two authors (RS, SF) summarized and categorized the errors by type using thematic analysis, meeting with additional authors (SMKM, JK, QZ) to refine categories. An emergent theme approach was utilized (i.e., in which content was grouped without \u003cem\u003ea priori\u0026nbsp;\u003c/em\u003ecategorizations in mind).[31] Due to the heterogeneity and breadth of included results, we did not conduct a critical appraisal.\u003c/p\u003e"},{"header":"RESULTS","content":"\u003cp\u003eA total of 3078 titles were screened. As shown in the PRISMA chart (Figure 1) after removing the duplicates, and screening by the titles and abstracts, 568 full-text articles were reviewed and assessed based on inclusion criteria. Finally 185 titles were included (for full list, see \u003cstrong\u003eAppendix B\u003c/strong\u003e).\u003c/p\u003e\n\u003cp\u003eOver the course of the last decade, there was an increasing trend in the number of publications with 11 in 2012, and 24 in 2021. The majority of included studies (n=110, 59%) had US affiliations followed by the United Kingdom (n=14, 8%), Canada (n=9, 5%), The Netherlands and Japan (each n=6, 3%).\u003c/p\u003e\n\u003cp\u003eStudies were performed in a variety of clinical or skills lab settings (Table 1). The most common robotic settings were simulated surgical procedures using inanimate models (48%), virtual reality (39%), and intraoperative assessments (24%). One fifth of the studies combined two or more surgical settings (e.g., comparing VR to intraoperative or training in both inanimate and VR models).\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eAfter reviewing the study designs, 98 out of 185 (53%) were non-randomized interventional, 38 (21%) were randomized controlled trials and 36 (19%) were observational.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable 1.\u003c/strong\u003e Frequency of Various Surgical Settings\u003c/p\u003e\n\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"358\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 204px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eSurgical Setting\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 154px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eNumbers of Publications (%)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 204px;\"\u003e\n \u003cp\u003eInanimate models\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 154px;\"\u003e\n \u003cp\u003e88 (48)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 204px;\"\u003e\n \u003cp\u003eVirtual Reality\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 154px;\"\u003e\n \u003cp\u003e72 (39)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 204px;\"\u003e\n \u003cp\u003eIntraoperative human\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 154px;\"\u003e\n \u003cp\u003e44 (24)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 204px;\"\u003e\n \u003cp\u003eEx vivo Animal\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 154px;\"\u003e\n \u003cp\u003e8 (4)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 204px;\"\u003e\n \u003cp\u003eIn vivo Animal\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 154px;\"\u003e\n \u003cp\u003e11 (6)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 204px;\"\u003e\n \u003cp\u003eCadaver\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 154px;\"\u003e\n \u003cp\u003e5 (3)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 204px;\"\u003e\n \u003cp\u003eNot applicable\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 154px;\"\u003e\n \u003cp\u003e3 (2)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eNote: Total is \u0026gt; 100% of articles due to studies using multiple settings (36 articles). Percentage based on the total number of included articles (N = 185).\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003ch2\u003eTypes of Rater\u003c/h2\u003e\n\u003cp\u003eWe compared the types of performance assessment raters across the three most frequently used surgical settings. The types of raters included human raters (n=137, 74%) and automated computer algorithms (n=85, 46%; see Table 2). Some studies (n=36, 19%) incorporated more than one type of rater. Considering the surgical settings of the studies, automated ratings were used more in inanimate vs intraoperative settings, vs expert raters.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable 2.\u003c/strong\u003e Rater Type by Surgical Setting\u003c/p\u003e\n\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"592\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 173px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eType of rater used for skill assessment\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 103px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eNumber of articles (n, %)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 103px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eArticles with an inanimate setting (n, %)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 106px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eArticles involving virtual reality (n, %)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 107px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eArticles with intraoperative setting (n, %)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 173px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eHuman rating\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 103px;\"\u003e\n \u003cp\u003e137 (74)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 103px;\"\u003e\n \u003cp\u003e61 (33)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 106px;\"\u003e\n \u003cp\u003e32 (17)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 107px;\"\u003e\n \u003cp\u003e44 (24)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 173px;\"\u003e\n \u003cp\u003eExpert rater\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 103px;\"\u003e\n \u003cp\u003e92 (49)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 103px;\"\u003e\n \u003cp\u003e40 (22)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 106px;\"\u003e\n \u003cp\u003e20 (11)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 107px;\"\u003e\n \u003cp\u003e32 (17)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 173px;\"\u003e\n \u003cp\u003eCrowdsourcing\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 103px;\"\u003e\n \u003cp\u003e17 (9)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 103px;\"\u003e\n \u003cp\u003e7 (4)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 106px;\"\u003e\n \u003cp\u003e3 (2)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 107px;\"\u003e\n \u003cp\u003e7 (4）\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 173px;\"\u003e\n \u003cp\u003eNon-Expert rater\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 103px;\"\u003e\n \u003cp\u003e10 (5)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 103px;\"\u003e\n \u003cp\u003e2 (1)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 106px;\"\u003e\n \u003cp\u003e4 (2)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 107px;\"\u003e\n \u003cp\u003e4 (2)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 173px;\"\u003e\n \u003cp\u003eNot reported in detail*\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 103px;\"\u003e\n \u003cp\u003e18 (11)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 103px;\"\u003e\n \u003cp\u003e12 (6)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 106px;\"\u003e\n \u003cp\u003e5 (3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 107px;\"\u003e\n \u003cp\u003e1 (1)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 173px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eAutomated rating\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 103px;\"\u003e\n \u003cp\u003e85 (46)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 103px;\"\u003e\n \u003cp\u003e18 (10)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 106px;\"\u003e\n \u003cp\u003e61 (33)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 107px;\"\u003e\n \u003cp\u003e6 (3)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 173px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eOther\u003csup\u003e\u0026dagger;\u003c/sup\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 103px;\"\u003e\n \u003cp\u003e5 (3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 103px;\"\u003e\n \u003cp\u003e3 (2)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 106px;\"\u003e\n \u003cp\u003e0 (0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 107px;\"\u003e\n \u003cp\u003e2 (1)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 173px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eNot applicable\u003csup\u003e\u0026Dagger;\u003c/sup\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 103px;\"\u003e\n \u003cp\u003e4 (2)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 103px;\"\u003e\n \u003cp\u003e4 (2)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 106px;\"\u003e\n \u003cp\u003e0 (0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 107px;\"\u003e\n \u003cp\u003e4 (2)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 173px;\"\u003e\n \u003cp\u003eSum\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 103px;\"\u003e\n \u003cp\u003e231 (125)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 103px;\"\u003e\n \u003cp\u003e86 (46)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 106px;\"\u003e\n \u003cp\u003e93 (50)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 107px;\"\u003e\n \u003cp\u003e56 (30)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eNote: Total is \u0026gt; 100% due to studies using multiple rating types (36 articles). Percentage was based on the total number of included articles (N = 185).\u003c/p\u003e\n\u003cp\u003e*Human raters, non-specific to expertise\u003c/p\u003e\n\u003cp\u003e\u003csup\u003e\u0026dagger;\u003c/sup\u003eInvolved alterations of automated data (e.g., author-supplied kinematics equation)\u003c/p\u003e\n\u003cp\u003e\u003csup\u003e\u0026Dagger;\u003c/sup\u003eArticle contains error definitions but no rating or direct assessment, e.g. Delphi study\u0026nbsp;\u003c/p\u003e\n\u003ch2\u003eExplicitly Defined Errors\u003c/h2\u003e\n\u003cp\u003eExplicitly defined errors were identified and labeled as errors by the manuscript authors independently of whether they were listed in previously-defined rating scales.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eFrom the total of 185, 104 (56%) articles explicitly defined performance errors (Table 3). Sixty-four studies exclusively used predefined scales (Table 4), the majority of which used GEARS followed by OSATS.\u003c/p\u003e\n\u003cp\u003eWe categorized author-defined explicit errors into seven themes (Table 3). The most frequent categories of explicitly defined errors were suture / needle placement errors (33%), tissue injury due to excessive force (32%), drops (23%), and instrument-movement related errors (23%). While these categories were mostly distinct, which made errors easy to attribute to a category, arguably some categories could have been merged, for instance suture and needle breakage and tissue injury are both suggestive of excessive force. Also, as shown in table 3, the majority of the articles which defined error categories used inanimate and VR settings, rather than intraoperative. However, the most common error category studied within each setting was not the same. We found \u0026ldquo;Suturing and needle placement\u0026rdquo; errors under inanimate settings (29.4%), \u0026ldquo;Instrument movement-related errors\u0026rdquo; under VR (28.2%), and \u0026ldquo;Tissue injury due to excessive force\u0026rdquo; in intraoperative setting (50%), were the most common error categories within each of those settings.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable 3\u003c/strong\u003e. Frequency of articles that explicitly defined errors and their settings\u003c/p\u003e\n\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"624\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 170px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eError category\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 114px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eNumber of articles (n)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 114px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eArticles with an inanimate setting (n)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 108px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eArticles involving virtual reality (n)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 118px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eArticles with intraoperative setting (n)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 170px;\"\u003e\n \u003cp\u003eSuturing and needle placement errors (including knot tying)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 114px;\"\u003e\n \u003cp\u003e31\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 114px;\"\u003e\n \u003cp\u003e25\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 108px;\"\u003e\n \u003cp\u003e5\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 118px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 170px;\"\u003e\n \u003cp\u003eTissue injury due to excessive force (including damage, tearing, and injuries)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 114px;\"\u003e\n \u003cp\u003e30\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 114px;\"\u003e\n \u003cp\u003e14\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 108px;\"\u003e\n \u003cp\u003e8\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 118px;\"\u003e\n \u003cp\u003e8\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 170px;\"\u003e\n \u003cp\u003eDrops (objects and needles)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 114px;\"\u003e\n \u003cp\u003e24\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 114px;\"\u003e\n \u003cp\u003e19\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 108px;\"\u003e\n \u003cp\u003e5\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 118px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 170px;\"\u003e\n \u003cp\u003eInstrument movement-related errors (including collisions, missed targets, depth perception)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 114px;\"\u003e\n \u003cp\u003e24\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 114px;\"\u003e\n \u003cp\u003e13\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 108px;\"\u003e\n \u003cp\u003e11\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 118px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 170px;\"\u003e\n \u003cp\u003eRobot control/camera control/instruments out of view\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 114px;\"\u003e\n \u003cp\u003e15\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 114px;\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 108px;\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 118px;\"\u003e\n \u003cp\u003e3\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 170px;\"\u003e\n \u003cp\u003eJudgment error (including autonomy)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 114px;\"\u003e\n \u003cp\u003e10\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 114px;\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 108px;\"\u003e\n \u003cp\u003e2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 118px;\"\u003e\n \u003cp\u003e2\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 170px;\"\u003e\n \u003cp\u003eUse of rating scale (combined with explicitly defined errors)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 114px;\"\u003e\n \u003cp\u003e6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 114px;\"\u003e\n \u003cp\u003e2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 108px;\"\u003e\n \u003cp\u003e2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 118px;\"\u003e\n \u003cp\u003e2\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 170px;\"\u003e\n \u003cp\u003eTotal\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 114px;\"\u003e\n \u003cp\u003e140\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 114px;\"\u003e\n \u003cp\u003e85\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 108px;\"\u003e\n \u003cp\u003e39\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 118px;\"\u003e\n \u003cp\u003e16\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eNote: Total exceeds 104 articles due to articles containing more than one explicitly defined error or surgical setting (i.e., 140 instances of explicitly defined errors were found in 104 of the included articles).\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eWe found 64 articles that used performance rating scales. The details are shown in Table 4. The most common rating scales were GEARS and OSATS, used in 75% and 50% of the articles, respectively. GEARS was equally utilized to rate intraoperative and inanimate settings while OSATS was mostly used for inanimate surgical settings. \u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable 4.\u003c/strong\u003e Performance Rating Scales Used in Robotic Studies\u003c/p\u003e\n\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"625\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 215px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eRating scale\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 81px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eTotal articles (n, %)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 102px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eArticles with an inanimate setting (n, %)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 111px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eArticles involving virtual reality\u003c/strong\u003e\u003c/p\u003e\n \u003cp\u003e\u003cstrong\u003e(n, %)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 116px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eArticles with intraoperative setting\u003c/strong\u003e\u003c/p\u003e\n \u003cp\u003e\u003cstrong\u003e(n, %)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 215px;\"\u003e\n \u003cp\u003eGEARS (Global Evaluative Assessment of Robotic Skills)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 81px;\"\u003e\n \u003cp\u003e48 (75)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 102px;\"\u003e\n \u003cp\u003e8 (13)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 111px;\"\u003e\n \u003cp\u003e15 (23)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 116px;\"\u003e\n \u003cp\u003e25 (39)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 215px;\"\u003e\n \u003cp\u003eOSATS (Objective Structured Assessment of Technical Skills)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 81px;\"\u003e\n \u003cp\u003e32 (50)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 102px;\"\u003e\n \u003cp\u003e16 (25)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 111px;\"\u003e\n \u003cp\u003e6 (9)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 116px;\"\u003e\n \u003cp\u003e2 (3)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 215px;\"\u003e\n \u003cp\u003eGOALS (Global Objective Assessment of Laparoscopic Skills)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 81px;\"\u003e\n \u003cp\u003e8 (13)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 102px;\"\u003e\n \u003cp\u003e2 (3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 111px;\"\u003e\n \u003cp\u003e4 (6)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 116px;\"\u003e\n \u003cp\u003e2 (3)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 215px;\"\u003e\n \u003cp\u003eRACE (Robotic Anastomosis Competency Evaluation)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 81px;\"\u003e\n \u003cp\u003e7 (11)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 102px;\"\u003e\n \u003cp\u003e2 (3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 111px;\"\u003e\n \u003cp\u003e2 (3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 116px;\"\u003e\n \u003cp\u003e3 (5)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 215px;\"\u003e\n \u003cp\u003eGERT (Generic Error Rating Tool)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 81px;\"\u003e\n \u003cp\u003e2 (3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 102px;\"\u003e\n \u003cp\u003e0 (0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 111px;\"\u003e\n \u003cp\u003e0 (0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 116px;\"\u003e\n \u003cp\u003e2 (3)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 215px;\"\u003e\n \u003cp\u003ePACE (Prostatectomy Assessment and Competence Evaluation)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 81px;\"\u003e\n \u003cp\u003e3 (5)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 102px;\"\u003e\n \u003cp\u003e2 (3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 111px;\"\u003e\n \u003cp\u003e0 (0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 116px;\"\u003e\n \u003cp\u003e1 (2)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 215px;\"\u003e\n \u003cp\u003eRHAS (robotic hysterectomy assessment score)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 81px;\"\u003e\n \u003cp\u003e1 (2)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 102px;\"\u003e\n \u003cp\u003e0 (0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 111px;\"\u003e\n \u003cp\u003e0 (0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 116px;\"\u003e\n \u003cp\u003e1 (2)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 215px;\"\u003e\n \u003cp\u003eRARP (Robot-assisted Radical Prostatectomy) Assessment Score\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 81px;\"\u003e\n \u003cp\u003e2 (3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 102px;\"\u003e\n \u003cp\u003e1 (2)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 111px;\"\u003e\n \u003cp\u003e0 (0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 116px;\"\u003e\n \u003cp\u003e1 (2)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"bottom\" style=\"width: 215px;\"\u003e\n \u003cp\u003eAPMs (Automated performance metrics)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 81px;\"\u003e\n \u003cp\u003e2 (3)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 102px;\"\u003e\n \u003cp\u003e0 (0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 111px;\"\u003e\n \u003cp\u003e0 (0)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"bottom\" style=\"width: 116px;\"\u003e\n \u003cp\u003e2 (3)\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eNote: Percentage out of 64 articles that included rating scales (total exceeds 100% due to some studies involving multiple scales)\u0026nbsp;\u003c/p\u003e"},{"header":"DISCUSSION","content":"\u003ch2\u003eKey Findings\u003c/h2\u003e\n\u003cp\u003eOur scoping review included studies of technical skill assessment during human, animal, inanimate and virtual robotic surgery, with a focus on identifying defined intraoperative technical performance errors. Approximately half of the included studies explicitly defined errors, while the remaining studies used rating scales or automated scoring without explicitly defining performance errors.\u003c/p\u003e\n\u003cp\u003eWe categorized the explicitly defined errors into seven themes. Although each of the category labels appear in various studies, the use of these seven categories as distinct themes for all the explicitly defined errors we encountered is novel. Categorizing explicit errors into a comprehensive set of themes may facilitate automated assessment of errors with emerging technology and help advance this technology.\u003c/p\u003e\n\u003cp\u003eThe method of rating was split between human raters (74%) and automated ratings (46%), inclusive of studies employing multiple rating methods. As expected, automated ratings predominated in VR-based assessments, while human raters (specifically experts) predominated in inanimate and intraoperative settings.\u003c/p\u003e\n\u003ch2\u003eComparison with the Literature\u003c/h2\u003e\n\u003cp\u003eSeveral recently published systematic reviews identified measures of performance in robotic assisted surgery,[20, 32\u0026ndash;34] commonly grouping assessment tools into manual and automated categories similar to our approach.[20, 33, 34] These reviews included studies using error-based assessments. However, none of these systematic reviews attempted to comprehensively list and categorize each of the errors specified within their included studies, in some cases only counting the number of studies using errors in the assessment method with no further synthesis on that topic. [33] Furthermore, none distinguished intraoperative errors from errors diagnosed postoperatively. For instance, Chen et al. included as \u0026ldquo;technical errors\u0026rdquo; both tissue trauma, attesting to a theme we also synthesized, and what we regarded as postoperative findings such as positive surgical margins.[20] We included only intraoperative errors in order to focus on errors available for detection by automated systems during the procedure rather than post hoc findings such as positive surgical margins or incontinence. Breadth of inclusion also depends on clear nomenclature which has by no means been standardized; Younes et al. for instance, included only nine studies versus our 185 based on narrower search criteria (clinically relevant performance metrics, or CRPM) and excluded APMs, unlike other reviews [20, 20, 33] including our own, despite potential relevance of APMs to technical assessment.\u003c/p\u003e\n\u003cp\u003eOur synthesized list of error themes could be compared to others. Of the manual rating tools noted in Table 4, one was explicitly labeled as a \u0026ldquo;generic error rating tool\u0026rdquo; (GERT), developed for laparoscopic hysterectomy. It contained four error modes: too much force or distance, too little force or distance, inadequate visualization and wrong orientation of the instrument. Each error was counted regardless of sequelae such as bleeding or tissue damage, which deviates from the semantics used by most robotic rating scales.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eIn a study focused on automated detection of suturing and needle-passing errors, Hutchinson et al.[22] proposed classifying errors into three broad categories\u0026ndash;efficiency, safety, and other task metrics (e.g., camera movement). Though grouped differently, their prime examples (multiple attempts, needle drop, needle orientation, and out of view), also fit within our categorization, though we did find the distinction between efficiency (i.e., what is optimal versus non-optimal) and error (what is a mistake versus acceptable) a difficult line to draw in practice.\u003c/p\u003e\n\u003cp\u003eAlthough many authors equate time to completion (efficiency) as a measure of skill, we specifically excluded time to completion as a skill assessment because it excludes performance quality and a threshold is not easily defined for competency. Operative time has \u0026ldquo;traditionally been used as a proxy for surgical skill\u0026rdquo; [35]; however, studies evaluating surgical skill show little correlation between skill ratings and operative time.[35] Of note, task-specific time penalties were encountered in some reviews; however, we elected not to classify time cut-offs as errors as none of the study authors explicitly defined them as such.\u003c/p\u003e\n\u003ch2\u003eThe State of Performance Assessment\u0026nbsp;\u003c/h2\u003e\n\u003cp\u003eThe cited reviews concluded similarly that universally accepted robotic skills\u0026rsquo; assessments [9] or performance metrics[32] do not currently exist, are not well validated[33], and that a standardized objective metric of technical performance is required.[34]\u003c/p\u003e\n\u003cp\u003eWe designed this review with an intent to scope the literature to find the state of technical skill assessment and error definition and likewise conclude that there is no consensus on error definition in literature. However, the future for automated scoring systems is very promising. Other reviews argued that more objectivity was needed in assessments, with some specifically mentioning APMs as offering promise, but needing further work.\u003c/p\u003e\n\u003ch2\u003eLimitations\u003c/h2\u003e\n\u003cp\u003eWe elected to scope nine years of literature (2012-2021) which was intended to capture work on APMs which peaked 2012 onwards. While our focus on intraoperative skill assessment can be seen as a significant limitation this was intentional, as it serves as a foundational basis for future work using computer vision. We did not include studies looking at short and long term clinical outcomes such as postoperative anastomotic leak, urinary incontinence and positive surgical margins as these outcomes are not detectable by contemporaneous automated assessments. Likewise, we excluded nontechnical performance (e.g., cognitive performance and mental workload; workflow interruptions, team communication and coordination metrics), which is recognized as an important component in overall performance (as in the counterpoint argument for more holistic, multi-level measurement by El-Sayed 2024 noted above).\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eNo attempt was made to assess bias or quality as our focus was on identifying gaps in the literature rather than demonstrating strength of conclusions. Lastly, we only included English-language sources; however, only three non-English records were excluded at screening based upon language, so the potential for bias based on language criteria is minimal.\u003c/p\u003e"},{"header":"Conclusion and Proposed Future Work","content":"\u003cp\u003eOur review provides a snapshot of the current state of technical skill assessment in robotic surgery and helps to identify the gaps in the literature that can inform skill assessment methods in the future. We found considerable variability in performance assessment methods and error identification, with little conceptual correlation. Intraoperative performance assessment is dominated by manual assessments, with a paucity of automated assessments developed for use in the operating room.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eWe believe the future of robotic surgical skill assessment will incorporate automated skill assessment because of its objectivity: It is more reproducible, reliable and faster than manual assessments. Future work should be directed to aligning errors with APMs, while incorporating computer vision, machine learning and deep learning methods. Automated techniques may not provide surgeons with real-time feedback as consistently as manual assessment; however, consensus definitions of intraoperative technical errors may provide context for such feedback and potentially allow immediate correction. Nonetheless, our review showed that additional work is needed to validate and standardize robotic skill assessment.\u0026nbsp;\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\n\u003cp\u003eD.S, G.L., G.S. and R.S conceptualized the study. SM.KM., M.F and J.K. provided the methodology. SM. KM.,S.F., Q.Z.,M.F., J.K., R.B., J.L., P.L and R.S., did the literature search and review and Data extraction. SM. KM.,S.F., Q.Z.,M.F., J.K. and R.S., did the critical analysis, interpreted the results and drafted the manuscript. All authors reviewed and edited the manuscript.\u003c/p\u003e\n\u003ch2\u003eAcknowledgement\u003c/h2\u003e\n\u003cp\u003eWe thank Dr. Ajit Sachdeva of the American College of Surgeons for valuable feedback on a draft of this manuscript and for inspiring us to pursue this topic.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eIntuitive Announces Preliminary Fourth Quarter and Full Year 2023 Results | Intuitive Surgical. https://isrg.intuitive.com/news-releases/news-release-details/intuitive-announces-preliminary-fourth-quarter-and-full-year-3/. Accessed 26 Jun 2024\u003c/li\u003e\n\u003cli\u003eSheetz KH, Claflin J, Dimick JB (2020) Trends in the adoption of robotic surgery for common surgical procedures. JAMA Netw Open 3:e1918911. https://doi.org/10.1001/jamanetworkopen.2019.18911\u003c/li\u003e\n\u003cli\u003eBasto M, Sathianathen N, Te Marvelde L, Ryan S, Goad J, Lawrentschuk N, Costello AJ, Moon DA, Heriot AG, Butler J, Murphy DG (2016) Patterns-of-care and health economic analysis of robot-assisted radical prostatectomy in the Australian public health system. BJU Int 117:930\u0026ndash;939. https://doi.org/10.1111/bju.13317\u003c/li\u003e\n\u003cli\u003eChae J, Choi Y, Cho S-J (2021) Changes in Patterns of Radical Prostatectomy due to Diffusion of Robotic Surgical System: A Nationwide Study Using Health Insurance Claims Data. Yonsei Med J 62:1155\u0026ndash;1161. https://doi.org/10.3349/ymj.2021.62.12.1155\u003c/li\u003e\n\u003cli\u003eParsons JK, Messer K, Palazzi K, Stroup SP, Chang D (2014) Diffusion of surgical innovations, patient safety, and minimally invasive radical prostatectomy. JAMA Surg 149:845\u0026ndash;851. https://doi.org/10.1001/jamasurg.2014.31\u003c/li\u003e\n\u003cli\u003eLee JY, Mucksavage P, Sundaram CP, McDougall EM (2011) Best practices for robotic surgery training and credentialing. J Urol 185:1191\u0026ndash;1197. https://doi.org/10.1016/j.juro.2010.11.067\u003c/li\u003e\n\u003cli\u003eHuffman EM, Rosen SA, Levy JS, Martino MA, Stefanidis D (2021) Are current credentialing requirements for robotic surgery adequate to ensure surgeon proficiency? Surg Endosc 35:2104\u0026ndash;2109. https://doi.org/10.1007/s00464-020-07608-2\u003c/li\u003e\n\u003cli\u003eLarson DW, Marcello PW, Larach SW, Wexner SD, Park A, Marks J, Senagore AJ, Thorson AG, Young-Fadok TM, Green E, Sargent DJ, Nelson H (2008) Surgeon volume does not predict outcomes in the setting of technical credentialing: results from a randomized trial in colon cancer. Ann Surg 248:746\u0026ndash;750. https://doi.org/10.1097/SLA.0b013e31818a157d\u003c/li\u003e\n\u003cli\u003eChen J, Cheng N, Cacciamani G, Oh P, Lin-Brande M, Remulla D, Gill IS, Hung AJ (2019) Objective Assessment of Robotic Surgical Technical Skill: A Systematic Review. J Urol 201:461\u0026ndash;469. https://doi.org/10.1016/j.juro.2018.06.078\u003c/li\u003e\n\u003cli\u003eBonrath EM, Zevin B, Dedy NJ, Grantcharov TP (2013) Error rating tool to identify and analyse technical errors and events in laparoscopic surgery. Br J Surg 100:1080\u0026ndash;1088. https://doi.org/10.1002/bjs.9168\u003c/li\u003e\n\u003cli\u003eChen J, Oh PJ, Cheng N, Shah A, Montez J, Jarc A, Guo L, Gill IS, Hung AJ (2018) Use of automated performance metrics to measure surgeon performance during robotic vesicourethral anastomosis and methodical development of a training tutorial. J Urol 200:895\u0026ndash;902. https://doi.org/10.1016/j.juro.2018.05.080\u003c/li\u003e\n\u003cli\u003eFrederick PJ, Szender JB, Hussein AA, Kesterson JP, Shelton JA, Anderson TL, Barnabei VM, Guru K (2017) Surgical Competency for Robot-Assisted Hysterectomy: Development and Validation of a Robotic Hysterectomy Assessment Score (RHAS). J Minim Invasive Gynecol 24:55\u0026ndash;61. https://doi.org/10.1016/j.jmig.2016.10.004\u003c/li\u003e\n\u003cli\u003eGoh AC, Goldfarb DW, Sander JC, Miles BJ, Dunkin BJ (2012) Global evaluative assessment of robotic skills: validation of a clinical assessment tool to measure robotic surgical skills. J Urol 187:247\u0026ndash;252. https://doi.org/10.1016/j.juro.2011.09.032\u003c/li\u003e\n\u003cli\u003eHung AJ, Chen J, Che Z, Nilanon T, Jarc A, Titus M, Oh PJ, Gill IS, Liu Y (2018) Utilizing Machine Learning and Automated Performance Metrics to Evaluate Robot-Assisted Radical Prostatectomy Performance and Predict Outcomes. J Endourol 32:438\u0026ndash;444. https://doi.org/10.1089/end.2018.0035\u003c/li\u003e\n\u003cli\u003eHussein AA, Ghani KR, Peabody J, Sarle R, Abaza R, Eun D, Hu J, Fumo M, Lane B, Montgomery JS, Hinata N, Rooney D, Comstock B, Chan HK, Mane SS, Mohler JL, Wilding G, Miller D, Guru KA, Michigan Urological Surgery Improvement Collaborative and Applied Technology Laboratory for Advanced Surgery Program (2017) Development and Validation of an Objective Scoring Tool for Robot-Assisted Radical Prostatectomy: Prostatectomy Assessment and Competency Evaluation. J Urol 197:1237\u0026ndash;1244. https://doi.org/10.1016/j.juro.2016.11.100\u003c/li\u003e\n\u003cli\u003eLovegrove C, Novara G, Mottrie A, Guru KA, Brown M, Challacombe B, Popert R, Raza J, Van Der Poel H, Peabody J, Dasgupta P, Ahmed K (2016) Structured and Modular Training Pathway for Robot-assisted Radical Prostatectomy (RARP): Validation of the RARP Assessment Score and Learning Curve Assessment. Eur Urol 69:526\u0026ndash;535. https://doi.org/10.1016/j.eururo.2015.10.048\u003c/li\u003e\n\u003cli\u003eMartin JA, Regehr G, Reznick R, Macrae H, Murnaghan J, Hutchison C, Brown M (1997) Objective structured assessment of technical skill (OSATS) for surgical residents: OBJECTIVE STRUCTURED ASSESSMENT OF TECHNICAL SKILL. Br J Surg 84:273\u0026ndash;278. https://doi.org/10.1046/j.1365-2168.1997.02502.x\u003c/li\u003e\n\u003cli\u003eRaza SJ, Field E, Jay C, Eun D, Fumo M, Hu JC, Lee D, Mehboob Z, Nyquist J, Peabody JO, Sarle R, Stricker H, Yang Z, Wilding G, Mohler JL, Guru KA (2015) Surgical Competency for Urethrovesical Anastomosis During Robot-assisted Radical Prostatectomy: Development and Validation of the Robotic Anastomosis Competency Evaluation. Urology 85:27\u0026ndash;32. https://doi.org/10.1016/j.urology.2014.09.017\u003c/li\u003e\n\u003cli\u003eVassiliou MC, Feldman LS, Andrew CG, Bergman S, Leffondr\u0026eacute; K, Stanbridge D, Fried GM (2005) A global assessment tool for evaluation of intraoperative laparoscopic skills. Am J Surg 190:107\u0026ndash;113. https://doi.org/10.1016/j.amjsurg.2005.04.004\u003c/li\u003e\n\u003cli\u003eChen J, Cheng N, Cacciamani G, Oh P, Lin-Brande M, Remulla D, Gill IS, Hung AJ (2019) Objective assessment of robotic surgical technical skill: a systematic review. J Urol 201:461\u0026ndash;469. https://doi.org/10.1016/j.juro.2018.06.078\u003c/li\u003e\n\u003cli\u003eGhani KR, Miller DC, Linsell S, Brachulis A, Lane B, Sarle R, Dalela D, Menon M, Comstock B, Lendvay TS, Montie J, Peabody JO, Michigan Urological Surgery Improvement Collaborative (2016) Measuring to Improve: Peer and Crowd-sourced Assessments of Technical Skill with Robot-assisted Radical Prostatectomy. Eur Urol 69:547\u0026ndash;550. https://doi.org/10.1016/j.eururo.2015.11.028\u003c/li\u003e\n\u003cli\u003eHutchinson K, Li Z, Cantrell LA, Schenkman NS, Alemzadeh H (2022) Analysis of executional and procedural errors in dry-lab robotic surgery experiments. Int J Med Robot Comput Assist Surg MRCAS 18:e2375. https://doi.org/10.1002/rcs.2375\u003c/li\u003e\n\u003cli\u003eOh D, Brown K, Yousaf S, Nesbitt J, Feins R, Sancheti M, Lin J, Yang S, D\u0026rsquo;Souza D, Jarc A (2023) Differences Between Attending and Trainee Surgeon Performance Using Objective Performance Indicators During Robot-Assisted Lobectomy. Innov Phila Pa 18:479\u0026ndash;488. https://doi.org/10.1177/15569845231204607\u003c/li\u003e\n\u003cli\u003eGoldenberg MG, Garbens A, Szasz P, Hauer T, Grantcharov TP (2017) Systematic review to establish absolute standards for technical performance in surgery. Br J Surg 104:13\u0026ndash;21. https://doi.org/10.1002/bjs.10313\u003c/li\u003e\n\u003cli\u003eLazar JF, Brown K, Yousaf S, Jarc A, Metchik A, Henderson H, Feins RH, Sancheti MS, Lin J, Yang S, Nesbitt J, D\u0026rsquo;Souza D, Oh DS (2023) Objective performance indicators of cardiothoracic residents are associated with vascular injury during robotic-assisted lobectomy on porcine models. J Robot Surg 17:669\u0026ndash;676. https://doi.org/10.1007/s11701-022-01476-9\u003c/li\u003e\n\u003cli\u003eDevin CL, Gillani M, Shields MC, Eldredge K, Kucera W, Rupji M, Purvis LA, Paul Olson TJ, Liu Y, Jarc A, Rosen SA (2023) Ratio of Economy of Motion: A New Objective Performance Indicator to Assign Consoles During Dual-Console Robotic Proctectomy. Am Surg 89:3416\u0026ndash;3422. https://doi.org/10.1177/00031348231161767\u003c/li\u003e\n\u003cli\u003eKaoukabani G, Gokcal F, Fanta A, Liu X, Shields M, Stricklin C, Friedman A, Kudsi OY (2023) A multifactorial evaluation of objective performance indicators and video analysis in the context of case complexity and clinical outcomes in robotic-assisted cholecystectomy. Surg Endosc 37:8540\u0026ndash;8551. https://doi.org/10.1007/s00464-023-10432-z\u003c/li\u003e\n\u003cli\u003eTricco AC, Lillie E, Zarin W, O\u0026rsquo;Brien KK, Colquhoun H, Levac D, Moher D, Peters MDJ, Horsley T, Weeks L, Hempel S, Akl EA, Chang C, McGowan J, Stewart L, Hartling L, Aldcroft A, Wilson MG, Garritty C, Lewin S, Godfrey CM, Macdonald MT, Langlois EV, Soares-Weiser K, Moriarty J, Clifford T, Tun\u0026ccedil;alp \u0026Ouml;, Straus SE (2018) PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med 169:467\u0026ndash;473. https://doi.org/10.7326/M18-0850\u003c/li\u003e\n\u003cli\u003ePage MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hr\u0026oacute;bjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, Stewart LA, Thomas J, Tricco AC, Welch VA, Whiting P, Moher D (2021) The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ n71. https://doi.org/10.1136/bmj.n71\u003c/li\u003e\n\u003cli\u003eCovidence systematic review software, Veritas Health Innovation, Melbourne, Australia. Available at www.covidence.org. https://support.covidence.org/help/how-can-i-cite-covidence. Accessed 26 Jun 2024\u003c/li\u003e\n\u003cli\u003eBraun V, Clarke V (2006) Using thematic analysis in psychology. Qual Res Psychol 3:77\u0026ndash;101. https://doi.org/10.1191/1478088706qp063oa\u003c/li\u003e\n\u003cli\u003eYounes MM, Larkins K, To G, Burke G, Heriot A, Warrier S, Mohan H (2022) What are clinically relevant performance metrics in robotic surgery? A systematic review of the literature. J Robot Surg 17:335\u0026ndash;350. https://doi.org/10.1007/s11701-022-01457-y\u003c/li\u003e\n\u003cli\u003eBoal MWE, Anastasiou D, Tesfai F, Ghamrawi W, Mazomenos E, Curtis N, Collins JW, Sridhar A, Kelly J, Stoyanov D, Francis NK (2024) Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review. Br J Surg 111:znad331. https://doi.org/10.1093/bjs/znad331\u003c/li\u003e\n\u003cli\u003eEl-Sayed C, Yiu A, Burke J, Vaughan-Shaw P, Todd J, Lin P, Kasmani Z, Munsch C, Rooshenas L, Campbell M, Bach SP (2024) Measures of performance and proficiency in robotic assisted surgery: a systematic review. J Robot Surg 18:16. https://doi.org/10.1007/s11701-023-01756-y\u003c/li\u003e\n\u003cli\u003eAddison P, Yoo A, Duarte-Ramos J, Addy J, Dechario S, Husk G, Jarrett M, Teixeira J, Antonacci A, Filicori F (2021) Correlation between operative time and crowd-sourced skills assessment for robotic bariatric surgery. Surg Endosc 35:5303\u0026ndash;5309. https://doi.org/10.1007/s00464-020-08019-z\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"journal-of-robotic-surgery","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"jors","sideBox":"Learn more about [Journal of Robotic Surgery](http://link.springer.com/journal/11701)","snPcode":"11701","submissionUrl":"https://submission.nature.com/new-submission/11701/3","title":"Journal of Robotic Surgery","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Robotic Surgical Procedure, Surgical Error, Task Performance, Clinical Competence, Intraoperative Period","lastPublishedDoi":"10.21203/rs.3.rs-7080528/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7080528/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003eBackground\u003c/strong\u003e: Defining performance errors in robotic surgery is critical for the assessment of robotic surgery skill. Our goal was to identify and categorize explicitly defined intraoperative technical errors in robotic surgery, how skill assessment was performed, and how ratings were conducted either manually by experts or via automated ratings.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eStudy Design\u003c/strong\u003e: This scoping review included studies involving general, urologic, obstetrics/gynecologic, and thoracic surgery, and general skills as practiced in inanimate, virtual reality, in vivo / ex vivo animal, cadaver, and human operations. Primary empirical and consensus-building studies were included if they addressed intraoperative performance assessment or error definition and identification. MEDLINE (Ovid), Embase (Ovid), and Compendex were queried for results from 2012 to May 19, 2022.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResults\u003c/strong\u003e: Of 2,642 studies screened, 185 were included. The majority (n=109, 60%) were US-based and involved either simulated surgical procedures using inanimate models (n=88), virtual reality (n=72), or intraoperative performance assessments of robotic surgeries in humans (n=44); 36 studies combined two or more of these settings. Performance errors were explicitly defined in 104 articles (56%), and 64 used previously defined performance rating scales. The method of rating was split between manual (n=137) and automated ratings (n=85).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConclusion\u003c/strong\u003e: Measures of performance vary considerably. More conceptual work is warranted to explicitly define errors that can inform robotic skill assessment. This is important given the growing interest in developing efficient and reliable objective measures of performance which are likely to rely on automated assessment methods.\u003c/p\u003e","manuscriptTitle":"Technical Performance Assessment of Robotic Surgery: A Systematic Scoping Review","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-07-18 12:30:55","doi":"10.21203/rs.3.rs-7080528/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Accepted","date":"2025-08-16T10:46:45+00:00","index":"","fulltext":""},{"type":"reviewerAgreed","content":"58625837610102992802301846938314578459","date":"2025-08-12T06:34:54+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-07-31T09:38:07+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"274196373407919095228977803644597778664","date":"2025-07-17T16:35:27+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"323803319501953555893973701298489430266","date":"2025-07-14T18:23:38+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"335597128372250941729692162971997798189","date":"2025-07-12T05:21:06+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-07-10T02:20:04+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-07-10T01:57:55+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-07-09T09:29:01+00:00","index":"","fulltext":""},{"type":"submitted","content":"Journal of Robotic Surgery","date":"2025-07-09T06:22:33+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"journal-of-robotic-surgery","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"jors","sideBox":"Learn more about [Journal of Robotic Surgery](http://link.springer.com/journal/11701)","snPcode":"11701","submissionUrl":"https://submission.nature.com/new-submission/11701/3","title":"Journal of Robotic Surgery","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"d2eb439d-a5ca-4954-af29-40e467759847","owner":[],"postedDate":"July 18th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2025-09-08T16:02:31+00:00","versionOfRecord":{"articleIdentity":"rs-7080528","link":"https://doi.org/10.1007/s11701-025-02689-4","journal":{"identity":"journal-of-robotic-surgery","isVorOnly":false,"title":"Journal of Robotic Surgery"},"publishedOn":"2025-09-04 15:57:28","publishedOnDateReadable":"September 4th, 2025"},"versionCreatedAt":"2025-07-18 12:30:55","video":"","vorDoi":"10.1007/s11701-025-02689-4","vorDoiUrl":"https://doi.org/10.1007/s11701-025-02689-4","workflowStages":[]},"version":"v1","identity":"rs-7080528","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7080528","identity":"rs-7080528","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}