Comparing artificial intelligence and healthcare professional performance in surgical and interventional video analysis: a systematic review and meta-analysis

This systematic review and meta-analysis was reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (Supplementary Table 7 ) 174 . The protocol was registered on PROSPERO (ID: CRD42024555207). The search strategy was developed and iteratively refined, including review by a medical librarian at the University of Cambridge (IK). Searches were performed in Ovid Medline (Wolters Kluwer, Netherlands), Web of Science, Scopus, and IEEE Xplore from inception to 5th June 2025 (Supplementary Table 8 ). Search sensitivity was evaluated against 11 studies known to meet inclusion criteria: all papers were successfully captured. Primary studies that developed or validated a clinical AI model for the analysis of surgical or interventional videos ( intervention ), and additionally quantitatively evaluated its performance ( outcome ) against healthcare professionals with or without AI assistance ( comparator ), as outlined in Table 3 , were included. Surgical or interventional procedures were defined as ‘invasive procedures, where deliberate access to the body is gained via an incision, percutaneous puncture, or natural orifice, performed by a healthcare professional using instruments including, but not limited to, endoscopes, catheters, scalpels, scissors, devices, and tubes’ 175 . Studies were excluded if the title and abstract indicated failure to meet all of the specified inclusion criteria or satisfied at least one of the exclusion criteria. Two reviewers (ARF and KJS) independently performed title and abstract screening with blinding using Rayyan (Rayyan Systems Inc., Cambridge, USA). A pilot screen of 100 publications was first conducted to ensure concordance between reviewers. Any disagreements following unblinding were resolved by discussion between the reviewers until mutual agreement was reached. Any outstanding disagreements were resolved through discussion with the senior reviewer (SCW). Table 3 Inclusion and exclusion criteria Inclusion criteria Exclusion Criteria • Primary study • AI model applied to surgical or interventional videos • Quantitative comparison of either: (1) ‘AI vs unassisted healthcare professionals’, (2) ‘AI-assisted healthcare professionals vs unassisted healthcare professionals’, or (3) ‘AI-assisted healthcare professionals vs AI’ • AI model not applied to surgical or interventional videos • AI study with either qualitative comparison or no comparison against healthcare professional performance • No separate ground truth from the comparators • Editorial, case report, opinion article, letter, conference abstract, correction • Systematic review or meta-analysis • Non-English language • Full text unavailable AI artificial intelligence. Inclusion and exclusion criteria • Primary study • AI model applied to surgical or interventional videos • Quantitative comparison of either: (1) ‘AI vs unassisted healthcare professionals’, (2) ‘AI-assisted healthcare professionals vs unassisted healthcare professionals’, or (3) ‘AI-assisted healthcare professionals vs AI’ • AI model not applied to surgical or interventional videos • AI study with either qualitative comparison or no comparison against healthcare professional performance • No separate ground truth from the comparators • Editorial, case report, opinion article, letter, conference abstract, correction • Systematic review or meta-analysis • Non-English language • Full text unavailable AI artificial intelligence. Articles were retrieved for full text screening and data extraction using a predefined data extraction sheet. Extracted variables included: author, year of publication, study design, subspecialty (including target condition and procedure), data source and range, sample characteristics (including selection criteria, size, and age), AI model (including architecture type, backbone model, and the presence or absence of transfer learning), type of comparison (i.e. unassisted healthcare professionals, AI-assisted healthcare professionals, or both), ground truth comparison, dataset splits (i.e. training, tuning defined as hyperparameter optimization, and validation defined as the final evaluation of model performance), quantitative outcomes of performance, relevant statistical analyses, main conclusions, and limitations. Where possible, 2 × 2 contingency tables were constructed consisting of true positives, false positives, true negatives, and false negatives. Full text screening and data extraction were assessed independently in duplicate (ARF and KJS/JKD/TF) with blinding. Any disagreements following unblinding were resolved by discussion between reviewers until mutual agreement was reached. Any outstanding questions were resolved by discussion with the senior reviewers (SCW and AD). A meta-analysis was conducted using a bivariate binomial method on studies providing contingency tables for two or more comparators (direct within-study comparison) 176 . Contingency tables were first grouped based on comparators: (1) ‘AI vs unassisted healthcare professionals’, (2) ‘AI-assisted healthcare professionals vs unassisted healthcare professionals’, and (3) ‘AI-assisted healthcare professionals vs AI’. For our ‘AI vs unassisted healthcare professionals’ analysis, contingency tables were further stratified based on the type of validation (external or internal, where the latter included temporally split data). For our ‘AI vs unassisted healthcare professionals’ analysis, sub-group analyses were performed, evaluating only the single highest accuracy contingency table for AI and unassisted healthcare professionals in each task, to compare their respective peak performance levels. For our ‘AI-assisted healthcare professionals vs unassisted healthcare professionals’ and ‘AI-assisted healthcare professionals vs AI’ analyses, sub-group analyses were performed to investigate differences in healthcare professional performance at the non-expert and expert levels, with experts broadly defined as the level of consultants or attending physicians. In our primary analysis, contingency tables for each comparator were averaged to ensure a direct one-to-one comparison between comparators and to prevent undue emphasis on studies with more individuals within each comparator group. As a sensitivity analysis, we also assessed the unaveraged contingency tables for each analysis to assess whether there was a significant change in results when information regarding the number of each type of comparator was incorporated. For each analysis, seven generalised linear mixed models were generated (Supplementary Table 9 ). Model A was the most simplistic and assumed a combined sensitivity and combined specificity for both comparators at the fixed effects and random effects level. Models B-D similarly incorporated a combined sensitivity and combined specificity at the random effects level, but assumed variable variance for sensitivity and/or specificity at the fixed effects level. Finally, models E-G incorporated separate random effects for the sensitivity and/or specificity of both comparators, whilst also assuming variable variance for sensitivity and/or specificity at the fixed effects level. The Akaike information criterion was used to select the final model by considering the trade-off between goodness of fit and simplicity of each model. As a secondary assessment of model fit, likelihood ratio tests were conducted comparing the final model with the two models that differed by a single step change, where the null hypothesis was rejected for p  < 0.05. The logarithmic odds ratio from the final selected model was used to assess statistical differences in sensitivity and specificity between comparators, where the null hypothesis was rejected for p  < 0.05. Relative differences in sensitivity and specificity between comparators in the final model were then calculated with 95% confidence intervals. Results of the bivariate binomial model were transformed to coefficients of the hierarchical summary receiver operating characteristic model, and the curves were visualised 176 . Due to the broad nature of our review across multiple types of tasks, procedures, and specialties, we were accepting of a large degree of heterogeneity across studies.

From 37,953 search results and three additional results from citation searching, 146 studies were included in our final analysis (Fig. 1 ) 7 – 152 . Fig. 1 PRISMA flow diagram of study selection. Illustration of the identification, screening, eligibility assessment, and inclusion of studies in the systematic review. AI artificial intelligence. Illustration of the identification, screening, eligibility assessment, and inclusion of studies in the systematic review. AI artificial intelligence. The designs of included studies are summarised in Table 1 and Supplementary Table 1 . Table 1 Summary of study design of included studies Reference Specialty Procedure AI model name, architecture, and backbone model Transfer learning Type of comparison Ground truth Same ground truth across all samples Norton et al. 7 Gastroenterology HPB surgery Endoscopic US • Neural network NR • AI vs HP Histopathology Yes Voigt et al. 8 ENT surgery Laryngoscopy • SVM with linear kernel classifier NR • AI vs HP Clinical examination Yes Andre et al. 9 Gastroenterology Colorectal surgery Colonoscopy • Bag-of-words • K-nearest neighbour NR • AI vs HP Histopathology Yes Takemura et al. 10 Gastroenterology Colorectal surgery Colonoscopy HuPAS • Bag-of-words • SVM with linear kernel classifier NR • AI vs HP Histopathology Yes Mori et al. 11 Gastroenterology Colorectal surgery Colonoscopy • SVM with linear kernel classifier NR • AI vs HP Histopathology Yes Urban et al. 12 Gastroenterology Colorectal surgery Colonoscopy • CNN (VGG-19) Yes (ImageNet) • AI vs HP Expert review Yes Cai et al. 13 Gastroenterology Upper GI surgery OGD • CNN NR • AI vs HP • HP + AI vs HP Histopathology Yes Sadda et al. 14 Obstetrics and gynaecology Fetoscopic laser photocoagulation • CNN (U-Net) NR • AI vs HP Expert review Yes Wu et al. 15 Gastroenterology Upper GI surgery OGD • CNN (VGG-16; ResNet-50) Yes (ImageNet) • AI vs HP Internal validation 1 : histopathology Internal validation 2 : expert consensus Internal validation 1 : yes Internal validation 2 : yes Wu et al. 16 Gastroenterology Upper GI surgery OGD WISENSE • CNN (VGG-16) • DRL Yes (specifics NR) • AI vs HP • HP + AI vs HP • HP + AI vs AI Expert consensus Yes Becq et al. 17 Gastroenterology Colorectal surgery Colonoscopy • CNN (SegNet) NR • AI vs HP Expert consensus Yes Cincar et al. 18 Gastroenterology Colorectal surgery Colonoscopy • Random forest NR • AI vs HP Histopathology Yes Fukuda et al. 19 Gastroenterology Upper GI surgery OGD • CNN (SSD) Yes (ImageNet) • AI vs HP Histopathology Yes Garcia-Peraza-Herrera et al. 20 Gastroenterology Upper GI surgery OGD ResNet-18-CAM-DS • CNN (ResNet-18) NR • AI vs HP Histopathology Yes Horiuchi et al. 21 Gastroenterology Upper GI surgery OGD • CNN (GoogLeNet) Yes (ImageNet) • AI vs HP Histopathology Yes Lecuyer et al. 22 Gastroenterology HPB surgery Ophthalmology Laparoscopic cholecystectomy Cataract surgery • CNN (VGG-19; InceptionV3; ResNet-50) Yes (ImageNet) • HP + AI vs HP Cholec80 : expert review CATARACTS : expert consensus Cholec80 : yes CATARACTS : yes Li et al. 23 Obstetrics and gynaecology Colposcopy • CNN (ResNet-50; ResNet-101) • GCN NR • AI vs HP Histopathology Yes Nakashima et al. 24 Gastroenterology OGD • CNN NR • AI vs HP Infection status : serum antibodies Eradication status : urea breath test Yes Shimamoto et al. 25 Gastroenterology Upper GI surgery OGD • CNN (SSD; VGG-16) Yes (ImageNet) • AI vs HP Histopathology Yes Su et al. 26 Gastroenterology Colorectal surgery Colonoscopy AQCS • CNN (AlexNet; ZFNet; YOLOv2; MobileNets) NR • HP + AI vs HP Histopathology Yes Tang et al. 27 Gastroenterology Upper GI surgery OGD • CNN (DarkNet-53) NR • AI vs HP • HP + AI vs HP • HP + AI vs AI Histopathology Yes Wang et al. 28 Gastroenterology Colorectal surgery Colonoscopy EndoScreener • CNN NR • HP + AI vs HP Histopathology Yes Xue et al. 29 Obstetrics and gynaecology Colposcopy CAIDS • CNN (ResNet-50; ResNet-101; U-Net; YOLOv3) • GCN NR • AI vs HP Histopathology Yes Zhou et al. 30 Gastroenterology Colorectal surgery Colonoscopy ENDOANGEL • CNN (DenseNet-121) Yes (specifics NR) • AI vs HP Expert consensus Yes Chen et al. 31 Gastroenterology Upper GI surgery OGD ENDOANGEL • CNN NR • AI vs HP Expert consensus Yes Cui et al. 32 Neurosurgery Percutaneous transforaminal endoscopic discectomy • CNN (YOLOv3) Yes (specifics NR) • AI vs HP • HP + AI vs HP • HP + AI vs AI Expert consensus Yes Everson et al. 33 Gastroenterology Upper GI surgery OGD ResNet-18-CAM-DS • CNN (ResNet-18) NR • AI vs HP Histopathology Yes Jiang et al. 34 Gastroenterology Colorectal surgery Colonoscopy • CNN (ResNet-50) Yes (ImageNet) • AI vs HP • HP + AI vs HP • HP + AI vs AI Histopathology Yes Li et al. 35 Gastroenterology Upper GI surgery OGD • CNN NR • AI vs HP • HP + AI vs HP • HP + AI vs AI Histopathology Yes Ling et al. 36 Gastroenterology Upper GI surgery OGD • CNN (VGG-16) Yes (ImageNet) • AI vs HP Histopathology Yes Livovsky et al. 37 Gastroenterology Colorectal surgery Colonoscopy DEEP 2 • CNN (RetinaNet; LSTM-SSD) NR Internal validation : • AI vs HP • HP + AI vs HP External validation : • AI vs HP Expert consensus Yes Shiroma et al. 38 Gastroenterology Upper GI surgery OGD • CNN (SSD) NR • AI vs HP • HP + AI vs HP • HP + AI vs AI Histopathology Yes Tang et al. 39 Gastroenterology Upper GI surgery OGD • CNN (ResNet-50) NR • AI vs HP • HP + AI vs HP • HP + AI vs AI Histopathology Yes Waki et al. 40 Gastroenterology Upper GI surgery OGD • CNN (BiSeNet; ResNet) NR • AI vs HP • HP + AI vs HP • HP + AI vs AI Histopathology Yes Wu et al. 41 Gastroenterology Upper GI surgery OGD • CNN Yes (ImageNet) • AI vs HP Histopathology Yes Xu et al. 42 Gastroenterology Upper GI surgery OGD ENDOANGEL • CNN (VGG-16) Yes (specifics NR) • AI vs HP Histopathology Yes Yang et al. 43 Gastroenterology Upper GI surgery OGD • CNN (InceptionResNet-V2) NR • AI vs HP • HP + AI vs HP • HP + AI vs AI Histopathology Yes Yoo et al. 44 Anaesthetics Pulmonology Bronchoscopy • CNN (EfficientNet-B1) Yes (ImageNet) • AI vs HP Automatic labelling by optical character recognition engine Yes Zhao et al. 45 Gastroenterology Colorectal surgery Colonoscopy • CNN (YOLOv2) NR • AI vs HP • HP + AI vs HP • HP + AI vs AI Histopathology Yes Zhi et al. 46 Pulmonology Cardiothoracic surgery Bronchoscopy • K-means clustering NR • AI vs HP Histopathology or clinical follow-up for at least 1 year No Ainechi et al. 47 Gastroenterology Colorectal surgery Colonoscopy EndoBRAIN-EYE • CNN (YOLOv3) NR • HP + AI vs HP Histopathology Yes Biffi et al. 48 Gastroenterology Colorectal surgery Colonoscopy GI Genius • CNN (ResNet-18) NR • AI vs HP Histopathology Yes He et al. 49 Gastroenterology Upper GI surgery OGD ENDOANGEL-ME • CNN (ResNet-50) NR Internal validation : • AI vs HP External validation : • AI vs HP • HP + AI vs HP • HP + AI vs AI Histopathology Yes Hussein et al. 50 Gastroenterology Upper GI surgery OGD • CNN (ResNet-101; FCNResNet-50) Yes (Gastrointestinal Artificial Intelligence Diagnostic System) • AI vs HP Histopathology Yes Li et al. 51 Anaesthetics Pulmonology Bronchoscopy • CNN (VGG-16) Yes (specifics NR) • HP + AI vs HP Expert consensus Yes Lu et al. 52 Gastroenterology Colorectal surgery Colonoscopy Endo-CRC • CNN (ResNet-50) Yes (specifics NR) • AI vs HP Histopathology in most cases No Ma et al. 53 Gastroenterology Upper GI surgery OGD • CNN (YOLOv4) NR • AI vs HP Histopathology Yes Pangal et al. 54 Neurosurgery ENT surgery Endoscopic transsphenoidal surgery SOCALNet • CNN (ResNet; LSTM) Yes (ImageNet) • AI vs HP Outcome of cadaveric simulations Yes Sato et al. 55 Upper GI surgery Thoracoscopic oesophagectomy • CNN (DeepLabv3+) Yes (2012 PASCAL Visual Object Classes Challenge) • AI vs HP Expert review Yes Tajiri et al. 56 Gastroenterology Upper GI surgery OGD • CNN (ResNet-101) Yes (ImageNet) • AI vs HP Histopathology Yes Tang et al. 57 Gastroenterology Upper GI surgery OGD • CNN (YOLOv3) NR • AI vs HP • HP + AI vs HP • HP + AI vs AI Histopathology Yes Tang et al. 58 Gastroenterology Colorectal surgery Colonoscopy • CNN (YOLOv4) • Yes (COCO) • AI vs HP Expert consensus Yes Wang et al. 59 Ophthalmology Cataract surgery DeepSurgery • CNN NR • AI vs HP Expert consensus Yes Wang et al. 60 ENT surgery Thyroid surgery PTAIR • CNN (Faster R-CNN) NR • AI vs HP Expert consensus Yes Wu et al. 61 Gastroenterology Upper GI surgery OGD ENDOANGEL-LD • CNN (YOLOv3; ResNet-50) Yes (ImageNet; COCO) • AI vs HP Histopathology Yes Wu et al. 62 Gastroenterology Upper GI surgery OGD ENDOANGEL • CNN (YOLOv3; ResNet-50) Yes (ImageNet; COCO) • AI vs HP Histopathology Yes Xie et al. 63 Gastroenterology Upper GI surgery Capsule endoscopy SmartScan • CNN (EfficientNet; YOLONet) NR • HP + AI vs HP Expert consensus Yes Yuan et al. 64 Gastroenterology Upper GI surgery OGD • CNN (YOLOv3) NR • AI vs HP Histopathology Yes Zhao et al. 65 Gastroenterology Upper GI surgery OGD • CNN (U-Net) NR • AI vs HP Histopathology Yes Zhao et al. 66 Gastroenterology Upper GI surgery OGD • CNN (U-Net) NR • AI vs HP Histopathology Yes Abdelrahim et al. 67 Gastroenterology Upper GI surgery OGD • CNN (16-SegNet) NR • AI vs HP Histopathology Yes Ding et al. 68 Gastroenterology Colorectal surgery Capsule endoscopy • CNN (ResNet-50) • CRNN (ResNet-50) Yes (ImageNet) • AI vs HP • HP + AI vs HP • HP + AI vs AI Expert consensus Yes Dong et al. 69 Gastroenterology Upper GI surgery OGD ENDOANGEL; ENDOANGEL-2022 • CNN (YOLOv3; ResNet-50) Yes (ENDOANGEL) • AI vs HP Histopathology Yes Dong et al. 70 Gastroenterology Upper GI surgery OGD ENDOANGEL-ED • random forest NR Internal validation : • AI vs HP • HP + AI vs HP • HP + AI vs AI External validation : • AI vs HP Histopathology Yes Du et al. 71 Gastroenterology Upper GI surgery OGD ENDOANGEL-MM • CNN (ResNet-50) Yes (specifics NR) • AI vs HP • HP + AI vs HP • HP + AI vs AI Expert consensus Yes Fang et al. 72 ENT surgery Laryngoscopy • CNN (ResNet-50; Faster R-CNN) Yes (ImageNet) • AI vs HP NR NR Feng et al. 73 Gastroenterology Colorectal surgery Colonoscopy ViENDO • CNN (ResNeXt101) NR • AI vs HP Expert consensus Yes Fer et al. 74 Upper GI surgery Roux-en-Y gastric bypass • CNN (ResNet) Yes (specifics NR) • AI vs HP Expert consensus Yes Fockens et al. 75 Gastroenterology Upper GI surgery OGD • CNN (EfficientNet, MobileNetV2) Yes (ImageNet; GastroNet) • AI vs HP • HP + AI vs HP • HP + AI vs AI Histopathology Yes Hossain et al. 76 Gastroenterology Colorectal surgery OGD WISE VISION • CNN (ResNet-152) NR • AI vs HP Histopathology Yes Koh et al. 77 Gastroenterology Colorectal surgery Colonoscopy GI Genius • CNN (SegNet) NR • AI vs HP • HP + AI vs HP • HP + AI vs AI Histopathology Yes Li et al. 78 ENT surgery Laryngoscopy LPAIDS • CNN (U-Net) NR • AI vs HP Histopathology Yes Li et al. 79 Gastroenterology OGD IDEA-HP • CNN (ZFNet) Yes (ImageNet and 132297 frames from other source) • AI vs HP Urea breath test Yes Matsuda et al. 80 Gastroenterology Upper GI surgery OGD • CNN (AlexNet) NR • AI vs HP Histopathology Yes Matsuda et al. 81 Gastroenterology Upper GI surgery OGD • CNN (AlexNet) NR • AI vs HP Histopathology Yes Misumi et al. 82 Gastroenterology Colorectal surgery Colonoscopy CAD EYE • CNN NR • AI vs HP Expert review Yes Nehme et al. 83 Gastroenterology Colorectal surgery Colonoscopy GI Genius • CNN (SegNet) NR • HP + AI vs HP NR NR Nemoto et al. 84 Gastroenterology Colorectal surgery Colonoscopy • CNN (ResNet-50) Yes (ImageNet) • AI vs HP Histopathology Yes O’Hara et al. 85 Gastroenterology Colorectal surgery Capsule endoscopy • CNN NR • HP + AI vs HP Expert consensus Yes Quek et al. 86 Gastroenterology Upper GI surgery OGD • CNN (SSD) NR • AI vs HP Histopathology Yes Sedighipour Chafjiri et al. 87 Gastroenterology OGD • SNN (DenseNet-121) Yes (ImageNet) • AI vs HP NR NR Shen et al. 88 Gastroenterology OGD CADSS-HP • CNN (ResNet-34) Yes (ImageNet) • AI vs HP Histopathology and/or urea breath test No Shi et al. 89 Gastroenterology OGD GAM-EfficientNet • CNN (EfficientNetV2-M) Yes (ImageNet) • AI vs HP Histopathology Yes Smedsrud et al. 90 Gastroenterology Colorectal surgery Colonoscopy • CNN (EfficientNet) Yes (specifics NR) • AI vs HP Histopathology Yes Sudaveric et al. 91 Gastroenterology Colorectal surgery Colonoscopy Poseidon • CNN NR • AI vs HP Manual measurement Yes Sun et al. 92 Urology Transrectal US 2D P-Net; 3D P-Net • CNN (ResNet-50) NR • AI vs HP Histopathology Yes Wang et al. 93 Obstetrics and gynaecology Laparoscopic hysterectomy or laparoscopic endometriosis resection • CNN (U-Net) NR • AI vs HP Expert consensus Yes Wang et al. 94 Gastroenterology Colorectal surgery Colonoscopy EMSEN • CNN (YOLOv5) NR • AI vs HP Histopathology Yes Wei et al. 95 Gastroenterology Colorectal surgery Colonoscopy EndoVigilant • specifics NR NR • HP + AI vs HP Histopathology Yes Xu et al. 96 Pulmonology EBUS-TBNA • GAT NR • AI vs HP Histopathology or clinical follow-up for at least 6 months No Yao et al. 97 Gastroenterology Colorectal surgery Colonoscopy CCIC • CNN (ResNet-50) • K-nearest neighbour Yes (specifics NR) External validation 1 : • AI vs HP • HP + AI vs HP • HP + AI vs AI External validation 2 : • AI vs HP Histopathology Yes Yuan et al. 98 Gastroenterology Upper GI surgery OGD YOLACT • CNN (ResNet-101) NR • AI vs HP Histopathology Yes Yui et al. 99 ENT surgery Endoscopic sinus surgery • CNN (MobileNet-v2) NR • AI vs HP Histopathology Yes Zhang et al. 100 HPB surgery Cholangioscopy MBSDeiT • ViT (DeiT) Yes (ImageNet) • AI vs HP Histopathology Yes Zhang et al. 101 Gastroenterology Upper GI surgery OGD AI-EARS • CNN (VGG-16; ResNet-50; YOLOv3; U-Net + +) NR • HP + AI vs HP Internal validation 1 : expert consensus Internal validation 2 : histopathology and expert consensus Internal validation 1 : yes Internal validation 2 : no Zhao et al. 102 Gastroenterology OGD • CNN (U-Net) NR • AI vs HP Histopathology Yes Zhou et al. 103 Neurosurgery Microsurgical aneurysm clipping surgery MACSSwin-T • ViT (SwinT) Yes (ImageNet) • AI vs HP Expert consensus Yes Chen et al. 104 Gastroenterology Colorectal surgery Capsule endoscopy • CNN (MobileNetv3-large) NR • AI vs HP Expert consensus Yes Choi et al. 105 Gastroenterology Colorectal surgery Capsule endoscopy • CNN (VGGNet) NR • AI vs HP Expert consensus Yes Furube et al. 106 Upper GI surgery Robot-assisted minimally invasive oesophagectomy • CNN (DeepLabv3 Plus) NR • HP + AI vs HP Expert review Yes Hardy et al. 107 Gastroenterology Colorectal surgery Colonoscopy AI fluorescence classifier • ML classifier (specifics NR) NR • AI vs HP Histopathology Yes He et al. 108 Gastroenterology Upper GI surgery OGD • CNN (YOLOv7) Yes (COCO) • AI vs HP • HP + AI vs HP • HP + AI vs AI Expert consensus Yes Hoffmann et al. 109 Surgery Open suturing • ViT (SwinTiny) Yes (Kinetics400) • AI vs HP OSATS GRS Yes Jukema et al. 110 Gastroenterology Upper GI surgery OGD • CNN (EfficientNet-Lite1) Yes (ImageNet; GastroNet) • AI vs HP • HP + AI vs HP • HP + AI vs AI Histopathology Yes Kavak et al. 111 ENT surgery Laryngoscopy • CNN NR • AI vs HP Expert consensus Yes Khan et al. 112 Neurosurgery Endoscopic transsphenoidal surgery • CNN (ResNet-50 + LSTM) NR • AI vs HP Surgical performance : expert consensus Clinical outcomes : clinical follow-up Yes Labrecque Langlais et al. 113 Cardiology Interventional radiology Coronary angiography DeepCoro • CNN (Xception; RetinaNet; DeepLabv3; DeepLabv3 + ) • ViT (Swin3D) Yes (Kinetics400; ImageNetV1) • AI vs HP Expert consensus Yes Lee et al. 114 Gastroenterology Upper GI surgery OGD ENAD CAD-G • CNN (YOLOv5; EfficientNetB0) NR • AI vs HP Histopathology Yes Li et al. 115 Gastroenterology Upper GI surgery Capsule endoscopy • ViT Yes (3343 unlabelled videos) • AI vs HP Expert consensus Yes Li et al. 116 Gastroenterology Colorectal surgery Capsule endoscopy CE-YOLOv5 • CNN (YOLOv5) NR • AI vs HP Expert consensus Yes Luo et al. 117 Gastroenterology Colorectal surgery Colonoscopy • CNN (ResNet-50) • ViT (Swin transformer) Yes (ImageNet) • AI vs HP Expert consensus Yes Meinikheim et al. 118 Gastroenterology Upper GI surgery OGD • CNN (DeepLabV3 + ; ResNet-50) NR • AI vs HP • HP + AI vs HP • HP + AI vs AI Histopathology Yes Robles-Medranda et al. 119 HPB surgery Cholangioscopy AIWorks-Cholangioscopy • CNN (YOLOv5) NR • AI vs HP Clinical follow-up, imaging, or histopathology over 12 months No Sampieri et al. 120 ENT surgery Laryngoscopy SegMENT-Plus • CNN (EfficientNet-B5) Yes (ImageNet) • AI vs HP Expert consensus Yes Sang et al. 121 ENT surgery Thyroid surgery Video-Trans-U-HRNet • CNN (Trans-U-Net) NR • AI vs HP Expert review Yes Seager et al. 122 Gastroenterology Colorectal surgery Colonoscopy GI Genius • CNN (SegNet) NR • HP + AI vs HP Histopathology Yes Shimada et al. 123 Pulmonology Cardiothoracic surgery Thoracoscopy • CNN (EfficientNet-B7) • ViT (ViT-B/16) Yes (ImageNet; JFT-200 M) • AI vs HP Histopathology Yes Tao et al. 124 ENT surgery Laryngoscopy • CNN (YOLOv5) Yes (COCO) • AI vs HP Macroscopic confirmation Yes Tao et al. 125 Gastroenterology Upper GI surgery OGD ENDOANGEL • CNN (UNet + +; ResNet-50) Yes (specifics NR) • AI vs HP Internal validation 1 : histopathology Internal validation 2 : histopathology Internal validation 3 : Kimura-Takemoto Classification Yes Tie et al. 126 ENT surgery Laryngoscopy • CNN (DeepLabv3-ResNet-101) NR • AI vs HP • HP + AI vs HP • HP + AI vs AI Histopathology Yes Wang et al. 127 ENT surgery Laryngoscopy • CNN (DeepLabv3-ResNet-101) NR • AI vs HP • HP + AI vs HP • HP + AI vs AI Histopathology Yes Wang et al. 128 ENT surgery Endoscopic thyroid surgery PTAIR 2.0 • CNN (YOLOX) NR • AI vs HP Expert consensus Yes Wang et al. 129 Gastroenterology Colorectal surgery OGD ENDOANGEL-CPS • CNN (DenseNet-169) Yes (ImageNet) • AI vs HP Manual measurement Yes Williams et al. 130 Neurosurgery Microsurgical aneurysm clipping surgery MACSSwin-T • ViT (SwinT) Yes (ImageNet) • AI vs HP • HP + AI vs HP • HP + AI vs AI Expert consensus Yes Xie et al. 131 Gastroenterology Upper GI surgery Colorectal surgery Capsule endoscopy SmartScan Plus • CNN (EfficientNet; YOLONet) NR • HP + AI vs HP Expert consensus Yes Xu et al. 132 Gastroenterology Colorectal surgery Colonoscopy • CNN NR • AI vs HP Expert consensus Yes Xu et al. 133 ENT surgery Nasal endoscopy • CNN (Att-Res2-UNet) NR • AI vs HP Histopathology Yes Yao et al. 134 Gastroenterology Colorectal surgery Colonoscopy e-BPPS • CNN NR • AI vs HP Histopathology Yes Zhu et al. 135 Urology Holmium laser surgery • CNN (ResNet-101) NR • AI vs HP Histopathology Yes Ahn et al. 136 Gastroenterology Upper GI surgery OGD Automated Photodocumentation Task • ViT (SwinT) Yes (ImageNet) • AI vs HP Expert consensus Yes Amante et al. 137 Pulmonology Cardiothoracic surgery Bronchoscopy • CNN (ResNet-50) Yes (ImageNet) • AI vs HP Histopathology Yes Aoyama et al. 138 Gastroenterology Upper GI surgery OGD • CNN (YOLOv3) NR AI vs HP • HP + AI vs HP • HP + AI vs AI Histopathology Yes Chen et al. 139 Gastroenterology Upper GI surgery OGD • CNN (EfficientNet) Yes (specifics NR) • AI vs HP Expert consensus Yes Feng et al. 140 Gastroenterology Upper GI surgery OGD • CNN (ResNet-18) NR • AI vs HP • HP + AI vs HP • HP + AI vs AI Histopathology Yes He et al. 141 ENT surgery Nasal endoscopy NPC-SDNet • CNN NR • AI vs HP Histopathology Yes Khanna et al. 142 Urology Robot-assisted radical prostatectomy • ViT NR • AI vs HP Expert review Yes Kim et al. 143 Gastroenterology Colorectal surgery Colonoscopy CADeye • CNN NR • AI vs HP • HP + AI vs HP • HP + AI vs AI Histopathology Yes Li et al. 144 Gastroenterology Upper GI surgery OGD • CNN NR • HP + AI vs HP Histopathology Yes Liu et al. 145 Gastroenterology Upper GI surgery OGD • CNN (ResNet-50) NR • HP + AI vs HP Expert consensus Yes Ma et al. 146 Gastroenterology Upper GI surgery OGD • CNN (Inception-ResNet-v2) NR • AI vs HP • HP + AI vs HP • HP + AI vs AI Histopathology Yes Majeedi et al. 147 Anaesthetics Laryngoscopy • CNN (YOLOv8) Yes (COCO) • AI vs HP Expert consensus Yes Mezzina et al. 148 Urology Robot-assisted partial nephrectomy • CNN (ResNet-50; ResNet-50-LSTM; TeCNO) Yes (ImageNet; Cholec80) • AI vs HP Expert consensus Yes Ogata et al. 149 Gastroenterology Colonoscopy • CNN (VGG-16) NR • AI vs HP • HP + AI vs HP • HP + AI vs AI Clinical follow-up for 12 months Yes Papachrysos et al. 150 Gastroenterology Colorectal surgery Colonoscopy PolypAid • CNN GI Genius • CNN EndoAid • NR NR • AI vs HP Expert consensus Yes Scheppach et al. 151 Gastroenterology Upper GI surgery OGD • CNN (DeepLabv3+; ResNet-50) NR • HP + AI vs HP Expert consensus Yes Tappermann et al. 152 Urology Robot-assisted radical prostatectomy • CNN (MoCov2) NR • AI vs HP Clinical follow-up Yes AI artificial intelligence, CNN convolutional neural network, CRNN convolutional recurrent neural network, DRL deep reinforcement learning, EBUS-TBNA endobronchial ultrasound-guided transbronchial needle aspiration, ENT ear, nose, and throat, GAT graph attention network, GCN graph convolutional network; GI gastrointestinal, HP healthcare professionals, HPB hepato-pancreato-biliary, ML machine learning, NR not reported, OGD oesophagogastroduodenoscopy, SNN Siamese neural network, SVM support vector machine, US ultrasound, ViT vision transformer. Summary of study design of included studies Gastroenterology HPB surgery Gastroenterology Colorectal surgery • Bag-of-words • K-nearest neighbour Gastroenterology Colorectal surgery HuPAS • Bag-of-words • SVM with linear kernel classifier Gastroenterology Colorectal surgery Gastroenterology Colorectal surgery Gastroenterology Upper GI surgery • AI vs HP • HP + AI vs HP Gastroenterology Upper GI surgery Internal validation 1 : histopathology Internal validation 2 : expert consensus Internal validation 1 : yes Internal validation 2 : yes Gastroenterology Upper GI surgery WISENSE • CNN (VGG-16) • DRL • AI vs HP • HP + AI vs HP • HP + AI vs AI Gastroenterology Colorectal surgery Gastroenterology Colorectal surgery Gastroenterology Upper GI surgery Gastroenterology Upper GI surgery ResNet-18-CAM-DS • CNN (ResNet-18) Gastroenterology Upper GI surgery Gastroenterology HPB surgery Ophthalmology Laparoscopic cholecystectomy Cataract surgery Cholec80 : expert review CATARACTS : expert consensus Cholec80 : yes CATARACTS : yes • CNN (ResNet-50; ResNet-101) • GCN Infection status : serum antibodies Eradication status : urea breath test Gastroenterology Upper GI surgery Gastroenterology Colorectal surgery AQCS • CNN (AlexNet; ZFNet; YOLOv2; MobileNets) Gastroenterology Upper GI surgery • AI vs HP • HP + AI vs HP • HP + AI vs AI Gastroenterology Colorectal surgery EndoScreener • CNN CAIDS • CNN (ResNet-50; ResNet-101; U-Net; YOLOv3) • GCN Gastroenterology Colorectal surgery ENDOANGEL • CNN (DenseNet-121) Gastroenterology Upper GI surgery ENDOANGEL • CNN • AI vs HP • HP + AI vs HP • HP + AI vs AI Gastroenterology Upper GI surgery ResNet-18-CAM-DS • CNN (ResNet-18) Gastroenterology Colorectal surgery • AI vs HP • HP + AI vs HP • HP + AI vs AI Gastroenterology Upper GI surgery • AI vs HP • HP + AI vs HP • HP + AI vs AI Gastroenterology Upper GI surgery Gastroenterology Colorectal surgery DEEP 2 • CNN (RetinaNet; LSTM-SSD) Internal validation : • AI vs HP • HP + AI vs HP External validation : • AI vs HP Gastroenterology Upper GI surgery • AI vs HP • HP + AI vs HP • HP + AI vs AI Gastroenterology Upper GI surgery • AI vs HP • HP + AI vs HP • HP + AI vs AI Gastroenterology Upper GI surgery • AI vs HP • HP + AI vs HP • HP + AI vs AI Gastroenterology Upper GI surgery Gastroenterology Upper GI surgery ENDOANGEL • CNN (VGG-16) Gastroenterology Upper GI surgery • AI vs HP • HP + AI vs HP • HP + AI vs AI Anaesthetics Pulmonology Gastroenterology Colorectal surgery • AI vs HP • HP + AI vs HP • HP + AI vs AI Pulmonology Cardiothoracic surgery Gastroenterology Colorectal surgery EndoBRAIN-EYE • CNN (YOLOv3) Gastroenterology Colorectal surgery GI Genius • CNN (ResNet-18) Gastroenterology Upper GI surgery ENDOANGEL-ME • CNN (ResNet-50) Internal validation : • AI vs HP External validation : • AI vs HP • HP + AI vs HP • HP + AI vs AI Gastroenterology Upper GI surgery Anaesthetics Pulmonology Gastroenterology Colorectal surgery Endo-CRC • CNN (ResNet-50) Gastroenterology Upper GI surgery Neurosurgery ENT surgery SOCALNet • CNN (ResNet; LSTM) Gastroenterology Upper GI surgery Gastroenterology Upper GI surgery • AI vs HP • HP + AI vs HP • HP + AI vs AI Gastroenterology Colorectal surgery DeepSurgery • CNN PTAIR • CNN (Faster R-CNN) Gastroenterology Upper GI surgery ENDOANGEL-LD • CNN (YOLOv3; ResNet-50) Gastroenterology Upper GI surgery ENDOANGEL • CNN (YOLOv3; ResNet-50) Gastroenterology Upper GI surgery SmartScan • CNN (EfficientNet; YOLONet) Gastroenterology Upper GI surgery Gastroenterology Upper GI surgery Gastroenterology Upper GI surgery Gastroenterology Upper GI surgery Gastroenterology Colorectal surgery • CNN (ResNet-50) • CRNN (ResNet-50) • AI vs HP • HP + AI vs HP • HP + AI vs AI Gastroenterology Upper GI surgery ENDOANGEL; ENDOANGEL-2022 • CNN (YOLOv3; ResNet-50) Gastroenterology Upper GI surgery ENDOANGEL-ED • random forest Internal validation : • AI vs HP • HP + AI vs HP • HP + AI vs AI External validation : • AI vs HP Gastroenterology Upper GI surgery ENDOANGEL-MM • CNN (ResNet-50) • AI vs HP • HP + AI vs HP • HP + AI vs AI Gastroenterology Colorectal surgery ViENDO • CNN (ResNeXt101) Gastroenterology Upper GI surgery • AI vs HP • HP + AI vs HP • HP + AI vs AI Gastroenterology Colorectal surgery WISE VISION • CNN (ResNet-152) Gastroenterology Colorectal surgery GI Genius • CNN (SegNet) • AI vs HP • HP + AI vs HP • HP + AI vs AI LPAIDS • CNN (U-Net) IDEA-HP • CNN (ZFNet) Gastroenterology Upper GI surgery Gastroenterology Upper GI surgery Gastroenterology Colorectal surgery CAD EYE • CNN Gastroenterology Colorectal surgery GI Genius • CNN (SegNet) Gastroenterology Colorectal surgery Gastroenterology Colorectal surgery Gastroenterology Upper GI surgery CADSS-HP • CNN (ResNet-34) GAM-EfficientNet • CNN (EfficientNetV2-M) Gastroenterology Colorectal surgery Gastroenterology Colorectal surgery Poseidon • CNN 2D P-Net; 3D P-Net • CNN (ResNet-50) Gastroenterology Colorectal surgery EMSEN • CNN (YOLOv5) Gastroenterology Colorectal surgery EndoVigilant • specifics NR Gastroenterology Colorectal surgery CCIC • CNN (ResNet-50) • K-nearest neighbour External validation 1 : • AI vs HP • HP + AI vs HP • HP + AI vs AI External validation 2 : • AI vs HP Gastroenterology Upper GI surgery YOLACT • CNN (ResNet-101) MBSDeiT • ViT (DeiT) Gastroenterology Upper GI surgery AI-EARS • CNN (VGG-16; ResNet-50; YOLOv3; U-Net + +) Internal validation 1 : expert consensus Internal validation 2 : histopathology and expert consensus Internal validation 1 : yes Internal validation 2 : no MACSSwin-T • ViT (SwinT) Gastroenterology Colorectal surgery Gastroenterology Colorectal surgery Gastroenterology Colorectal surgery AI fluorescence classifier • ML classifier (specifics NR) Gastroenterology Upper GI surgery • AI vs HP • HP + AI vs HP • HP + AI vs AI Gastroenterology Upper GI surgery • AI vs HP • HP + AI vs HP • HP + AI vs AI Surgical performance : expert consensus Clinical outcomes : clinical follow-up Cardiology Interventional radiology DeepCoro • CNN (Xception; RetinaNet; DeepLabv3; DeepLabv3 + ) • ViT (Swin3D) Gastroenterology Upper GI surgery ENAD CAD-G • CNN (YOLOv5; EfficientNetB0) Gastroenterology Upper GI surgery Gastroenterology Colorectal surgery CE-YOLOv5 • CNN (YOLOv5) Gastroenterology Colorectal surgery • CNN (ResNet-50) • ViT (Swin transformer) Gastroenterology Upper GI surgery • AI vs HP • HP + AI vs HP • HP + AI vs AI AIWorks-Cholangioscopy • CNN (YOLOv5) SegMENT-Plus • CNN (EfficientNet-B5) Video-Trans-U-HRNet • CNN (Trans-U-Net) Gastroenterology Colorectal surgery GI Genius • CNN (SegNet) Pulmonology Cardiothoracic surgery • CNN (EfficientNet-B7) • ViT (ViT-B/16) Gastroenterology Upper GI surgery ENDOANGEL • CNN (UNet + +; ResNet-50) Internal validation 1 : histopathology Internal validation 2 : histopathology Internal validation 3 : Kimura-Takemoto Classification • AI vs HP • HP + AI vs HP • HP + AI vs AI • AI vs HP • HP + AI vs HP • HP + AI vs AI PTAIR 2.0 • CNN (YOLOX) Gastroenterology Colorectal surgery ENDOANGEL-CPS • CNN (DenseNet-169) MACSSwin-T • ViT (SwinT) • AI vs HP • HP + AI vs HP • HP + AI vs AI Gastroenterology Upper GI surgery Colorectal surgery SmartScan Plus • CNN (EfficientNet; YOLONet) Gastroenterology Colorectal surgery Gastroenterology Colorectal surgery e-BPPS • CNN Gastroenterology Upper GI surgery Automated Photodocumentation Task • ViT (SwinT) Pulmonology Cardiothoracic surgery Gastroenterology Upper GI surgery AI vs HP • HP + AI vs HP • HP + AI vs AI Gastroenterology Upper GI surgery Gastroenterology Upper GI surgery • AI vs HP • HP + AI vs HP • HP + AI vs AI NPC-SDNet • CNN Gastroenterology Colorectal surgery CADeye • CNN • AI vs HP • HP + AI vs HP • HP + AI vs AI Gastroenterology Upper GI surgery Gastroenterology Upper GI surgery Gastroenterology Upper GI surgery • AI vs HP • HP + AI vs HP • HP + AI vs AI • AI vs HP • HP + AI vs HP • HP + AI vs AI Gastroenterology Colorectal surgery PolypAid • CNN GI Genius • CNN EndoAid • NR Gastroenterology Upper GI surgery AI artificial intelligence, CNN convolutional neural network, CRNN convolutional recurrent neural network, DRL deep reinforcement learning, EBUS-TBNA endobronchial ultrasound-guided transbronchial needle aspiration, ENT ear, nose, and throat, GAT graph attention network, GCN graph convolutional network; GI gastrointestinal, HP healthcare professionals, HPB hepato-pancreato-biliary, ML machine learning, NR not reported, OGD oesophagogastroduodenoscopy, SNN Siamese neural network, SVM support vector machine, US ultrasound, ViT vision transformer. Most studies spanned across multiple specialties, including: gastroenterology (103 studies), upper gastrointestinal surgery (57 studies), colorectal surgery (41 studies), ear, nose, and throat surgery (15 studies), pulmonology (six studies), neurosurgery (five studies), urology (five studies), hepato-pancreato-biliary surgery (four studies), obstetrics and gynaecology (four studies), anaesthetics (three studies), cardiothoracic surgery (three studies), ophthalmology (two studies), cardiology (one study), and interventional radiology (one study). One study assessed suturing skill without a specific specialty designation 109 . Procedures assessed varied widely, with oesophagogastroduodenoscopy (59 studies) and colonoscopy (34 studies) being the most frequent. Other procedures included laryngoscopy (nine studies), capsule endoscopy (eight studies), bronchoscopy (five studies), thyroid surgery (three studies), cataract surgery (two studies), cholangioscopy (two studies), colposcopy (two studies), endoscopic transsphenoidal surgery (two studies), microsurgical aneurysm clipping (two studies), nasal endoscopy (two studies), and robot-assisted radical prostatectomy (two studies). Fifteen other procedures were assessed in one study each. Most (89/146) studies used solely retrospective cohorts, whilst 21 studies used only prospective cohorts, and 21 studies combined retrospective and prospective data. Fifteen studies did not adequately report on their cohorts 12 , 23 , 26 , 35 , 59 , 60 , 72 , 82 , 91 , 104 , 113 , 118 , 135 , 139 , 147 . A priori sample size calculations were performed in 36 studies. Data from open-access sources were used in 13 studies. Poor-quality videos and frames were reported to be excluded in 78 studies. AI architectures most frequently incorporated convolutional neural networks (132 models), and less commonly vision transformers (11 models), support vector machines (three models), bag-of-words (two models), graph convolutional networks (two models), K-nearest neighbour (two models), random forests (two models), convolutional recurrent neural networks (one model), deep reinforcement learning (one model), graph attention networks (one model), K-means clustering (one model), and Siamese neural networks (one model). Four studies did not adequately report on AI model architecture 7 , 95 , 107 , 150 . Transfer learning was reported in 55 studies, with most (34/55) using the ImageNet dataset, whilst 12 studies did not specify the dataset used. Ground truth comparators used to assess accuracy of AI and healthcare professionals included: histopathology (87 studies), expert consensus (43 studies), single expert review (eight studies), clinical follow-up (six studies), urea breath test (three studies), and manual measurements (two studies). Three studies did not adequately report on their ground truth 72 , 83 , 87 . Whilst the same ground truth was used across all samples in most tasks, this was not the case for six studies 46 , 52 , 88 , 96 , 101 , 119 . To aid visual interpretability of AI output, 51 models provided bounding boxes, 35 models provided heatmaps, and 11 models provided segmentations. Training and validation of AI models are summarised in Table 2 . Most (125/146) studies trained their own AI model, whilst 21 studies assessed pre-trained models. Of the 125 studies that trained their own model, six did not report the size of their training dataset 7 , 80 , 81 , 97 , 121 , 143 , and 79 did not report on a tuning dataset. Internal validation was performed in 115 studies, where 10 studies did not report on how their training and validation datasets were split 7 , 22 , 68 , 70 , 79 , 91 , 105 , 119 , 143 , 148 . Of these 115 studies, 67 assessed video footage, 40 assessed video frames, and eight assessed a combination of both. External validation was performed in 39 studies, of which 25 assessed video footage, 11 assessed video frames, and three assessed both. Most (130/146) studies evaluated ‘AI vs unassisted healthcare professionals’, whilst 46 studies evaluated ‘AI-assisted healthcare professionals vs unassisted healthcare professionals’, and 29 studies evaluated ‘AI-assisted healthcare professionals vs AI’. The Idea, Development, Exploration, Assessment, and Long-term follow-up (IDEAL) stage of most (113/146) studies was IDEAL stage 0 (pre-clinical) 6 ; 16 studies were stage 1 (first-in-human), 13 studies were stage 2 (iterative improvement clinical studies), and four studies were stage 3 (high-quality randomised controlled trials) 95 , 101 , 122 , 144 . Table 2 Summary of training and validation of AI models in included studies Reference Task Number of videos/frames in training and tuning datasets Internal validation Number of videos/frames in internal validation dataset(s) External validation Number of videos/frames in external validation dataset(s) AI visual interpretability IDEAL stage Norton et al. 7 Classification of pancreatic adenocarcinoma and pancreatitis NR/NR NR AI: 35 frames HP: 35 videos None NA None 0 Voigt et al. 8 Classification of vocal fold paresis and healthy vocal folds 45 videos/NR 10-fold cross-validation repeated 3 times 45 videos None NA None 0 Andre et al. 9 Classification of neoplastic and non-neoplastic colorectal lesions 135 videos/NR Leave-one-patient-out cross-validation 135 videos None NA Heatmap 0 Takemura et al. 10 Classification of neoplastic and non-neoplastic colorectal lesions 1519 frames/NR Split sample 371 frames None NA None 0 Mori et al. 11 Classification of neoplastic and non-neoplastic colorectal lesions 6051 frames/NR Split sample 205 frames None NA None 0 Urban et al. 12 Internal validation 1 : detection of colorectal tumours Internal validation 2 : detection of colorectal tumours Internal validation 3 : detection of colorectal tumours Internal validation 1 : 8641 frames/NR Internal validation 2 : 8641 frames/NR Internal validation 3 : 8641 frames/9 videos Internal validation 1 : split sample Internal validation 2 : split sample Internal validation 3 : split sample Internal validation 1 : 9 videos Internal validation 2 : 11 videos Internal validation 3 : 11 videos None NA Bounding box 0 Cai et al. 13 Classification of oesophageal SCC and normal oesophagus 2428 frames/NR Split sample 187 frames None NA Bounding box 0 Sadda et al. 14 Segmentation of placental vessels 345 frames/NR Leave-one-patient-out cross-validation 345 frames None NA Segmentation 0 Wu et al. 15 Internal validation 1 : classification of neoplastic or benign gastric lesions Internal validation 2 : classification of gastric locations Internal validation 1 : 7321 frames/1830 frames Internal validation 2 : 19503 frames/4876 frames Internal validation 1 : split sample Internal validation 2 : split sample Internal validation 1 : 200 frames Internal validation 2 : 170 frames None NA Heatmap 0 Wu et al. 16 Internal validation 1 : classification of gastric locations Internal validation 2 : photo-documentation Internal validation 1 : 79844 frames + 30 videos/NR Internal validation 2 : 79844 frames + 30 videos/NR Internal validation 1 : RCT Internal validation 2 : RCT Internal validation 1 : • With AI assistance: 153 videos • Without AI assistance: 150 videos Internal validation 2 : • With AI assistance: 153 videos • Without AI assistance: 150 videos None NA None 2b Becq et al. 17 Detection of colorectal tumours NA None NA Yes 50 videos Bounding box 0 Cincar et al. 18 Classification of hyperplastic, serrated and adenoma colorectal lesions 100 videos/NR Randomly split sample 52 videos None NA None 0 Fukuda et al. 19 Internal validation 1 : classification of neoplastic and benign oesophageal lesions on non-magnified video Internal validation 2 : classification of neoplastic and benign oesophageal lesions on magnified video Internal validation 1 : 17274 frames/5277 frames Internal validation 2 : 17274 frames/5277 frames Internal validation 1 : split sample Internal validation 2 : split sample Internal validation 1 : 144 videos Internal validation 2 : 94 videos None NA Bounding box 0 Garcia-Peraza-Herrera et al. 20 Classification of normal and abnormal intrapapillary capillary loops 54194 frames/6774 frames 5-fold cross-validation AI: 6774 frames HP: 158 frames None NA Heatmap 0 Horiuchi et al. 21 Classification of neoplastic and benign gastric lesions 2570 frames/NR Temporally split 174 videos None NA Bounding box 0 Lecuyer et al. 22 Classification of phases and steps of cataract surgery Cholec80 : ~90000 frames/NR CATARACTS : ~65000 frames/NR NR 1 video None NA None 0 Li et al. 23 Classification of neoplastic and benign cervical lesions 30665 frames/NR Split sample 300 frames None NA Bounding box 0 Nakashima et al. 24 Classification of uninfected, currently infected, and post-eradication Helicobacter pylori status WLI : 6639 frames/NR LCI : 6248 frames/NR Temporally split 120 videos None NA Heatmap 0 Shimamoto et al. 25 Internal validation 1 : classification of oesophageal SCC invasion depth on non-magnified video Internal validation 2 : classification of oesophageal SCC invasion depth on magnified video Internal validation 1 : 6857 WLI frames + 17120 NBI/BLI frames/NR Internal validation 2 : 6857 WLI frames + 17120 NBI/BLI frames/NR Internal validation 1 : split sample Internal validation 2 : split sample Internal validation 1 : 102 videos Internal validation 2 : 102 videos None NA Bounding box 0 Su et al. 26 Detection of colorectal tumours 17605 frames/4173 frames RCT • With AI assistance: 308 videos • Without AI assistance: 315 videos None NA Bounding box 2b Tang et al. 27 Classification of neoplastic and benign gastric lesions 35823 frames/NR Temporally split 600 frames None NA Bounding box 0 Wang et al. 28 Detection of colorectal tumours NA None NA Yes (RCT) • With AI assistance: 184 videos • Without AI assistance: 185 videos Bounding box 2a Xue et al. 29 Classification of neoplastic and benign cervical lesions 68037 frames/9751 frames Random split sample • AI: 3887 videos • HP: 3887 videos None NA None 0 Zhou et al. 30 Classification of bowel preparation using BBPS 4764 frames/592 frames Split sample 120 frames None NA None 0 Chen et al. 31 External validation 1 : classification of presence or absence of oesophageal varices External validation 2 : classification of presence or absence of gastric varices External validation 3 : treatment suggestions for oesophageal varices External validation 4 : treatment suggestions for gastric varices External validation 1 : 11774 frames/1472 frames External validation 2 : 11774 frames/1472 frames External validation 3 : 11774 frames/1472 frames External validation 4 : 11,774 frames/1472 frames None NA Yes External validation 1 : 200 frames External validation 2 : 200 frames External validation 3 : 50 videos External validation 4 : 50 videos None 0 Cui et al. 32 Detection of nerve and dura mater 8363 frames/2091 frames Randomly split sample 12000 frames None NA Bounding box 0 Everson et al. 33 Classification of normal and abnormal intrapapillary capillary loops 54194 frames/6774 frames 5-fold cross-validation • AI: 6774 frames • HP: 158 frames None NA Heatmap 0 Jiang et al. 34 Classification of neoplastic and benign colorectal tumours, colitis, and normal colon 41115 frames/NR None NA Yes 248 frames None 1 Li et al. 35 Classification of neoplastic oesophageal lesions and normal oesophagus 4735 frames/NR Temporally split 316 frames None NA Bounding box 0 Ling et al. 36 Classification of differentiation status of gastric cancer 1774 frames/443 frames None NA Yes 822 frames None 0 Livovsky et al. 37 Detection of colorectal tumours >86 million frames/NR Split sample 1393 videos Yes 100 videos Bounding box 1 Shiroma et al. 38 Detection of oesophageal SCC 6026 WLI frames + 2402 NBI frames/NR Temporally split 40 WLI videos + 40 NBI videos None NA Bounding box 0 Tang et al. 39 Internal validation 1 : classification of intramucosal and advanced gastric cancer Internal validation 2 : classification of intramucosal and advanced gastric cancer Internal validation 1 : 3407 frames/NR Internal validation 2 : 3407 frames/NR Internal validation 1 : randomly split sample Internal validation 2 : randomly split sample Internal validation 1 : 228 frames Internal validation 2 : 54 videos None NA Heatmap 0 Waki et al. 40 Detection of oesophageal SCC 18,797 frames/NR Temporally split 100 videos None NA Heatmap 0 Wu et al. 41 Classification of neoplastic and benign gastric lesions 4167 frames/NR Randomly split sample 1329 frames None NA Bounding box 1 Xu et al. 42 Internal validation 1 : classification of presence and absence of gastric atrophy Internal validation 2 : classification of presence and absence of intestinal metaplasia Internal validation 1 : 4678 frames/520 frames Internal validation 2 : 4678 frames/520 frames Internal validation 1 : Temporally split Internal validation 2 : Temporally split Internal validation 1 : 107 frames Internal validation 2 : 107 frames None NA None 1 Yang et al. 43 Classification of oesophageal SCC and benign lesions or advanced cancer Feature extraction : 12420 frames/NR Diagnostic network : 414 frames/NR Split sample 222 frames None NA Bounding box; heatmap 1 Yoo et al. 44 Classification of carina and bronchi 6806 frames/1191 frames Split sample 200 frames None NA Heatmap 0 Zhao et al. 45 Detection of colorectal tumours 59,708 frames/NR Split sample 209 videos None NA Bounding box 1 Zhi et al. 46 Selection of representative frames of intrathoracic lymph nodes 311 videos/104 videos Temporally split 91 videos None NA None 0 Ainechi et al. 47 Detection of colorectal tumours NA Temporally split 200 videos None NA Bounding box 0 Biffi et al. 48 Classification of adenoma and non-adenoma colorectal lesions 63445 frames/8645 frames None NA Yes 513 videos Bounding box 2b He et al. 49 Internal validation : classification of neoplastic and benign gastric lesions External validation : classification of neoplastic and benign gastric lesions Internal validation : 3734 frames/933 frames Enternal validation : 3734 frames/933 frames Split sample 93 videos Yes 94 videos Heatmap 2b Hussein et al. 50 Classification of dysplastic and non-dysplastic Barrett’s oesophagus 149,030 frames/25173 frames Randomly split sample 61 frames None NA Heatmap 0 Li et al. 51 Classification of bronchial lumen segments 17,060 frames/5686 frames Split sample 372 frames None NA None 0 Lu et al. 52 Classification of endoscopic resection suitable and unsuitable colorectal tumours 12795 frames/NR None NA Yes 143 frame pairs (1 WLI and 1 NBI/BLI) Heatmap 0 Ma et al. 53 Classification of neoplastic and benign gastric lesions 3734 frames/671 frames and 101 videos Split sample 117 videos None NA Bounding box 0 Pangal et al. 54 Task 1 : prediction of success/failure in haemostasis control Task 2 : prediction of volume of blood loss 127 videos/NR Split sample 20 videos None NA None 0 Sato et al. 55 Detection of recurrent laryngeal nerve 2700 frames/300 frames Randomly split sample 40 frames None NA Segmentation 0 Tajiri et al. 56 Classification of oesophageal SCC and non-SCC 29,794 frames/NR Temporally split • AI: 147 frames • HPs: 147 videos None NA None 0 Tang et al. 57 Classification of neoplastic and benign gastric lesions 13,151 frames/NR Temporally split 600 frames None NA Bounding box 0 Tang et al. 58 Detection of colorectal tumours 26,371 frames/NR Split sample 245 videos None NA Bounding box 0 Wang et al. 59 Classification of grading performance of cataract surgical steps 150 videos/38 videos Split sample 54 videos None NA None 1 Wang et al. 60 Detection of parathyroid glands 1500 frames/NR Temporally split 20 videos None NA Bounding box 0 Wu et al. 61 Detection of gastric cancer 30,824 frames/NR Split sample 100 videos None NA Bounding box 2b Wu et al. 62 External validation 1 : classification of neoplastic and benign gastric lesions on WLI External validation 2 : classification of neoplastic and benign gastric lesions on NBI External validation 3 : classification of invasion depth on WLI External validation 4 : classification of differentiation status on NBI External validation 1 : 43296 frames/NR External validation 2 : 43296 frames/NR External validation 3 : 43296 frames/NR External validation 4 : 43296 frames/NR None NA Yes External validation 1 : 100 videos External validation 2: 100 videos External validation 3 : 28 videos External validation 4 : 28 videos Bounding box 2b Xie et al. 63 Detection of small bowel pathology 118686338 frames/29671584 frames None NA Yes 2898 videos None 0 Yuan et al. 64 Classification of neoplastic and benign oesophageal lesions 45,770 frames/NR Temporally split 1517 frames None NA Bounding box 0 Zhao et al. 65 Detection of chronic atrophic gastritis and chronic non-atrophic gastritis 3703 frames/1587 frames Temporally split 676 videos None NA Bounding box 1 Zhao et al. 66 Classification of chronic atrophic and chronic non-atrophic gastritis 3703 frames/1587 frames Temporally split 268 videos None NA Bounding box 1 Abdelrahim et al. 67 Detection of neoplastic and non-neoplastic Barrett’s oesophagus 1,090,171 frames/NR None NA Yes 75 videos Heatmap 0 Ding et al. 68 Classification of various small bowel pathologies 280,426 frames/NR NR 240 videos None NA Heatmap 0 Dong et al. 69 External validation 1 : detection of gastric cancer on WLI External validation 2 : classification of neoplastic and benign gastric lesions on NBI External validation 3 : detection of gastric cancer on WLI External validation 4 : classification of neoplastic and benign gastric lesions on NBI External validation 1 : 172058 WLI frames + 5247 NBI frames/NR External validation 2 : 172058 WLI frames + 5247 NBI frames/NR External validation 3 : 172058 WLI frames + 5247 NBI frames/NR External validation 4 : 172058 WLI frames + 5247 NBI frames/NR None NA Yes External validation 1 : 100 videos External validation 2 : 100 videos External validation 3 : 100 videos External validation 4 : 100 videos None 0 Dong et al. 70 Internal validation : classification of neoplastic and benign gastric lesions External validation : classification of neoplastic and benign gastric lesions 3612 frames/NR NR 115 videos Yes 85 videos Bounding box 0 Du et al. 71 Classification of neoplastic and benign gastric lesions 3449 frames/NR Temporally split 100 videos None NA None 0 Fang et al. 72 Classification of various laryngeal lesions 279 frames/NR Split sample 100 frames None NA Bounding box 0 Feng et al. 73 Internal validation : classification of bowel preparation using BBPS External validation : classification of bowel preparation using BBPS Internal validation : 7945 frames and 3799 videos/764 frames and 555 videos External validation : 7945 frames and 3799 videos/764 frames and 555 videos Randomly split sample 16 videos Yes 12 videos None 0 Fer et al. 74 Classification of steps in bariatric surgery 390 videos/95 videos Split sample 60 videos None NA None 0 Fockens et al. 75 Internal validation 1 : classification of neoplastic and non-dysplastic Barrett’s oesophagus Internal validation 2 : classification of neoplastic and non-dysplastic Barrett’s oesophagus Internal validation 1 : 14046 frames/200 frames and 200 videos Internal validation 2 : 14046 frames/200 frames and 200 videos Internal validation 1 : split sample Internal validation 2 : split sample Internal validation 1 : 400 frames Internal validation 2 : 188 videos None NA Bounding box 0 Hossain et al. 76 External validation 1 : detection of neoplastic or non-neoplastic adenomas on WLI External validation 2 : classification of neoplastic or non-neoplastic adenomas on NBI/BLI NA None NA Yes External validation 1 : 52 frames External validation 2 : 63 frames Bounding box 2b Koh et al. 77 Detection of colorectal tumours NA None NA Yes 298 videos Bounding box 2b Li et al. 78 Classification of neoplastic and benign laryngopharyngeal lesions 18,111 frames/6128 frames Temporally split 200 videos None NA Heatmap 0 Li et al. 79 Classification of Helicobacter pylori infection status 3803 videos/NR NR 100 videos None NA None 1 Matsuda et al. 80 Classification of pathological complete response NR/NR Randomly split sample 20 frames None NA Heatmap 0 Matsuda et al. 81 Classification of pathological complete response NR/NR Split sample 20 frames None NA Heatmap 0 Misumi et al. 82 Detection of colorectal tumours NA None NA Yes 21 videos Bounding box 2a Nehme et al. 83 Detection of colorectal tumours NA None NA Yes 1044 videos Bounding box 2b Nemoto et al. 84 Classification of colorectal cancer invasion depth 3716 frames/NR Randomly split sample 1392 frames None NA Heatmap 0 O’Hara et al. 85 Classification of various small bowel pathologies NA None NA Yes 39 videos None 0 Quek et al. 86 Classification of neoplastic and benign gastric lesions NA None NA Yes 300 frames None 0 Sedighipour Chafjiri et al. 87 Classification of gastrointestinal tract segments 105 frames Split sample 25700 frames None NA None 0 Shen et al. 88 Classification of Helicobacter pylori infection status 479 videos/120 videos Temporally split 240 videos Yes 216 videos None 1 Shi et al. 89 Internal validation 1 : classification of chronic atrophic and chronic non-atrophic gastritis Internal validation 2 : classification of chronic atrophic and chronic non-atrophic gastritis Internal validation 1 : 6126 frames/2072 frames Internal validation 2 : 6126 frames/2072 frames Internal validation 1 : randomly split sample Internal validation 2 : temporally split Internal validation 1 : 2073 frames Internal validation 2 : 118 videos Yes 600 frames Heatmap 0 Smedsrud et al. 90 Detection of colorectal tumours ~1,000,000 frames/NR Split sample 420 videos None NA Bounding box 0 Sudaveric et al. 91 Estimation of colorectal tumour size NA NR Patients: 29 videos Models: 28 videos None NA Bounding box 0 Sun et al. 92 Classification of clinically significant and clinically insignificant prostate cancer 559 videos/NR Randomly split sample 140 videos Yes 133 videos Heatmap 0 Wang et al. 93 Turing test assessing segmentation of ureter by AI and gynaecologists 7 videos/NR Split sample 1 video None NA Heatmap 0 Wang et al. 94 Classification of hyperplastic, serrated, and adenoma colorectal lesions 231,633 frames/66181 frames Split sample 33090 frames None NA Heatmap 0 Wei et al. 95 Detection of colorectal tumours NA None NA Yes • With AI assistance: 387 videos • Without AI assistance: 382 videos Bounding box 3 Xu et al. 96 Classification of representative frames of intrathoracic lymph nodes 727 videos 5-fold cross-validation 727 videos None NA None 0 Yao et al. 97 External validation 1 : classification of colorectal cancer endoscopic resection suitability External validation 2 : classification of colorectal cancer endoscopic resection suitability External validation 1 : NR/NR External validation 2 : NR/NR None NA Yes External validation 1 : 198 frames External validation 2 : 64 videos None 0 Yuan et al. 98 Detection of oesophageal SCC 7530 frames/NR None NA Yes 241 frames Segmentation 1 Yui et al. 99 Classification of inverted papilloma and chronic rhinosinusitis with nasal polyps 143167 frames/NR 8-fold cross-validation 53 videos None NA None 0 Zhang et al. 100 Detection of malignant biliary strictures 2475 frames/329 frames Randomly split sample 200 frames None NA Heatmap 0 Zhang et al. 101 Internal validation 1 : classification of oesophageal and gastric segments Internal validation 2 : detection of oesophageal and gastric pathology Internal validation 1 : 252,111 frames/NR Internal validation 2 : 252,111 frames/NR RCT Internal validation 1 : 44 videos Internal validation 2 : 44 videos None NA None 3 Zhao et al. 102 External validation 1 : classification of chronic atrophic and non-atrophic gastritis External validation 2 : classification of gastritis OLGA I/II and 0/III/IV status External validation 3 : classification of gastritis OLGA 0/I/II and III/IV status NA None NA Yes External validation 1 : 524 videos External validation 2 : 524 videos External validation 3 : 524 videos Bounding box 2a Zhou et al. 103 Classification of presence and absence of cerebral aneurysms 356,165 frames/NR Split sample 15 frames None NA Heatmap 0 Chen et al. 104 Classification of various small bowel pathologies 26,638 frames/6652 frames Randomly split sample 1013 frames None NA Heatmap 0 Choi et al. 105 Detection of various small bowel pathologies NA NR 1037286 frames None NA Heatmap 0 Furube et al. 106 External validation 1 : classification of recurrent laryngeal nerve External validation 2 : identification of recurrent laryngeal nerve External validation 1 : 120 videos/NR External validation 2 : 120 videos/NR None NA Yes External validation 1 : • With AI assistance: 4 videos • Without AI assistance: 4 videos External validation 2 : • With AI assistance: 4 videos • Without AI assistance: 4 videos Segmentation 0 Hardy et al. 107 Classification of neoplastic and benign rectal lesions NA Split sample (nested from RCT) 14 videos None NA Bounding box 0 He et al. 108 Classification of Forrest status of peptic ulcers 3868 frames/NR Temporally split 607 frames Yes 100 frames Bounding box 0 Hoffmann et al. 109 Classification of suturing skill 220 videos/48 videos Randomly split sample 46 videos None NA None 0 Jukema et al. 110 Classification of non-dysplastic Barrett’s oesophagus and Barrett’s oesophagus-related neoplasia 3468 frames/128 frames + 122 videos Split sample 90 videos None NA None 0 Kavak et al. 111 Classification of vocal fold sulcus and benign pathologies 11150 frames/2788 frames Split sample 544 frames None NA None 0 Khan et al. 112 Surgical coaching NA Temporally split • With AI assistance: 6 videos • Without AI assistance: 6 videos None NA None 1 Labrecque Langlais et al. 113 Classification of severe coronary artery stenosis and moderate or no coronary artery stenosis 800 frames and 37,266 videos/200 frames and 4608 videos Split sample 1926 videos None NA Segmentation 0 Lee et al. 114 Internal validation 1 : classification of early gastric cancer differentiation status Internal validation 2 : classification of early gastric cancer invasion depth Internal validation 3 : prediction of early gastric cancer lymphovascular invasion Internal validation 4 : prediction of early gastric cancer lymph node metastasis Internal validation 1 : 3469 frames + 122 videos/867 frames + 31 videos Internal validation 2 : 3469 frames + 122 videos/867 frames + 31 videos Internal validation 3 : 3469 frames + 122 videos/867 frames + 31 videos Internal validation 4 : 3469 frames + 122 videos/867 frames + 31 videos Internal validation 1 : split sample Internal validation 2 : split sample Internal validation 3 : split sample Internal validation 4 : split sample Internal validation 1 : 100 frames + 10 videos Internal validation 2 : 100 frames + 10 videos Internal validation 3 : 100 frames + 10 videos Internal validation 4 : 100 frames + 10 videos None NA Bounding box 0 Li et al. 115 Classification of upper GI structures 2433 videos/NR Split sample 118 videos None NA Heatmap 0 Li et al. 116 Internal validation 1 : classification of various small bowel pathologies Internal validation 2 : classification of various small bowel pathologies Internal validation 1 : 144678 frames/NR Internal validation 2 : 144678 frames/NR Internal validation 1 : temporally split Internal validation 2 : temporally split Internal validation 1 : 47586 frames Internal validation 2 : 298 videos None NA Bounding box 0 Luo et al. 117 Classification of various colorectal benign ulceration pathologies 11,706 frames/NR Randomly split sample 5989 frames None NA Heatmap 0 Meinikheim et al. 118 Detection of non-dysplastic Barrett’s oesophagus and Barrett’s oesophagus-related neoplasia 51,273 frames/NR RCT 96 videos None NA Segmentation 0 Robles-Medranda et al. 119 Classification of neoplastic and benign biliary strictures NA NR 90 videos None NA Bounding box 0 Sampieri et al. 120 Segmentation of laryngeal cancer 3146 frames/393 frames Split sample 100 frames None NA Segmentation 0 Sang et al. 121 Detection of parathyroid glands NR/NR Split sample 15 videos None NA Segmentation 0 Seager et al. 122 Detection of colorectal tumours NA None NA Yes • With AI assistance: 1015 videos • Without AI assistance: 1017 videos Bounding box 3 Shimada et al. 123 Classification of visceral pleural invasion status of lung adenocarcinoma 463 frames/NR Temporally split 46 frames None NA None 0 Tao et al. 124 Detection of fish bone impaction 3077 frames/NR Split sample 56 frames None NA Heatmap 1 Tao et al. 125 Internal validation 1 : classification of chronic atrophic and non-atrophic gastritis Internal validation 2 : classification of chronic atrophic and non-atrophic gastritis Internal validation 3 : risk stratification of chronic atrophic gastritis Internal validation 1 : 8151 frames/NR Internal validation 2 : 8151 frames/NR Internal validation 3 : 8151 frames/NR Internal validation 1 : temporally split Internal validation 2 : temporally split Internal validation 3 : temporally split Internal validation 1 : 869 frames Internal validation 2 : 119 videos Internal validation 3 : 102 photographic documentations None NA None 0 Tie et al. 126 Classification of neoplastic and benign vocal fold leucoplakia 5390 frames/NR None NA Yes 100 videos None 0 Wang et al. 127 Classification of neoplastic and benign vocal fold leucoplakia 3414 frames/NR None NA Yes 200 frames Segmentation 1 Wang et al. 128 Classification of presence and absence of parathyroid glands 32,428 frames/NR Split sample 54 videos None NA Bounding box 0 Wang et al. 129 Estimation of colorectal tumour size 2732 frames/304 frames None NA Yes 69 videos Heatmap 0 Williams et al. 130 Classification of presence and absence of cerebral aneurysms 356,165 frames/NR Split sample 15 frames None NA Heatmap 0 Xie et al. 131 Classification of various stomach and small bowel pathologies 2,672,542 frames/NR Split sample 342 videos None NA Heatmap 0 Xu et al. 132 External validation 1 : classification of polyp and non-polyp colorectal lesions External validation 2 : classification of colorectal tumours of varying sizes External validation 1 : 40266 frames/NR External validation 2 : 40,266 frames/NR None NA Yes External validation 1 : 1500 frames External validation 2 : 1200 frames Bounding box 0 Xu et al. 133 Classification of nasal malignant tumours, nasal inverted papillomas, nasal polyps, and fungal masses 3780 frames/1260 frames Temporally split 1290 frames None NA Heatmap 0 Yao et al. 134 Classification of bowel preparation using BBPS NA None NA Yes 393 videos None 2a Zhu et al. 135 Classification of renal stone composition 485 frames/NR Split sample 50 frames None NA None 0 Ahn et al. 136 Photodocumentation 95,047 frames/NR Temporally split 37 videos None NA None 0 Amante et al. 137 Classification of neoplastic and benign pulmonary nodules 41 videos/NR Randomly split sample 20 videos None NA None 0 Aoyama et al. 138 Classification of neoplastic and benign oesophageal lesions 280 videos/NR Temporally split 167 videos None NA Bounding box 0 Chen et al. 139 Classification of gastro-oesophageal valve function using Hill criteria 739 frames/185 frames None NA Yes 219 frames Heatmap 0 Feng et al. 140 External validation 1 : classification of neoplastic and benign gastric lesions External validation 2 : classification of neoplastic and benign gastric lesions External validation 1 : 11800 frames/NR External validation 2 : 11,800 frames/NR None NA Yes External validation 1 : 1289 frames External validation 2 : 130 videos Bounding box 0 He et al. 141 Classification of neoplastic and benign nasopharyngeal lesions 5406 NBI frames + 4140 WLI frames/1533 NBI frames + 1293 WLI frames Randomly split sample 200 frames None NA Heatmap 0 Khanna et al. 142 Photodocumentation NA Split sample 158 videos None NA None 0 Kim et al. 143 Classification of colorectal tumours NR/NR NR 115 videos None NA None 0 Li et al. 144 Detection of neoplastic oesophageal lesions NA Temporally split • With AI assistance first: 1698 videos • Without AI assistance first: 1698 videos None NA Segmentation 3 Liu et al. 145 Classification of surgical phases 120 videos/NR Temporally split 30 videos None NA None 0 Ma et al. 146 Classification of neoplastic and benign oesophageal lesions 22,972 frames/2084 frames Temporally split 187 videos None NA None 0 Majeedi et al. 147 Classification of glottis opening 1623 frames/NR Split sample 10 videos None NA Bounding box 0 Mezzina 148 Classification of surgical phases 143 videos/NR NR NR None NA None 0 Ogata 149 Classification of Mayo endoscopic subscore of ulcerative colitis 74,713 frames/NR Temporally split 124 videos None NA None 1 Papachrysos 150 Detection of colorectal tumours NA None NA Yes 522 videos Bounding box 0 Scheppach 151 Detection of submucosal vessels 185,151 frames/NR Split sample 101 videos None NA Segmentation 0 Tappermann 152 Prediction of early urinary continence 70 videos/15 videos Split sample 15 videos None NA None 0 AI artificial intelligence, BBPS Boston bowel preparation scale, GI gastrointestinal, HP healthcare professional, IDEAL idea,development, exploration, assessment, and long-term follow-up framework, LCI linked-colour imaging, NA not applicable, NBI narrow-band imaging, NR not reported, OLGA operative link for gastritis assessment, RCT randomised controlled trial, SCC squamous cell carcinoma, WLI white-light imaging. Summary of training and validation of AI models in included studies AI: 35 frames HP: 35 videos Internal validation 1 : detection of colorectal tumours Internal validation 2 : detection of colorectal tumours Internal validation 3 : detection of colorectal tumours Internal validation 1 : 8641 frames/NR Internal validation 2 : 8641 frames/NR Internal validation 3 : 8641 frames/9 videos Internal validation 1 : split sample Internal validation 2 : split sample Internal validation 3 : split sample Internal validation 1 : 9 videos Internal validation 2 : 11 videos Internal validation 3 : 11 videos Internal validation 1 : classification of neoplastic or benign gastric lesions Internal validation 2 : classification of gastric locations Internal validation 1 : 7321 frames/1830 frames Internal validation 2 : 19503 frames/4876 frames Internal validation 1 : split sample Internal validation 2 : split sample Internal validation 1 : 200 frames Internal validation 2 : 170 frames Internal validation 1 : classification of gastric locations Internal validation 2 : photo-documentation Internal validation 1 : 79844 frames + 30 videos/NR Internal validation 2 : 79844 frames + 30 videos/NR Internal validation 1 : RCT Internal validation 2 : RCT Internal validation 1 : • With AI assistance: 153 videos • Without AI assistance: 150 videos Internal validation 2 : • With AI assistance: 153 videos • Without AI assistance: 150 videos Internal validation 1 : classification of neoplastic and benign oesophageal lesions on non-magnified video Internal validation 2 : classification of neoplastic and benign oesophageal lesions on magnified video Internal validation 1 : 17274 frames/5277 frames Internal validation 2 : 17274 frames/5277 frames Internal validation 1 : split sample Internal validation 2 : split sample Internal validation 1 : 144 videos Internal validation 2 : 94 videos AI: 6774 frames HP: 158 frames Cholec80 : ~90000 frames/NR CATARACTS : ~65000 frames/NR WLI : 6639 frames/NR LCI : 6248 frames/NR Internal validation 1 : classification of oesophageal SCC invasion depth on non-magnified video Internal validation 2 : classification of oesophageal SCC invasion depth on magnified video Internal validation 1 : 6857 WLI frames + 17120 NBI/BLI frames/NR Internal validation 2 : 6857 WLI frames + 17120 NBI/BLI frames/NR Internal validation 1 : split sample Internal validation 2 : split sample Internal validation 1 : 102 videos Internal validation 2 : 102 videos • With AI assistance: 308 videos • Without AI assistance: 315 videos • With AI assistance: 184 videos • Without AI assistance: 185 videos • AI: 3887 videos • HP: 3887 videos External validation 1 : classification of presence or absence of oesophageal varices External validation 2 : classification of presence or absence of gastric varices External validation 3 : treatment suggestions for oesophageal varices External validation 4 : treatment suggestions for gastric varices External validation 1 : 11774 frames/1472 frames External validation 2 : 11774 frames/1472 frames External validation 3 : 11774 frames/1472 frames External validation 4 : 11,774 frames/1472 frames External validation 1 : 200 frames External validation 2 : 200 frames External validation 3 : 50 videos External validation 4 : 50 videos • AI: 6774 frames • HP: 158 frames Internal validation 1 : classification of intramucosal and advanced gastric cancer Internal validation 2 : classification of intramucosal and advanced gastric cancer Internal validation 1 : 3407 frames/NR Internal validation 2 : 3407 frames/NR Internal validation 1 : randomly split sample Internal validation 2 : randomly split sample Internal validation 1 : 228 frames Internal validation 2 : 54 videos Internal validation 1 : classification of presence and absence of gastric atrophy Internal validation 2 : classification of presence and absence of intestinal metaplasia Internal validation 1 : 4678 frames/520 frames Internal validation 2 : 4678 frames/520 frames Internal validation 1 : Temporally split Internal validation 2 : Temporally split Internal validation 1 : 107 frames Internal validation 2 : 107 frames Feature extraction : 12420 frames/NR Diagnostic network : 414 frames/NR Internal validation : classification of neoplastic and benign gastric lesions External validation : classification of neoplastic and benign gastric lesions Internal validation : 3734 frames/933 frames Enternal validation : 3734 frames/933 frames Task 1 : prediction of success/failure in haemostasis control Task 2 : prediction of volume of blood loss • AI: 147 frames • HPs: 147 videos External validation 1 : classification of neoplastic and benign gastric lesions on WLI External validation 2 : classification of neoplastic and benign gastric lesions on NBI External validation 3 : classification of invasion depth on WLI External validation 4 : classification of differentiation status on NBI External validation 1 : 43296 frames/NR External validation 2 : 43296 frames/NR External validation 3 : 43296 frames/NR External validation 4 : 43296 frames/NR External validation 1 : 100 videos External validation 2: 100 videos External validation 3 : 28 videos External validation 4 : 28 videos External validation 1 : detection of gastric cancer on WLI External validation 2 : classification of neoplastic and benign gastric lesions on NBI External validation 3 : detection of gastric cancer on WLI External validation 4 : classification of neoplastic and benign gastric lesions on NBI External validation 1 : 172058 WLI frames + 5247 NBI frames/NR External validation 2 : 172058 WLI frames + 5247 NBI frames/NR External validation 3 : 172058 WLI frames + 5247 NBI frames/NR External validation 4 : 172058 WLI frames + 5247 NBI frames/NR External validation 1 : 100 videos External validation 2 : 100 videos External validation 3 : 100 videos External validation 4 : 100 videos Internal validation : classification of neoplastic and benign gastric lesions External validation : classification of neoplastic and benign gastric lesions Internal validation : classification of bowel preparation using BBPS External validation : classification of bowel preparation using BBPS Internal validation : 7945 frames and 3799 videos/764 frames and 555 videos External validation : 7945 frames and 3799 videos/764 frames and 555 videos Internal validation 1 : classification of neoplastic and non-dysplastic Barrett’s oesophagus Internal validation 2 : classification of neoplastic and non-dysplastic Barrett’s oesophagus Internal validation 1 : 14046 frames/200 frames and 200 videos Internal validation 2 : 14046 frames/200 frames and 200 videos Internal validation 1 : split sample Internal validation 2 : split sample Internal validation 1 : 400 frames Internal validation 2 : 188 videos External validation 1 : detection of neoplastic or non-neoplastic adenomas on WLI External validation 2 : classification of neoplastic or non-neoplastic adenomas on NBI/BLI External validation 1 : 52 frames External validation 2 : 63 frames Internal validation 1 : classification of chronic atrophic and chronic non-atrophic gastritis Internal validation 2 : classification of chronic atrophic and chronic non-atrophic gastritis Internal validation 1 : 6126 frames/2072 frames Internal validation 2 : 6126 frames/2072 frames Internal validation 1 : randomly split sample Internal validation 2 : temporally split Internal validation 1 : 2073 frames Internal validation 2 : 118 videos Patients: 29 videos Models: 28 videos • With AI assistance: 387 videos • Without AI assistance: 382 videos External validation 1 : classification of colorectal cancer endoscopic resection suitability External validation 2 : classification of colorectal cancer endoscopic resection suitability External validation 1 : NR/NR External validation 2 : NR/NR External validation 1 : 198 frames External validation 2 : 64 videos Internal validation 1 : classification of oesophageal and gastric segments Internal validation 2 : detection of oesophageal and gastric pathology Internal validation 1 : 252,111 frames/NR Internal validation 2 : 252,111 frames/NR Internal validation 1 : 44 videos Internal validation 2 : 44 videos External validation 1 : classification of chronic atrophic and non-atrophic gastritis External validation 2 : classification of gastritis OLGA I/II and 0/III/IV status External validation 3 : classification of gastritis OLGA 0/I/II and III/IV status External validation 1 : 524 videos External validation 2 : 524 videos External validation 3 : 524 videos External validation 1 : classification of recurrent laryngeal nerve External validation 2 : identification of recurrent laryngeal nerve External validation 1 : 120 videos/NR External validation 2 : 120 videos/NR External validation 1 : • With AI assistance: 4 videos • Without AI assistance: 4 videos External validation 2 : • With AI assistance: 4 videos • Without AI assistance: 4 videos Classification of Forrest status of peptic ulcers • With AI assistance: 6 videos • Without AI assistance: 6 videos Internal validation 1 : classification of early gastric cancer differentiation status Internal validation 2 : classification of early gastric cancer invasion depth Internal validation 3 : prediction of early gastric cancer lymphovascular invasion Internal validation 4 : prediction of early gastric cancer lymph node metastasis Internal validation 1 : 3469 frames + 122 videos/867 frames + 31 videos Internal validation 2 : 3469 frames + 122 videos/867 frames + 31 videos Internal validation 3 : 3469 frames + 122 videos/867 frames + 31 videos Internal validation 4 : 3469 frames + 122 videos/867 frames + 31 videos Internal validation 1 : split sample Internal validation 2 : split sample Internal validation 3 : split sample Internal validation 4 : split sample Internal validation 1 : 100 frames + 10 videos Internal validation 2 : 100 frames + 10 videos Internal validation 3 : 100 frames + 10 videos Internal validation 4 : 100 frames + 10 videos Internal validation 1 : classification of various small bowel pathologies Internal validation 2 : classification of various small bowel pathologies Internal validation 1 : 144678 frames/NR Internal validation 2 : 144678 frames/NR Internal validation 1 : temporally split Internal validation 2 : temporally split Internal validation 1 : 47586 frames Internal validation 2 : 298 videos • With AI assistance: 1015 videos • Without AI assistance: 1017 videos Internal validation 1 : classification of chronic atrophic and non-atrophic gastritis Internal validation 2 : classification of chronic atrophic and non-atrophic gastritis Internal validation 3 : risk stratification of chronic atrophic gastritis Internal validation 1 : 8151 frames/NR Internal validation 2 : 8151 frames/NR Internal validation 3 : 8151 frames/NR Internal validation 1 : temporally split Internal validation 2 : temporally split Internal validation 3 : temporally split Internal validation 1 : 869 frames Internal validation 2 : 119 videos Internal validation 3 : 102 photographic documentations External validation 1 : classification of polyp and non-polyp colorectal lesions External validation 2 : classification of colorectal tumours of varying sizes External validation 1 : 40266 frames/NR External validation 2 : 40,266 frames/NR External validation 1 : 1500 frames External validation 2 : 1200 frames External validation 1 : classification of neoplastic and benign gastric lesions External validation 2 : classification of neoplastic and benign gastric lesions External validation 1 : 11800 frames/NR External validation 2 : 11,800 frames/NR External validation 1 : 1289 frames External validation 2 : 130 videos • With AI assistance first: 1698 videos • Without AI assistance first: 1698 videos AI artificial intelligence, BBPS Boston bowel preparation scale, GI gastrointestinal, HP healthcare professional, IDEAL idea,development, exploration, assessment, and long-term follow-up framework, LCI linked-colour imaging, NA not applicable, NBI narrow-band imaging, NR not reported, OLGA operative link for gastritis assessment, RCT randomised controlled trial, SCC squamous cell carcinoma, WLI white-light imaging. Seventy-six studies provided sufficient information to enable calculation of contingency tables - 61 at the internal validation level (Supplementary Table 2 ) and 21 at the external validation level (Supplementary Table 3 ) - with six studies providing contingency tables for both internal and external validation. In total, 29 different tasks across 20 studies (58 contingency tables) were included in our meta-analysis comparing ‘AI vs unassisted healthcare professionals’ at the external validation level with the hierarchical summary receiver operating characteristic (ROC) curves shown in Fig. 2a . AI had significantly greater sensitivity and specificity relative to unassisted healthcare professionals with a relative risk of 1.16 (95% CI 1.11–1.22, p  < 0.001) and 1.08 (95% CI 1.02–1.14, p  = 0.002) respectively. The hierarchical summary ROC curves following selection of the single highest accuracy contingency table for each of AI and unassisted healthcare professionals (58 contingency tables) are shown in Fig. 2b . When considering only peak performance, AI had significantly greater sensitivity, but comparable specificity relative to unassisted healthcare professionals with a relative risk of 1.12 (95% CI 1.07–1.19, p  < 0.001) and 1.04 (95% CI 0.98–1.10, p  = 0.224) respectively. Fig. 2 ‘AI vs unassisted healthcare professionals’ meta-analysis. Hierarchical summary receiver operating characteristic curves of ‘AI vs unassisted healthcare professionals’ analyses. Artificial intelligence models (AI) are in red; unassisted healthcare professionals (HP) are in blue. a ‘AI vs unassisted healthcare professionals’ analysis including all contingency tables per task at external validation level (29 tasks across 20 studies). b ‘AI vs unassisted healthcare professionals’ analysis selecting for only the single highest contingency table per task at external validation level (29 tasks across 20 studies). c ‘AI vs unassisted healthcare professionals’ analysis including all contingency tables per task at internal validation level (64 tasks across 57 studies). d ‘AI vs unassisted healthcare professionals’ analysis selecting for only the single highest contingency table per task at internal validation level (64 tasks across 57 studies). Hierarchical summary receiver operating characteristic curves of ‘AI vs unassisted healthcare professionals’ analyses. Artificial intelligence models (AI) are in red; unassisted healthcare professionals (HP) are in blue. a ‘AI vs unassisted healthcare professionals’ analysis including all contingency tables per task at external validation level (29 tasks across 20 studies). b ‘AI vs unassisted healthcare professionals’ analysis selecting for only the single highest contingency table per task at external validation level (29 tasks across 20 studies). c ‘AI vs unassisted healthcare professionals’ analysis including all contingency tables per task at internal validation level (64 tasks across 57 studies). d ‘AI vs unassisted healthcare professionals’ analysis selecting for only the single highest contingency table per task at internal validation level (64 tasks across 57 studies). At the internal validation level, 64 different tasks across 57 studies (128 contingency tables) were included in our meta-analysis with the hierarchical summary ROC curves shown in Fig. 2c . AI had significantly greater sensitivity and specificity relative to unassisted healthcare professionals with a relative risk of 1.13 (95% CI 1.09–1.18, p  < 0.001) and 1.06 (95% CI 1.02–1.10, p  = 0.004) respectively. The hierarchical summary ROC curves following selection of the single highest accuracy contingency table for each of AI and unassisted healthcare professionals (128 contingency tables) are shown in Fig. 2d . Again, when considering only peak performance, AI had significantly greater sensitivity, but comparable specificity relative to unassisted healthcare professionals with a relative risk of 1.06 (95% CI 1.02–1.10, p  < 0.001) and 0.99 (95% CI 0.96–1.02, p  = 0.577) respectively. Sensitivity analyses for all ‘AI vs unassisted healthcare professionals’ analyses are provided in Supplementary Table 4 . In total, 30 different tasks across 26 studies (60 contingency tables) were included in our meta-analysis comparing ‘AI-assisted healthcare professionals vs unassisted healthcare professionals’ with hierarchical summary ROC curves shown in Fig. 3a . AI-assisted healthcare professionals had significantly greater sensitivity and specificity relative to unassisted healthcare professionals with a relative risk of 1.18 (95% CI 1.12–1.25, p  < 0.001) and 1.05 (95% CI 1.02–1.08, p  < 0.001). As a sub-group analysis, hierarchical summary ROC curves of 26 different tasks across 24 studies (52 contingency tables) focussing on non-expert healthcare professionals are shown in Fig. 3b . AI-assisted non-expert healthcare professionals had significantly greater sensitivity and specificity relative to unassisted non-expert healthcare professionals with a relative risk of 1.19 (95% CI 1.13–1.26, p  < 0.001) and 1.06 (95% CI 1.03–1.11, p  < 0.001). Hierarchical summary ROC curves of 17 different tasks across 16 studies (34 contingency tables) focussing on expert healthcare professionals are shown in Fig. 3c . AI-assisted expert healthcare professionals had significantly greater sensitivity, but comparable specificity relative to unassisted expert healthcare professionals with a relative risk of 1.08 (95% CI 1.03–1.12, p  < 0.001) and 1.00 (95% CI 1.00–1.00, p value not applicable due to the best predictive model not modelling differences in specificity between comparators) respectively. Fig. 3 ‘AI-assisted healthcare professionals vs unassisted healthcare professionals’ meta-analysis. Hierarchical summary receiver operating characteristic curves of ‘AI-assisted healthcare professionals vs unassisted healthcare professionals’ analyses. AI-assisted healthcare professionals (HP+AI) are shown in green; unassisted healthcare professionals (HP) are in blue. a ‘AI-assisted healthcare professionals vs unassisted healthcare professionals’ analysis, including contingency tables of all healthcare professionals combined irrespective of expertise level (30 tasks across 26 studies). b ‘AI-assisted healthcare professionals vs unassisted healthcare professionals’ analysis, including contingency tables of non-expert healthcare professionals only (26 tasks across 24 studies). c ‘AI-assisted healthcare professionals vs unassisted healthcare professionals’ analysis, including contingency tables of expert healthcare professionals only (17 tasks across 16 studies). Hierarchical summary receiver operating characteristic curves of ‘AI-assisted healthcare professionals vs unassisted healthcare professionals’ analyses. AI-assisted healthcare professionals (HP+AI) are shown in green; unassisted healthcare professionals (HP) are in blue. a ‘AI-assisted healthcare professionals vs unassisted healthcare professionals’ analysis, including contingency tables of all healthcare professionals combined irrespective of expertise level (30 tasks across 26 studies). b ‘AI-assisted healthcare professionals vs unassisted healthcare professionals’ analysis, including contingency tables of non-expert healthcare professionals only (26 tasks across 24 studies). c ‘AI-assisted healthcare professionals vs unassisted healthcare professionals’ analysis, including contingency tables of expert healthcare professionals only (17 tasks across 16 studies). Sensitivity analyses for all ‘AI-assisted healthcare professionals vs unassisted healthcare professionals’ analyses are provided in Supplementary Table 5 . In total, 23 different tasks across 21 studies (46 contingency tables) were included in our meta-analysis comparing ‘AI-assisted healthcare professionals vs AI’ with hierarchical summary ROC curves shown in Fig. 4a . AI had significantly greater sensitivity, but comparable specificity relative to AI-assisted healthcare professionals with a relative risk of 0.96 (95% CI 0.93–1.00, p  = 0.048) and 1.02 (95% CI 0.99–1.06, p  = 0.235) respectively. As a sub-group analysis, hierarchical summary ROC curves of 23 different tasks across 21 studies (46 contingency tables) focussing on non-expert healthcare professionals are shown in Fig. 4b . AI had significantly greater sensitivity, but comparable specificity relative to AI-assisted non-expert healthcare professionals with a relative risk of 0.95 (95% CI 0.92–0.99, p  = 0.010) and 1.02 (95% CI 0.98–1.06, p  = 0.279) respectively. Hierarchical summary ROC curves of 15 different tasks across 14 studies (30 contingency tables) focussing on expert healthcare professionals are shown in Fig. 4c . There was no significant difference in sensitivity or specificity between AI-assisted expert healthcare professionals and AI with a relative risk of 0.99 (95% CI 0.95–1.04, p  = 0.787) and 1.03 (95% CI 0.97–1.08, p  = 0.395) respectively. Fig. 4 ‘AI-assisted healthcare professionals vs AI’ meta-analysis. Hierarchical summary receiver operating characteristic curves of ‘AI-assisted healthcare professionals vs AI’ analyses. AI-assisted healthcare professionals (HP+AI) are shown in green; artificial intelligence models (AI) are shown in red. a ‘AI-assisted healthcare professionals vs AI’ analysis, including contingency tables of all healthcare professionals combined irrespective of expertise level (23 tasks across 21 studies). b ‘AI-assisted healthcare professionals vs AI’ analysis, including contingency tables of non-expert healthcare professionals only (23 tasks across 21 studies). c ‘AI-assisted healthcare professionals vs AI’ analysis, including contingency tables of expert healthcare professionals only (15 tasks across 14 studies). Hierarchical summary receiver operating characteristic curves of ‘AI-assisted healthcare professionals vs AI’ analyses. AI-assisted healthcare professionals (HP+AI) are shown in green; artificial intelligence models (AI) are shown in red. a ‘AI-assisted healthcare professionals vs AI’ analysis, including contingency tables of all healthcare professionals combined irrespective of expertise level (23 tasks across 21 studies). b ‘AI-assisted healthcare professionals vs AI’ analysis, including contingency tables of non-expert healthcare professionals only (23 tasks across 21 studies). c ‘AI-assisted healthcare professionals vs AI’ analysis, including contingency tables of expert healthcare professionals only (15 tasks across 14 studies). Sensitivity analyses for all ‘AI-assisted healthcare professionals vs AI’ analyses are provided in Supplementary Table 6 .

The literature on the comparative performance of AI and healthcare professionals in surgical and interventional videos has been rapidly expanding, with over 90% (136/146) of included studies in this review published since 2020. This review aimed to take stock of the current literature to assess how AI performance is evaluated in the surgical and interventional specialties. Through meta-analysis, we quantitatively assessed the pooled performance of AI, unassisted healthcare professionals, and AI-assisted healthcare professionals across a range of tasks and specialties. Five principal findings were established from our review. Firstly, findings from our review provide support for the potential clinical benefit of integrating AI into the analysis of videos from surgical and interventional procedures. In short, our meta-analysis showed that: AI demonstrated greater sensitivity and comparable specificity to unassisted healthcare professionals at their respective peak performance levels for both external and internal validation; AI-assisted healthcare professionals exhibited improved sensitivity and specificity compared to healthcare professionals without AI assistance; and AI-assisted expert healthcare professionals and AI alone performed at a similar level in terms of both sensitivity and specificity. Whilst the field remains nascent, these findings are encouraging and warrant further investigation of AI in the field of surgical and interventional video analysis. Secondly, most studies to date evaluated the performance of AI alone against either no benchmark, as indicated by the significant number of studies excluded at study selection, or solely against unassisted healthcare professionals, aligning with the broader landscape of AI evaluation in other medical fields 4 , 5 . In contrast, far fewer studies evaluated AI as an assistive tool for healthcare professionals. This finding likely reflects the current early stage of development and validation of AI, with most studies included in our review being IDEAL stage 0 (pre-clinical). However, whilst evaluating AI diametrically against healthcare professionals has been valuable in demonstrating a proof-of-concept for its utility, it is not reflective of the real-world integration of AI. Instead, as a complex intervention, AI should be evaluated as close to its real-world application as possible, which is more likely to be in the form of AI-assistance, rather than autonomous functioning 6 , 153 , 154 . Thirdly, as concluded in previous reviews 4 , 5 , most studies included in our review assessed AI performance in a setting and manner not reflective of clinical practice. Most studies assessed AI and healthcare professionals in isolation without additional clinical information (e.g. history or examination findings). Furthermore, over half of studies (78/146) removed poor-quality videos and frames from their datasets before analysis, despite the performance of contemporary AI architectures remaining significantly affected when encountering low-quality data, thereby inflating reported performance compared with real-world conditions 155 . Additionally, although all included studies focused on data from surgical and interventional videos, a significant number of studies assessed performance using solely video frames rather than actual video footage. Analysis of continuous video footage, which is the envisioned media format that will be presented to AI in most real-world surgical and interventional settings, typically requires more complex models capable of capturing temporal patterns and motion dynamics 156 . Of those that built AI models to analyse continuous video footage, fewer performed a real-time assessment of AI performance that matched real clinical environments, with most studies being retrospective and in silico. Ultimately, current evaluation skews the data in favour of AI by either replicating the conditions in which AI is trained or relying too heavily on conditions that are optimal, thereby not truly reflecting the real-world. This may also explain why so few AI models survive as they proceed down their evaluative pathway in real-world settings. Future studies should ensure that their design reflects the real-world clinical environment as close as possible from the earliest stages of evaluation, to prioritise AI models with the highest potential, whilst minimising the waste of valuable resources on models unlikely to succeed. Fourthly, the generalisability of the included studies is limited by their design, as most studies in our review performed only internal validation. Nevertheless, our finding that there was no significant overestimation in the comparative performance of AI and healthcare professionals at internal and external validation is encouraging. It underscores the value of both validation types: internal validation as an initial indicator of a model’s potential, and external validation as a subsequent indicator of a model’s generalisability. Furthermore, most studies relied on high-quality video capturing equipment, which further restricts generalisability of their findings, particularly in the context of resource-limited healthcare systems. Moving forward, the development and sharing of publicly available datasets - including both algorithms and diverse, global training data - would be beneficial for improving model generalisability. Complementary strategies, such as multi-centre collaborations, federated learning across heterogenous sites, and synthetic data augmentation, will further enhance the robustness and generalisability of AI models. Fifthly, we observed substantial heterogeneity in the transparency of reporting across studies, both in terms of AI development and evaluation, limiting the reproducibility of AI models and study results. Whilst only a small minority of studies (6/125) that developed their own model failed to report on their training datasets, the majority (79/125) inadequately reported their tuning datasets. The most rigorous studies provided clear flow diagrams outlining training, tuning, and validation data splits, along with their respective sources 27 . Furthermore, poor reporting at the evaluation stage hindered, or in some cases prevented, scrutiny of reported results. The development and widespread adoption of standardised reporting guidelines for AI algorithms analysing clinical video data, building on frameworks such as PROBAST + AI and TRIPOD + AI for prediction models 157 , 158 , would significantly improve transparency and reproducibility across the field. AI integration into healthcare is unlikely to replace healthcare professionals, but rather complement their decision-making - this is particularly the case in surgical and interventional procedures, where healthcare professionals are essential for procedural execution, irrespective of any comparative difference in performance between healthcare professionals and AI in specific tasks 159 . We expect that as AI algorithms proceed along their evaluative pathway, as outlined by the IDEAL framework 6 , emphasis will shift from the proof-of-concept ‘AI versus healthcare professional’ comparisons towards an ‘AI-assisted healthcare professional’ framework. Extending this, we would suggest that all evaluations of AI, irrespective of stage of development or validation, should reflect how AI is likely to be used - as an assistive tool in surgical and interventional practice. Conceptually, adopting the ‘AI-assisted healthcare professional’ framework from the early stages of model development may yield dividends by providing a more real-world evaluation of its application, and, in turn, a more realistic roadmap for its integration. This will be important for addressing current challenges in clinical AI adoption and is supported by recent expert consensus 160 . From the clinician's perspective, continuous feedback from healthcare professionals has been shown to improve model accuracy, generalisability, and clinician utility 161 , 162 , whilst their awareness of specific AI biases can be key to mitigating them 163 . Furthermore, whilst traditional performance measures may indicate that AI performs at the same level as (or even surpasses) that of healthcare professionals with or without AI assistance, it is important to acknowledge that AI is trained to purely optimise on a mathematical basis, whereas healthcare professionals prioritise for clinical outcomes. This distinction is particularly important in the high-stakes context of surgical and interventional decision-making, where it is often more desirable to avoid rare but critical events (e.g. a potential cancer diagnosis) rather than merely maximising overall performance 164 . From the patient perspective, patient attitudes towards clinical AI systems show a strong preference for maintaining healthcare professionals within the decision-making process, maximising the benefits from both healthcare professionals and AI 165 . Importantly, although the ‘AI-assisted healthcare professional’ framework may be appealing in theory, its practical implementation presents distinct challenges. Clinicians must be able to recognise potential AI biases, a task complicated by the ‘black-box’ nature of many models, in which decision-making processes of AI are not readily interpretable 166 . This lack of transparency can lead to both over-reliance and under-reliance on AI outputs 167 . Optimising the human-AI interaction will require strategies that address both the technology itself and the human factors influencing its use. Explainable AI is a subfield that has been gaining traction in recent years as a potential solution by increasing human understanding of AI decision-making 168 . The formal integration of AI training into medical education and specialty training programmes will also be essential to ensure clinicians can use these tools to their full potential 169 . Crucially, evaluating AI from the outset in terms of its ability to assist healthcare professionals will allow early identification of issues related to human-AI interaction and trust, preventing the advancement of models that perform well in isolation but falter when integrated into clinical decision-making. Moreover, the scientific community should critically appraise the metrics that define AI success in order to better match real-world applications. This includes moving beyond simplistic measures, such as true and false positives or negatives, towards composite measures that reflect clinical priorities by weighting errors by their potential clinical consequences 170 . Furthermore, whilst perhaps initially surprising, our finding that the current performance of AI-assisted healthcare professionals is comparable to AI alone aligns with a recent meta-analysis of human-AI collaboration 171 . The likely explanation for this is that AI models are already performing close to the upper limit of current performance metrics, rendering further gains in performance by AI-assisted healthcare professionals challenging to both assess and achieve. Non-linear evaluation scales have been proposed as one potential solution for assessing incremental improvements and comparing AI, unassisted healthcare professionals, and AI-assisted healthcare professionals in the future 172 . Whilst there have been several published reviews evaluating AI for decision support using text and imaging, mainly within the diagnostic specialties, this is the first systematic review and meta-analysis (to our knowledge) to focus on videos across all surgical and interventional procedures - that is, not just for diagnostic purposes, but also for treatment. By incorporating a wide inclusion criterion of studies from any surgical or interventional procedure, we provide the most comprehensive summary of the literature to date. Given the early stages of the field at present, this decision was purposeful to ensure that our findings were generalizable across specialties and in order to guide the next phase of AI development and evaluation. Following our broad search with nearly 38,000 studies identified, we strictly selected only the highest quality of studies, resulting in less than 0.5% of initially identified studies included in our final synthesis. Rigorous meta-analytical techniques, including selection of the model with the least over- and under-fitting for each analysis, and sensitivity analyses that closely match the results from our primary analyses, provide further confidence in our findings. Despite these strengths, our study has several limitations. Although the introduction of several quality assessment tools for AI in healthcare, including PROBAST + AI and TRIPOD + AI 157 , 158 , we have not conducted a systematic assessment of potential biases across included studies. This decision stems primarily from the inherently broad scope of our review, which encompasses a diverse range of tasks, including both diagnostic and workflow analyses that cannot be uniformly assessed using a single tool. Nevertheless, we observed substantial heterogeneity in the transparency of reporting across studies, both in terms of AI development and evaluation, which prevented the construction of contingency tables for several studies otherwise eligible for meta-analysis, potentially impacting our findings. Whilst we were accepting of a large degree of between-study heterogeneity due to the broad nature of our review, our review lacks a quantitative exploration of heterogeneity beyond the between-study variance and covariance values provided in Supplementary Tables 4 – 6 . Furthermore, we acknowledge that our meta-analysis inherently favours studies, and hence AI applications, that report performance metrics that permit the construction of a 2 × 2 contingency table. Extending this, our strict inclusion criteria may introduce a selection bias towards AI applications in surgical and interventional procedures that can be quantitatively assessed. Consequently, we recognise that our review omits several important AI applications that either can only be or, at least thus far, only have been qualitatively assessed. Finally, whilst our review focused on objective performance metrics, we acknowledge that these measures alone are unlikely to fully capture the complexities of surgical and interventional tasks. Research beyond conventional metrics - particularly those assessing the subjective dynamics of clinician-AI interactions, such as an AI model’s ability to assist healthcare professionals - will be essential in determining how this technology can be safely and effectively incorporated into real-world practice 173 . In conclusion, the literature provides mounting evidence supporting the integration of AI into surgical and interventional procedures. Findings from our exploratory meta-analysis indicate that whilst early studies suggest that AI may perform better than unassisted healthcare professionals, they are limited by evaluation methods that fail to truly reflect the complexities of real-world clinical environments. More compelling, however, is the potential for AI to enhance healthcare professional performance, highlighting the need for future evaluation to shift towards an ‘AI-assisted healthcare professional’ framework. Realising this shift will introduce its own challenges, both in terms of developing AI models that complement clinical expertise and establishing robust evaluation frameworks to assess their real-world impact.

Surgical and interventional procedures are an indispensable part of modern healthcare and represent the mainstay for the diagnosis and treatment of many diseases, with estimates of at least 310 million procedures performed worldwide annually 1 . In the digital era of healthcare, and with the emergence of fields such as surgical data science 2 , there is an ever-increasing availability of surgical and interventional videos, with artificial intelligence (AI) posited as a complementary adjunct to enhance the standard of care provided by healthcare professionals. Proposed applications of AI in surgical and interventional video analysis are numerous and can be broadly categorized as spatial, such as the real-time identification of pathology or high-risk structures, and temporal, such as workflow analyses to automate procedural documentation, provide early warning prior to high-risk phases, and to predict the likelihood of adverse events 3 . Specific examples include using AI to identify polyps during colonoscopy, recognise normal anatomy to avoid during thyroid surgery, or detect which steps of the procedure are being performed during pituitary surgery. However, as highlighted by two previous landmark reviews comparing AI with healthcare professionals in medical imaging 4 , 5 , it is critical to distinguish between AI hype and theoretical applications generated from experimental studies and real-world applications evaluated by well-designed clinical studies. Furthermore, most studies to date evaluate the performance of AI alone against either no benchmark or solely against unassisted healthcare professionals. Whilst such evaluation likely reflects the current stage of development and clinical translation of AI, real-world integration of AI in healthcare is more likely to be in the form of AI-assistance, rather than autonomous functioning. As such, evaluation in a manner truly reflective of clinical practice remains crucial in assessing whether AI integration is of clinical benefit, and if so, for providing a framework for its safe and ethical clinical translation 6 . As the evidence for the application of AI to the analysis of videos from surgical and interventional procedures grows, it is now timely to systematically evaluate this evidence. To this end, our objective was to critically appraise current methodological approaches and findings relating to AI integration in surgical and interventional video analysis, with particular emphasis on how AI performance is evaluated. Through meta-analysis, we compare the performance of AI, unassisted healthcare professionals, and AI-assisted healthcare professionals in the analysis of surgical and interventional videos.

Supplemental Information Supplemental Information