Ensembles of temporal models and forward-backward smoothing for surgical phase recognition

Abstract Surgical phase recognition from endoscopic video could enable numerous context-aware technologies that impact efficiency and performance of surgeons and minimally invasive care teams. However, surgical phases can vary greatly (from seconds to minutes) due to patient factors and surgeon workflows along with many other reasons. Given this wide range of phase durations, fixed temporal parameters of neural networks and machine learning models constrain surgical phases with different temporal rhythms. To address this problem, we ensemble neural networks and other machine learning models with different architectures and temporal parameters to recognize surgical phases. The probability estimates of the ensemble are then used as priors for forward-backward smoothing to generate posteriors of surgical phases. We demonstrate the performance of this modeling process on three data sets: 1) robot-assisted inguinal hernia (five phases), 2) robot-assisted training in a porcine model (seven phases), and 3) laparoscopic cholecystectomy (Cholec80, seven phases). The results suggest that this novel method to address varying phases in different procedures holds promise for the future of surgical phase recognition. Competing Interest Statement Authors are employees at Intuitive Surgical, Inc. Funding Statement This study did not receive any funding. Author Declarations I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained. Yes The details of the IRB/oversight body that provided approval or exemption for the research described are given below: Ethics approval was provided for this work by Western IRB, Inc. Puyallup, WA. Now known as WCG Clinical Services. I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals. Yes I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance). Yes I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable. Yes Data Availability Two of the datasets used in this study are available by request; the Cholec80 dataset and the RMISTrain150 dataset. Please note that the RMISTrain150 dataset is now referred to as the SurgVU dataset released recently. One other dataset used in this study is not yet publicly available but the authors are working towards this goal and hope to release it soon.