Development of an Autonomous Robot for Fully Automated Colonoscope Insertion: The Autonomous Colonoscope Robot System (ACRS)

Development of an Autonomous Robot for Fully Automated Colonoscope Insertion: The Autonomous Colonoscope Robot System (ACRS) | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Development of an Autonomous Robot for Fully Automated Colonoscope Insertion: The Autonomous Colonoscope Robot System (ACRS) Keiichiro Kume, Tatsuru Taira, Seigo Terao, Nobuo Sakai This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7769716/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted You are reading this latest preprint version Abstract Background In the field of colonoscopy, robotic systems have been developed to support or replace human operators due to a shortage of trained endoscopists. A master-slave robotic system, the Endoscopic Operation Robot version 4 (EOR ver.4), in which a colonoscope is mounted on the slave unit, and tactile manipulation is enabled via the master unit, was developed. Using teaching data obtained from full colonoscope insertions performed on a colonoscopy training model by expert endoscopists, an artificial intelligence (AI) model was constructed to achieve autonomous insertion. Using this model, EOR ver.4 was developed as an autonomous system, the Autonomous Colonoscope Robot System (ACRS), and its ability to perform completely automated insertion was evaluated. Methods ACRS performed fully automated insertions during which insertion images and insertion times were recorded. Completely automated insertions were designated Level 4, whereas insertions requiring some manual assistance were designated Level 3. Separately from the expert who provided the training data, insertion images and times in manual insertions performed by trainees were also recorded, and they were compared with completely automated insertions (Level 4). A colonoscopy training model was used for these insertions. Results Of the 72 automated insertions at Level 3 or higher, 62 were classified as Level 4, giving a success rate of 86.1%. The average insertion time for Level 4 procedures was 2.92 ± 1.20 minutes, significantly longer than that of the expert (1.42 ± 0.31 minutes), but comparable to the time taken by trainees (2.94 ± 1.34 minutes). Conclusion ACRS demonstrated a high success rate for fully automated colonoscope insertion. Health sciences/Gastroenterology/Colonoscopy Physical sciences/Engineering/Mechanical engineering Figures Figure 1 Figure 2 Figure 3 Figure 4 Introduction Colorectal cancer is the third most common cancer worldwide and the second leading cause of cancer-related death 1 . The gold standard for early detection of colorectal cancer is colonoscopy, which is performed approximately 19 million times annually in Europe and the United States 2 , and it has been shown to reduce both the incidence and mortality of colorectal cancer 3 . However, pain-free insertion of a colonoscope requires advanced training, and a large number of skilled endoscopists need to be trained. Currently, there are not enough practitioners to meet the demand 4 , 5 . To overcome this problem, several groups have undertaken the development of robotic colonoscopy systems 6 , 7 , with two autonomous insertion robots in particular attracting attention. One of these systems is the Aer-O-Scope (GI View Ltd., Ramat-Gan, Israel), which uses a balloon-based insertion mechanism resembling that of the double-balloon enteroscope. It achieves fully automated insertion through an inchworm-like motion generated by the inflation and deflation of balloons mounted at the distal ends of both the scope and its outer sheath. The system reached commercial availability, but was withdrawn from the market in its early stages 8 . Another system, called the Magnetic Flexible Endoscope, features a catheter-like tube with a camera at its tip, which is navigated autonomously via an external magnetic arm. To date, this system has only been tested in live pigs 9 . However, due to the nature of the propulsion mechanisms specific to these systems, both of which are primarily designed for insertion operations, they struggle with navigating complex and variable colonic anatomies, including cases where insertion is difficult. Moreover, these systems are not well suited for handling the detailed observation required to detect small lesions hidden in blind spots behind colonic folds. These precise operations currently remain achievable only through manual colonoscopy performed directly by an endoscopist 10 , 11 . In light of this situation, the Endoscopic Operation Robot (EOR), a master-slave robotic system for robotic insertion of a colonoscope, was developed. The EOR has a colonoscope (PCF-240, Olympus, Tokyo, Japan) mounted on the slave unit, and insertion and retraction, rotation, and angulation can be intuitively controlled using a proprietary master unit 12 – 14 . In the present study, version 4 of the system was used, and a proper artificial intelligence (AI) model trained on insertion data from an expert endoscopist was constructed to develop an autonomous robot capable of fully automated colonoscope insertion, the Autonomous Colonoscope Robot System (ACRS), and the feasibility of this approach was validated. Methods After an overview of the specifications of EOR ver.4, the AI model developed for autonomous insertion and the mechanism by which autonomous operation is achieved are described. The method used to validate the fully automated colonoscope insertion by ACRS is then presented. This validation was conducted using Insertion Pattern 1 of a colonoscope training model (Kyoto Kagaku Co., Ltd., Kyoto, Japan). Development history and specifications of EOR ver.4 The EOR system has evolved progressively from version 1 12 , which was controlled via a joystick. In version 3, a proprietary master unit with haptic feedback was introduced, enabling intuitive one-handed insertion, retraction, and rotation operations while providing force sensation 13 , 14 . Building on this, version 4 incorporated haptic feedback into all operations, including vertical and horizontal angulation of the scope tip (Japanese Patent No. 7401075) (Fig. 1 , Supplementary Video 1). EOR ver.4 is a system capable of complete monitoring of 16 parameters, including force sensation, angle, speed, and scope insertion length for all operations, recorded at a minimum resolution of 1/1000 second, together with synchronized endoscopic video image data recorded in high definition, and both data streams can be annotated. In this study, a single expert endoscopist used EOR ver.4 to perform total colonoscope insertion using Insertion Pattern 1 of a colonoscope training model (Kyoto Kagaku Co., Ltd.), generating a teaching dataset from 100 insertions recorded at 1/100-second intervals. Of these, 12 were used to develop AI models. The expert endoscopist had experience with over 10,000 colonoscopy procedures. Development of an AI model for autonomous insertion and its mechanism of operation Selection and training of the AI models used to construct a proprietary AI model for autonomous insertion To construct a proprietary AI model capable of autonomous insertion, the existing AI models YOLOv5 15 and DenseNet-121 were used 16 . The former is an object detection model, which was used to predict the direction of angulation. The latter is a convolutional neural network (CNN) model for image recognition, which was used to predict the state of advancement (forward), retraction (backward), and angulation speed. The training dataset for YOLOv5 was created by annotating images extracted from the endoscopic video image data in the teaching dataset. Specifically, annotations were made by enclosing the target regions towards which the endoscope was to be angled within the endoscopic images. These enclosed regions are referred to as bounding boxes, and they are categorized into seven classes according to state of advancement on the image: “Hole,” “Gap,” “Wall,” “Fold (Right),” “Fold (Left),” “Fold (Up),” and “Fold (Down).” “Hole” refers to regions in which the colonic lumen is visible. “Gap” refers to regions in which the elongated, flattened lumen is visible. “Wall” refers to regions in which the intestinal wall is visible throughout the entire endoscopic image. For these three classes, the angulation direction was determined using the coordinates at the center of the bounding box. Specifically, the coordinates from the center of the image to the center of the bounding box defined the direction of angulation (Fig. 2 a). “Fold” refers primarily to regions in which colonic folds such as those in the sigmoid colon are prominently visible. In the sigmoid colon, folds overlap, resulting in complex structures. For this reason, images of colonic folds such as those in the sigmoid colon were annotated as a separate class from other image types. Folds were categorized into four classes based on the direction from which they projected: “Right,” “Left,” “Up,” and “Down.” For example, in the case of a fold projecting from the right (Fold [Right]), the direction of angulation was determined using the coordinates at the center of the left edge of the bounding box. Specifically, the coordinates from the center of the image to the center of the bounding box edge defined the direction of angulation (Fig. 2 b). For these four classes, the angulation direction was determined using the center of the bounding box edge opposite the direction in which the fold projects. The training dataset for DenseNet-121 was obtained using endoscopic images from the teaching dataset and their corresponding manipulation data. Five types of manipulation data were used: insertion length of the endoscope [mm]; torque applied during left–right angulation [N·m]; torque applied during up–down angulation [N·m]; torque applied during scope rotation [N·m]; and reactive force in the insertion direction during advancement [N]. For each paired set of images and manipulation data, the scope manipulation speed was calculated, and a one-hot vector label corresponding to the speed classification was assigned, thereby generating the final training dataset. For forward and backward manipulation, speeds were labelled in three categories: “Forward” for 15 mm/s and above; “Back” for − 15 mm/s and below; and “Stop” for all other speeds. For angulation, speeds were labelled in three categories: “Slow” for ≤ 25 mm/s; “Normal” for 25 mm/s to < 62.5 mm/s; and “Fast” for ≥ 62.5 mm/s. Training for the above two models was primarily conducted using PyTorch, a machine learning library for Python. For YOLOv5, the training dataset consisted of 2,100 images, the test dataset consisted of 470 images, and training was performed over 1,000 epochs with a batch size of 16. The confusion matrix resulting from this training process is shown in Fig. 3 a. The classification accuracy exceeded 95%, except for the Fold class, where the accuracy was 74% to 96%. It was considered that the situation in the direction of scope advancement could be predicted using the model weights obtained from the results of this training. Misclassifications were more frequent in the Fold class than in other classes, particularly between the Fold subclasses, but it was decided to adopt the weights obtained from this training in light of the fact that even manual insertions are conducted while continuously correcting decisions. The training data for DenseNet-121 included 1,745 training images and 300 test images for forward/backward manipulation, together with 1,443 training images and 187 test images for angulation. To ensure a sufficient number of training images, several image transformations were randomly combined and applied to the model input data. (The optimization method used was Adam 17 , with a learning rate set to 0.0005. Mean squared error was used as the loss function for both forward/backward manipulation and angulation. After calculating the individual losses, a weighted sum was calculated for each batch, with the loss for forward/backward manipulation weighted at 0.7, and the loss for angulation weighted at 0.3.) Training was conducted over 5,000 epochs with a batch size of 128. The resulting confusion matrices for forward/backward manipulation and angulation are shown in Fig. 3 b and Fig. 3 c, respectively. The average per-batch classification accuracy was 95.3% for forward/backward manipulation and 66.9% for angulation. An analysis of the classification accuracy for each class showed particularly high accuracy for the “Forward” and “Back” classes in forward/backward manipulation, whereas accuracy for the remaining classes was only around 60%. However, the weights obtained from this training were adopted for use in the system given that correct predictions were achieved for the majority of the data, and that “Forward,” which accounts for the majority of endoscope insertion maneuvers, had especially high accuracy. Based on the above, a proprietary AI model for autonomous insertion was successfully developed using datasets generated for YOLOv5 and DenseNet-121. Mechanism of operation using the AI model for autonomous insertion Two personal computers (PCs) were used: one PC ran the program for EOR ver.4 (EOR ver.4 PC), and the other ran the program for the AI model (AI PC). These two PCs communicated across the network via inter-process communication using named pipes. First, the EOR ver.4 PC sends a message to the AI PC to predict the appropriate operation. Upon receiving the message, the AI PC acquires the endoscopic image output from the flexible endoscope system and uses it to predict the appropriate endoscope operation. Finally, the AI PC sends the result of its prediction back to the EOR ver.4 PC, which then operates the EOR Ver.4 system accordingly based on the received instruction. The above process, including communication and AI prediction, was executed within 0.05 seconds, and by repeating this cycle every 0.05 seconds, autonomous operation was achieved. The forward speed was set to 6.5 mm/s, the backward speed to 4.0 mm/s, and the angulation speed was determined based on the prediction by the AI. For safety, the upper limit of force applied in the forward/backward direction was set to 20 N, and the torque limits for angulation in all directions were set to 0.5 N·m. With these specifications, ACRS was completed. Study design and protocol Study design and protocol Automated insertion was performed using the ACRS. The entire automated insertion process was recorded along with time measurements using a four-panel video configuration of the endoscope screen, master unit, slave unit, and colonoscope training model (Fig. 4 , Supplementary Video 2). Based on the Level of Autonomy classification for medical robots 18 , trials in which completely automated insertion was successfully achieved were designated Level 4. Trials that were completed but required manual intervention to pass through the location where the AI failed to make a decision were classified as Level 3. The success rate for Level 4 insertions and the mean insertion time (minutes) were determined. In addition, the mean insertion time (minutes) for Level 4 trials was compared with that of manual insertions performed by the expert (used as teaching data) and trainees. The trainees had experience performing fewer than 200 colonoscopy procedures, and they performed five consecutive insertion trials after completing two practice sessions to familiarize themselves with the operation of the EOR ver.4 master unit. Statistical analysis Data are expressed as mean ± standard error (SE) values. Since the Level 4 data did not follow a normal distribution on the Kolmogorov–Smirnov test, comparisons with other data were conducted using the Mann–Whitney U test with the Bonferroni correction. A p-value of < 0.05 was considered significant. All analyses were performed using EZR (version 1.68) 19 . Results A total of 87 consecutive automated insertion trials were performed using the system described above. Of these, 15 trials were excluded because image interpretation was not possible due to gel-like lubricant adhering to the tip of the endoscope. All 72 remaining trials achieved Level 3 or higher. Level 4 was attained in 62 trials, with a mean insertion time of 2.92 ± 1.20 minutes; Level 3 was attained in 10 trials, with a mean insertion time of 5.22 ± 1.67 minutes. The Level 4 success rate was 86.1%. Manual insertion by one expert over 100 trials yielded a mean insertion time of 1.42 ± 0.31 minutes. Manual insertion by six trainees over 30 trials yielded a mean insertion time of 2.94 ± 1.34 minutes. Although automated insertion at Level 4 took significantly longer than insertion by the expert, it was comparable to that of the trainees (Table 1 ). Table 1 Comparison of outcomes between AI model: Level 4 and manual insertion (experts and trainees). Time (mean ± SD), min P value vs AI model: Level 4 AI model: Level 4 2.92 ± 1.20 - Experts 1.42 ± 0.31 < 0.001 Trainees 2.94 ± 1.34 1 Min; minutes. Significant at < 0.05. Discussion The following two principles guided the development of this system. First, when pursuing AI-driven autonomy, it is essential that the current level of care in colonoscopy be maintained, and this is the ultimate goal of the system’s development. Second, although the main focus of medical AI is currently diagnostic imaging 20 , 21 , physicians remain the principal agents of action in technical or therapeutic procedures, and the focus of development has therefore been on supporting practitioners, such as through 3D reconstruction of surgical organs or navigational guidance within the surgical site 22 , 23 . In considering medical procedures or treatments currently performed by the physician as the main agent that could be handled by an AI-driven autonomous robotic system, we focused on total colonoscopy as a procedure that demands a high level of skill but operates within limited control axes and degrees of freedom, making it a technically achievable target for development. Current colonoscopy practice relies on both the colonoscope itself, which is designed to enable highly precise manipulation, and the skill of the endoscopist who operates it 10,11 . The establishment of axis-keeping shortening (AKS) and other techniques, insertion in patients with a history of abdominal surgery or complex bowel anatomy 11 , high adenoma detection rates (ADRs) through detailed observation 24 , and procedures such as endoscopic submucosal dissection (ESD) 25 , 26 for early-stage colorectal cancer would not be achievable without both of these elements. An attempt to achieve autonomous colonoscope insertion using AI has been reported by Hwang et al 27 . However, the platform used to mount the colonoscope had a similar configuration to the first-generation EOR system. In other words, because it lacked the haptic feedback functionality essential for precise manipulation, this system was limited to simple training models for the bowel and could not be applied to more complex situations. A platform that provides haptic feedback across all aspects of colonoscope manipulation (insertion and retraction, rotation, and angulation of the scope tip in all directions) and enables comprehensive monitoring of both manipulation and corresponding image data is a requirement for AI-driven autonomous insertion. To meet this requirement, EOR ver.4 was developed. However, several challenges were identified in the results of these autonomous insertion trials. To achieve completely automated insertion under Insertion Pattern 1 of the colonoscope training model, teaching data were collected using EOR ver.4 operated by an expert. EOR ver.4 is equipped with a two-stage foot switch for air feed and lens irrigation, as well as a separate foot switch for suction control. However, in the Insertion Pattern 1 procedures performed by the expert, insertion was successfully completed without the need to operate either of these foot switches due to the expert’s skill. As a result, there were no recorded instances in which the colonoscope tip became obscured by gel-like lubricant and prevented image interpretation, and thus no corresponding annotations or training data were available for that situation. Furthermore, when the forward direction becomes unclear during standard insertion, the endoscope is typically retracted to broaden the field of view and reassess the lumen before determining the next direction of advancement. However, because the expert was able to complete insertion with virtually no backward movement, there were insufficient annotations and training data for this scenario; therefore, some trials remained at Level 3. In addition, once the lumen was identified, the expert quickly adjusted the angulation to capture the forward path with minimal manipulation, whereas the AI model continuously performed incremental angulation adjustments. Combined with the decision to maintain constant speeds for insertion and retraction based on the processing capacity of the PC and the trained AI model, this resulted in longer insertion times. Although the teaching data were based on insertions performed by an expert with the aim of achieving ideal automated insertion, this AI model performed at the level of trainees. Just as endoscopists become experts through the acquisition of the various elements of insertion techniques, it is necessary to advance the system step by step towards greater proficiency by gradually incorporating these elements into the teaching data, introducing new learning challenges for the AI model to overcome, and, where necessary, integrating AI models other than YOLO or DenseNet and deploying higher-performance PCs. In conclusion, ACRS demonstrated a high success rate for fully automated colonoscope insertion. Although insertion times were at the level of trainees, the results suggest the potential for clinical application of completely automated colonoscope insertion. Development of EOR ver.4 and ACRS After an overview of the specifications of EOR ver.4, the AI model developed for autonomous insertion and the mechanism by which autonomous operation is achieved are described. Development History and Specifications of EOR ver.4 The EOR system has evolved progressively from version 1 12 , which was controlled via a joystick. In version 3, a proprietary master unit with haptic feedback was introduced, enabling intuitive one-handed insertion, retraction, and rotation operations while providing force sensation 13,14 . Building on this, version 4 incorporated haptic feedback into all operations, including vertical and horizontal angulation of the scope tip (Fig. 2, Supplementary VIDEO 2). EOR ver.4 is a system capable of complete monitoring of 16 parameters, including force sensation, angle, speed, and scope insertion length for all operations, recorded at a minimum resolution of 1/1000 second, together with synchronized endoscopic video image data recorded in high definition, and both data streams can be annotated. In this study, a single expert endoscopist used EOR ver.4 to perform total colonoscope insertion using Insertion Pattern 1 of a colonoscope training model (Kyoto Kagaku Co., Ltd.), generating a teaching dataset from 100 insertions recorded at 1/100-second intervals. Of these, 12 were used to develop AI models. The expert endoscopist had experience with over 10,000 colonoscopy procedures. Development of an AI Model for Autonomous Insertion and Its Mechanism of Operation Selection and training of the AI models used to construct a proprietary AI model for autonomous insertion To construct a proprietary AI model capable of autonomous insertion, the existing AI models YOLOv5 25 and DenseNet-121 were used 26 . The former is an object detection model, which was used to predict the direction of angulation. The latter is a convolutional neural network (CNN) model for image recognition, which was used to predict the state of advancement (forward), retraction (backward), and angulation speed. The training dataset for YOLOv5 was created by annotating images extracted from the endoscopic video image data in the teaching dataset. Specifically, annotations were made by enclosing the target regions towards which the endoscope was to be angled within the endoscopic images. These enclosed regions are referred to as bounding boxes, and they are categorized into seven classes according to state of advancement on the image: “Hole,” “Gap,” “Wall,” “Fold (Right),” “Fold (Left),” “Fold (Up),” and “Fold (Down).” “Hole” refers to regions in which the colonic lumen is visible. “Gap” refers to regions in which the elongated, flattened lumen is visible. “Wall” refers to regions in which the intestinal wall is visible throughout the entire endoscopic image. For these three classes, the angulation direction was determined using the coordinates at the center of the bounding box. Specifically, the coordinates from the center of the image to the center of the bounding box defined the direction of angulation (Fig. 2A). “Fold” refers primarily to regions in which colonic folds such as those in the sigmoid colon are prominently visible. In the sigmoid colon, folds overlap, resulting in complex structures. For this reason, images of colonic folds such as those in the sigmoid colon were annotated as a separate class from other image types. Folds were categorized into four classes based on the direction from which they projected: “Right,” “Left,” “Up,” and “Down.” For example, in the case of a fold projecting from the right (Fold [Right]), the direction of angulation was determined using the coordinates at the center of the left edge of the bounding box. Specifically, the coordinates from the center of the image to the center of the bounding box edge defined the direction of angulation (Fig. 2B). For these four classes, the angulation direction was determined using the center of the bounding box edge opposite the direction in which the fold projects. The training dataset for DenseNet-121 was obtained using endoscopic images from the teaching dataset and their corresponding manipulation data. Five types of manipulation data were used: insertion length of the endoscope [mm]; torque applied during left–right angulation [N·m]; torque applied during up–down angulation [N·m]; torque applied during scope rotation [N·m]; and reactive force in the insertion direction during advancement [N]. For each paired set of images and manipulation data, the scope manipulation speed was calculated, and a one-hot vector label corresponding to the speed classification was assigned, thereby generating the final training dataset. For forward and backward manipulation, speeds were labelled in three categories: “Forward” for 15 mm/s and above; “Back” for –15 mm/s and below; and “Stop” for all other speeds. For angulation, speeds were labelled in three categories: “Slow” for ≤25 mm/s; “Normal” for 25 mm/s to <62.5 mm/s; and “Fast” for ≥62.5 mm/s. Training for the above two models was primarily conducted using PyTorch, a machine learning library for Python. For YOLOv5, the training dataset consisted of 2,100 images, the test dataset consisted of 470 images, and training was performed over 1,000 epochs with a batch size of 16. The confusion matrix resulting from this training process is shown in Fig. 4A. The classification accuracy exceeded 95%, except for the Fold class, where the accuracy was 74% to 96%. It was considered that the situation in the direction of scope advancement could be predected using the model weights obtained from the results of this training. Misclassifications were more frequent in the Fold class than in other classes, particularly between the Fold subclasses, but it was decided to adopt the weights obtained from this training in light of the fact that even manual insertions are conducted while continuously correcting decisions. The training data for DenseNet-121 included 1,745 training images and 300 test images for forward/backward manipulation, together with 1,443 training images and 187 test images for angulation. To ensure a sufficient number of training images, several image transformations were randomly combined and applied to the model input data. (The optimization method used was Adam 27 , with a learning rate set to 0.0005. Mean squared error was used as the loss function for both forward/backward manipulation and angulation. After calculating the individual losses, a weighted sum was calculated for each batch, with the loss for forward/backward manipulation weighted at 0.7, and the loss for angulation weighted at 0.3.) Training was conducted over 5,000 epochs with a batch size of 128. The resulting confusion matrices for forward/backward manipulation and angulation are shown in Fig. 4B and Fig. 4C, respectively. The average per-batch classification accuracy was 95.3% for forward/backward manipulation and 66.9% for angulation. An analysis of the classification accuracy for each class showed particularly high accuracy for the “Forward” and “Back” classes in forward/backward manipulation, whereas accuracy for the remaining classes was only around 60%. However, the weights obtained from this training were adopted for use in the system given that correct predictions were achieved for the majority of the data, and that “Forward,” which accounts for the majority of endoscope insertion maneuvers, had especially high accuracy. Based on the above, a proprietary AI model for autonomous insertion was successfully developed using datasets generated for YOLOv5 and DenseNet-121. Mechanism of Operation using the AI Model for Autonomous Insertion Two personal computers (PCs) were used: one PC ran the program for EOR ver.4 (EOR ver.4 PC), and the other ran the program for the AI model (AI PC). These two PCs communicated across the network via inter-process communication using named pipes. First, the EOR ver.4 PC sends a message to the AI PC to predict the appropriate operation. Upon receiving the message, the AI PC acquires the endoscopic image output from the flexible endoscope system and uses it to predict the appropriate endoscope operation. Finally, the AI PC sends the result of its prediction back to the EOR ver.4 PC, which then operates the EOR Ver.4 system accordingly based on the received instruction. The above process, including communication and AI prediction, was executed within 0.05 seconds, and by repeating this cycle every 0.05 seconds, autonomous operation was achieved. The forward speed was set to 6.5 mm/s, the backward speed to 4.0 mm/s, and the angulation speed was determined based on the prediction by the AI. For safety, the upper limit of force applied in the forward/backward direction was set to 20 N, and the torque limits for angulation in all directions were set to 0.5 N·m. With these specifications, ACRS was completed. Declarations Competing Interests None Acknowledgments This research was supported by AMED under Grant Number JP18lm0203009. Author Contributions K. KUME.: Planning and development of EOR ver.4 and the AI model; overall planning and supervision of the research; acquisition of training data and manual data; statistical analysis; drafting of the manuscript. R. TAIRA: Central role in development and explanation of the AI model; trial insertions using the AI model. S. TERAO: Development of the AI model; trial insertions using the AI model. N. SAKAI: Development of EOR ver.4; development and supervision of the AI model; trial insertions using the AI model; technical oversight. References Sung, H. et al. Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J. Clin. 71, 209-249 (2021). Joseph, D. A. et al. Colorectal cancer screening: Estimated future colonoscopy need and current volume and capacity. Cancer 122, 2479-2486 (2016). Bretthauer, M. et al. Effect of colonoscopy screening on risks of colorectal cancer and related death. N. Engl. J. Med. 387, 1547-1556 (2022). Spier, B. J. et al. Colonoscopy training in gastroenterology fellowships: Determining competence. Gastrointest. Endosc. 71, 319-324 (2010). Lee, S.-H., Park, Y.-K., Lee, D.-J. & Kim, K.-M. Colonoscopy procedural skills and training for new beginners. World J. Gastroenterol. 20, 16984-16995 (2014). Winters, C., Subramanian, V. & Valdastri, P. Robotic, self-propelled, self-steerable, and disposable colonoscopes: Reality or pipe dream? A state of the art review. World J. Gastroenterol. 28, 5093-5110 (2022). Ciuti, G. et al. Frontiers of robotic colonoscopy: A comprehensive review of robotic colonoscopes and technologies. J. Clin. Med. 9, 1648 (2020). Vucelic, B. et al. The aer-o-scope: Proof of concept of a pneumatic, skill-independent, self-propelling, self-navigating colonoscope. Gastroenterology 130, 672-677 (2006). Slawinski, P. R. et al. Autonomously controlled magnetic flexible endoscope for colon exploration. Gastroenterology 154, 1577-1579.e1 (2018). Mazurek, M. et al. Association between endoscopist specialty and colonoscopy quality: A systematic review and meta-analysis. Clin. Gastroenterol. Hepatol. 20, 1931-1946 (2022). Hsu, W. F. & Chiu, H. M. Optimization of colonoscopy quality: Comprehensive review of the literature and future perspectives. Dig. Endosc. 35, 822-834 (2023). Kume, K., Kuroki, T., Sugihara, T. & Shinngai, M. Development of a novel endoscopic manipulation system: The endoscopic operation robot. World J. Gastrointest. Endosc. 3, 145-150 (2011). Kume, K., Sakai, N. & Goto, T. Development of a novel endoscopic manipulation system: The endoscopic operation robot ver.3. Endoscopy 47, 815-819 (2015). Kume, K., Sakai, N. & Goto, T. Haptic feedback is useful in remote manipulation of flexible endoscopes. Endosc. Int. Open 6, E1134-E1139 (2018). Yang, G. Z. et al. Medical robotics - Regulatory, ethical, and legal considerations for increasing levels of autonomy. Sci. Robot. 2, eaam8638 (2017). Kanda, Y. Investigation of the freely available easy-to-use software “EZR” for medical statistics. Bone Marrow Transplant. 48, 452-458 (2013). Zhou, S. K. et al. A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises. Proc. IEEE Inst. Electr. Electron. Eng. 109, 820-838 (2021).. Okagawa, Y., Abe, S., Yamada, M., Oda, I. & Saito, Y. Artificial intelligence in endoscopy. Dig. Dis. Sci. 67, 1553-1572 (2022). Moglia, A., Georgiou, K., Georgiou, E., Satava, R. M. & Cuschieri, A. A systematic review on artificial intelligence in robot-assisted surgery. Int. J. Surg. 95, 106151 (2021) Göbel, B., Reiterer, A. & Möller, K. Image-based 3D reconstruction in laparoscopy: A review focusing on the quantitative evaluation by applying the reconstruction error. J. Imaging 10 , 180 (2024). Corley, D. A. et al. Adenoma detection rate and risk of colorectal cancer and death. N. Engl. J. Med. 370, 1298-1306 (2014). Ohata, K. et al. Long-term outcomes after endoscopic submucosal dissection for large colorectal epithelial neoplasms: A prospective, multicenter, cohort trial from Japan. Gastroenterology 163, 1423-1434.e2 (2022). Kume, K. Endoscopic therapy for early gastric cancer: Standard techniques and recent advances in ESD. World J. Gastroenterol. 20, 6425-6432 (2014). Hwang, B., Changbeom, S., Seonggun, J. et al. A preliminary study on autonomous robotic colonoscopy via deep neural network. 13th International Conference on Control, Automation and Information Sciences (ICCAIS) (2024). Redmon, J., Divvala, S., Girshick, R. & Farhadi, A. You only look once: Unified, real-time object detection. In proceedings of the IEEE conference on computer vision and pattern recognition 779-788 (2016). Huang, G., Liu, Z., Maaten, L. & Weinberger, K. Q. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition 4700-4708 (2017). Kingma, D. P. & Ba, J. Adam: A method for stochastic optimization. ArXiv:1412.6980 (2014). Additional Declarations There is NO Competing Interest. Supplementary Files SupplementaryVIDEO.zip Supplementary Video 1: The ACRS was used to perform total colonoscopy using a colonoscopy training model with an insertion time to the cecum of 152 seconds (un cut video). Supplementary Video 2: The EOR ver.4 was used to perform total colonoscopy using a colonoscopy training model with an insertion time to the cecum of 53 seconds (un cut video). Cite Share Download PDF Status: Under Review Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7769716","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":529305401,"identity":"5c5bb284-dac8-4299-8acc-3704b44dc0ae","order_by":0,"name":"Keiichiro Kume","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA9UlEQVRIiWNgGAWjYBACCQY2EGUDxAkgxgGoOA9BLWmkazmMrgUPkGxgS5O6UXE+mp89gU3iQ8UdBoMDzA8/MMjcwalFmoHtmHTOmdu5M3sesEnOOPMMqIXNWIKB5xlOLXLyz9ukc9tu5264kcAmzdt2uH7DAQYzoF8O49bCwA7U8u9c7n6wln+Hgbawf8OrBeyw3IYDuRskQFoaQFp48NsC9H6ydc6x5NwZZx42W844dphB8jBPsUQCHr9IHGAzvJ1TY5fb35588MaHmsMMfMfbN3742IM7xJAAYwOEZgbixJ4DxGhBAT9I1zIKRsEoGAXDFgAAvcJSwsrbxJsAAAAASUVORK5CYII=","orcid":"https://orcid.org/0000-0002-0444-4739","institution":"University of Occupational and Environmental Health, Japan","correspondingAuthor":true,"prefix":"","firstName":"Keiichiro","middleName":"","lastName":"Kume","suffix":""},{"id":529305402,"identity":"028473b8-b804-49b9-8030-6c1f8d193f09","order_by":1,"name":"Tatsuru Taira","email":"","orcid":"","institution":"Kyushu Institute of Technology","correspondingAuthor":false,"prefix":"","firstName":"Tatsuru","middleName":"","lastName":"Taira","suffix":""},{"id":529305403,"identity":"36824d26-6e9c-4c94-ab58-7c99e45b48ec","order_by":2,"name":"Seigo Terao","email":"","orcid":"","institution":"Kyushu Institute of Technology","correspondingAuthor":false,"prefix":"","firstName":"Seigo","middleName":"","lastName":"Terao","suffix":""},{"id":529305404,"identity":"e3aa38a9-bcb8-46ca-b5cf-8e8b46516f1a","order_by":3,"name":"Nobuo Sakai","email":"","orcid":"","institution":"Kyushu Institute of Technology","correspondingAuthor":false,"prefix":"","firstName":"Nobuo","middleName":"","lastName":"Sakai","suffix":""}],"badges":[],"createdAt":"2025-10-03 02:25:15","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7769716/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7769716/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":96654465,"identity":"29247dbf-0250-4f57-83e3-670fc29ca42c","added_by":"auto","created_at":"2025-11-24 16:44:40","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":1109263,"visible":true,"origin":"","legend":"\u003cp\u003eThe robot operation monitor that displays four divided images; these show the operation, as captured by the three robot operation cameras, of the master unit (right lower), slave unit (right upper), colonoscopy training model (left lower), and the endoscope monitor (left upper).\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-7769716/v1/ce63f008b5aa797b5065fe9d.png"},{"id":96708798,"identity":"bbb88647-2074-421b-83c6-5d5975667f0e","added_by":"auto","created_at":"2025-11-25 10:05:29","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":596226,"visible":true,"origin":"","legend":"\u003cp\u003eFig. 2A: The Endoscopic Operation Robot (EOR) ver. 4. and the colonoscopy training model; a master unit (1), a slave unit (2), a PC (3), an endoscope monitor (4), a robot operation monitor (5), the colonoscope (Olympus PCF-240; Tokyo, Japan) (6), an endoscope system (Olympus CLV-U240D, CV-240; Tokyo, Japan) (7) and the colonoscopy training model (8).\u003c/p\u003e\n\u003cp\u003eFig. 2B: The master unit of EOR ver. 4 consists of a knob-like rotating part (1) (rotating knob), a joystick (2) and a linear slider (3).\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-7769716/v1/5dfa5fc822beb0b93b3365df.png"},{"id":96654468,"identity":"94f5db20-94e7-49c3-99c3-57900ffa9ee6","added_by":"auto","created_at":"2025-11-24 16:44:40","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":608188,"visible":true,"origin":"","legend":"\u003cp\u003eFig. 3 (A) Angulation direction based on YOLOv5. (A) Hole: The detected Hole is enclosed by the bounding box (red frame). The angulation direction (red arrow) is determined as being from the center coordinates of the display (gray dot) to the center coordinates of the bounding box (red dot).\u003c/p\u003e\n\u003cp\u003eFig. 3 (B) Fold: When the Fold class protruding from the right side is detected, to avoid the fold, the angulation direction (blue arrow) is determined as being toward the center coordinates (blue dot) of the left edge of the bounding box (blue frame).\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-7769716/v1/ee85c1ef22877846e15485c8.png"},{"id":96654467,"identity":"b7b328aa-2666-49d9-aad4-d3702ed76572","added_by":"auto","created_at":"2025-11-24 16:44:40","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":104445,"visible":true,"origin":"","legend":"\u003cp\u003eFig. 4 (A) Confusion matrices from learning by YOLOv5 and DenseNet-121 training results (vertical axis, actual = actual measurements; horizontal axis: predict = predictions). (A) YOLOv5 confusion matrix.\u003c/p\u003e\n\u003cp\u003eFig. 4 (B) DenseNet-121 confusion matrix for forward/backward manipulation.\u003c/p\u003e\n\u003cp\u003eFig. 4 (C) DenseNet-121 confusion matrix for angulation.\u003c/p\u003e","description":"","filename":"4.png","url":"https://assets-eu.researchsquare.com/files/rs-7769716/v1/47f45a3f8825d93441689dea.png"},{"id":96712202,"identity":"bf0005d8-d517-4707-bbf3-b340f2589b1f","added_by":"auto","created_at":"2025-11-25 10:15:25","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":3973933,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7769716/v1/c07bd0a3-6eb8-4376-be96-33e41cded333.pdf"},{"id":96654470,"identity":"7f906cd9-168f-4b2b-91c0-c3e84eaccf6d","added_by":"auto","created_at":"2025-11-24 16:44:42","extension":"zip","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":161244629,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eSupplementary Video 1:\u003c/strong\u003e The ACRS was used to perform total colonoscopy using a colonoscopy training model with an insertion time to the cecum of 152 seconds (un cut video).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSupplementary Video 2:\u003c/strong\u003e The EOR ver.4 was used to perform total colonoscopy using a colonoscopy training model with an insertion time to the cecum of 53 seconds (un cut video).\u003c/p\u003e","description":"","filename":"SupplementaryVIDEO.zip","url":"https://assets-eu.researchsquare.com/files/rs-7769716/v1/f26d51994bd371dc66f62cee.zip"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"Development of an Autonomous Robot for Fully Automated Colonoscope Insertion: The Autonomous Colonoscope Robot System (ACRS)","fulltext":[{"header":"Introduction","content":"\u003cp\u003eColorectal cancer is the third most common cancer worldwide and the second leading cause of cancer-related death\u003csup\u003e\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u003c/sup\u003e. The gold standard for early detection of colorectal cancer is colonoscopy, which is performed approximately 19\u0026nbsp;million times annually in Europe and the United States\u003csup\u003e\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e, and it has been shown to reduce both the incidence and mortality of colorectal cancer\u003csup\u003e\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e\u003c/sup\u003e. However, pain-free insertion of a colonoscope requires advanced training, and a large number of skilled endoscopists need to be trained. Currently, there are not enough practitioners to meet the demand\u003csup\u003e\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e,\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e\u003c/sup\u003e. To overcome this problem, several groups have undertaken the development of robotic colonoscopy systems\u003csup\u003e\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e,\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e\u003c/sup\u003e, with two autonomous insertion robots in particular attracting attention. One of these systems is the Aer-O-Scope (GI View Ltd., Ramat-Gan, Israel), which uses a balloon-based insertion mechanism resembling that of the double-balloon enteroscope. It achieves fully automated insertion through an inchworm-like motion generated by the inflation and deflation of balloons mounted at the distal ends of both the scope and its outer sheath. The system reached commercial availability, but was withdrawn from the market in its early stages\u003csup\u003e\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e\u003c/sup\u003e. Another system, called the Magnetic Flexible Endoscope, features a catheter-like tube with a camera at its tip, which is navigated autonomously via an external magnetic arm. To date, this system has only been tested in live pigs\u003csup\u003e\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e\u003c/sup\u003e. However, due to the nature of the propulsion mechanisms specific to these systems, both of which are primarily designed for insertion operations, they struggle with navigating complex and variable colonic anatomies, including cases where insertion is difficult. Moreover, these systems are not well suited for handling the detailed observation required to detect small lesions hidden in blind spots behind colonic folds. These precise operations currently remain achievable only through manual colonoscopy performed directly by an endoscopist\u003csup\u003e\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e,\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e\u003cp\u003eIn light of this situation, the Endoscopic Operation Robot (EOR), a master-slave robotic system for robotic insertion of a colonoscope, was developed. The EOR has a colonoscope (PCF-240, Olympus, Tokyo, Japan) mounted on the slave unit, and insertion and retraction, rotation, and angulation can be intuitively controlled using a proprietary master unit\u003csup\u003e\u003cspan additionalcitationids=\"CR13\" citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e\u003c/sup\u003e. In the present study, version 4 of the system was used, and a proper artificial intelligence (AI) model trained on insertion data from an expert endoscopist was constructed to develop an autonomous robot capable of fully automated colonoscope insertion, the Autonomous Colonoscope Robot System (ACRS), and the feasibility of this approach was validated.\u003c/p\u003e"},{"header":"Methods","content":"\u003cp\u003eAfter an overview of the specifications of EOR ver.4, the AI model developed for autonomous insertion and the mechanism by which autonomous operation is achieved are described. The method used to validate the fully automated colonoscope insertion by ACRS is then presented. This validation was conducted using Insertion Pattern 1 of a colonoscope training model (Kyoto Kagaku Co., Ltd., Kyoto, Japan).\u003c/p\u003e\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e\u003ch2\u003eDevelopment history and specifications of EOR ver.4\u003c/h2\u003e\u003cp\u003eThe EOR system has evolved progressively from version 1\u003csup\u003e12\u003c/sup\u003e, which was controlled via a joystick. In version 3, a proprietary master unit with haptic feedback was introduced, enabling intuitive one-handed insertion, retraction, and rotation operations while providing force sensation\u003csup\u003e\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e,\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e\u003c/sup\u003e. Building on this, version 4 incorporated haptic feedback into all operations, including vertical and horizontal angulation of the scope tip (Japanese Patent No. 7401075) (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e1\u003c/span\u003e, Supplementary Video 1). EOR ver.4 is a system capable of complete monitoring of 16 parameters, including force sensation, angle, speed, and scope insertion length for all operations, recorded at a minimum resolution of 1/1000 second, together with synchronized endoscopic video image data recorded in high definition, and both data streams can be annotated. In this study, a single expert endoscopist used EOR ver.4 to perform total colonoscope insertion using Insertion Pattern 1 of a colonoscope training model (Kyoto Kagaku Co., Ltd.), generating a teaching dataset from 100 insertions recorded at 1/100-second intervals. Of these, 12 were used to develop AI models. The expert endoscopist had experience with over 10,000 colonoscopy procedures.\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003c/div\u003e\n\u003ch3\u003eDevelopment of an AI model for autonomous insertion and its mechanism of operation\u003c/h3\u003e\n\u003cp\u003e\u003cem\u003eSelection and training of the AI models used to construct a proprietary AI model for autonomous insertion\u003c/em\u003e\u003c/p\u003e\u003cp\u003eTo construct a proprietary AI model capable of autonomous insertion, the existing AI models YOLOv5\u003csup\u003e15\u003c/sup\u003e and DenseNet-121 were used\u003csup\u003e\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e\u003c/sup\u003e. The former is an object detection model, which was used to predict the direction of angulation. The latter is a convolutional neural network (CNN) model for image recognition, which was used to predict the state of advancement (forward), retraction (backward), and angulation speed.\u003c/p\u003e\u003cp\u003eThe training dataset for YOLOv5 was created by annotating images extracted from the endoscopic video image data in the teaching dataset. Specifically, annotations were made by enclosing the target regions towards which the endoscope was to be angled within the endoscopic images. These enclosed regions are referred to as bounding boxes, and they are categorized into seven classes according to state of advancement on the image: \u0026ldquo;Hole,\u0026rdquo; \u0026ldquo;Gap,\u0026rdquo; \u0026ldquo;Wall,\u0026rdquo; \u0026ldquo;Fold (Right),\u0026rdquo; \u0026ldquo;Fold (Left),\u0026rdquo; \u0026ldquo;Fold (Up),\u0026rdquo; and \u0026ldquo;Fold (Down).\u0026rdquo; \u0026ldquo;Hole\u0026rdquo; refers to regions in which the colonic lumen is visible. \u0026ldquo;Gap\u0026rdquo; refers to regions in which the elongated, flattened lumen is visible. \u0026ldquo;Wall\u0026rdquo; refers to regions in which the intestinal wall is visible throughout the entire endoscopic image. For these three classes, the angulation direction was determined using the coordinates at the center of the bounding box. Specifically, the coordinates from the center of the image to the center of the bounding box defined the direction of angulation (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e2\u003c/span\u003ea). \u0026ldquo;Fold\u0026rdquo; refers primarily to regions in which colonic folds such as those in the sigmoid colon are prominently visible. In the sigmoid colon, folds overlap, resulting in complex structures. For this reason, images of colonic folds such as those in the sigmoid colon were annotated as a separate class from other image types. Folds were categorized into four classes based on the direction from which they projected: \u0026ldquo;Right,\u0026rdquo; \u0026ldquo;Left,\u0026rdquo; \u0026ldquo;Up,\u0026rdquo; and \u0026ldquo;Down.\u0026rdquo; For example, in the case of a fold projecting from the right (Fold [Right]), the direction of angulation was determined using the coordinates at the center of the left edge of the bounding box. Specifically, the coordinates from the center of the image to the center of the bounding box edge defined the direction of angulation (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e2\u003c/span\u003eb). For these four classes, the angulation direction was determined using the center of the bounding box edge opposite the direction in which the fold projects.\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003eThe training dataset for DenseNet-121 was obtained using endoscopic images from the teaching dataset and their corresponding manipulation data. Five types of manipulation data were used: insertion length of the endoscope [mm]; torque applied during left\u0026ndash;right angulation [N\u0026middot;m]; torque applied during up\u0026ndash;down angulation [N\u0026middot;m]; torque applied during scope rotation [N\u0026middot;m]; and reactive force in the insertion direction during advancement [N]. For each paired set of images and manipulation data, the scope manipulation speed was calculated, and a one-hot vector label corresponding to the speed classification was assigned, thereby generating the final training dataset. For forward and backward manipulation, speeds were labelled in three categories: \u0026ldquo;Forward\u0026rdquo; for 15 mm/s and above; \u0026ldquo;Back\u0026rdquo; for \u0026minus;\u0026thinsp;15 mm/s and below; and \u0026ldquo;Stop\u0026rdquo; for all other speeds. For angulation, speeds were labelled in three categories: \u0026ldquo;Slow\u0026rdquo; for \u0026le;\u0026thinsp;25 mm/s; \u0026ldquo;Normal\u0026rdquo; for 25 mm/s to \u0026lt;\u0026thinsp;62.5 mm/s; and \u0026ldquo;Fast\u0026rdquo; for \u0026ge;\u0026thinsp;62.5 mm/s.\u003c/p\u003e\u003cp\u003eTraining for the above two models was primarily conducted using PyTorch, a machine learning library for Python.\u003c/p\u003e\u003cp\u003eFor YOLOv5, the training dataset consisted of 2,100 images, the test dataset consisted of 470 images, and training was performed over 1,000 epochs with a batch size of 16. The confusion matrix resulting from this training process is shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig7\" class=\"InternalRef\"\u003e3\u003c/span\u003ea. The classification accuracy exceeded 95%, except for the Fold class, where the accuracy was 74% to 96%. It was considered that the situation in the direction of scope advancement could be predicted using the model weights obtained from the results of this training. Misclassifications were more frequent in the Fold class than in other classes, particularly between the Fold subclasses, but it was decided to adopt the weights obtained from this training in light of the fact that even manual insertions are conducted while continuously correcting decisions.\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003eThe training data for DenseNet-121 included 1,745 training images and 300 test images for forward/backward manipulation, together with 1,443 training images and 187 test images for angulation. To ensure a sufficient number of training images, several image transformations were randomly combined and applied to the model input data. (The optimization method used was Adam\u003csup\u003e\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e\u003c/sup\u003e, with a learning rate set to 0.0005. Mean squared error was used as the loss function for both forward/backward manipulation and angulation. After calculating the individual losses, a weighted sum was calculated for each batch, with the loss for forward/backward manipulation weighted at 0.7, and the loss for angulation weighted at 0.3.) Training was conducted over 5,000 epochs with a batch size of 128. The resulting confusion matrices for forward/backward manipulation and angulation are shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig7\" class=\"InternalRef\"\u003e3\u003c/span\u003eb and Fig.\u0026nbsp;\u003cspan refid=\"Fig7\" class=\"InternalRef\"\u003e3\u003c/span\u003ec, respectively. The average per-batch classification accuracy was 95.3% for forward/backward manipulation and 66.9% for angulation. An analysis of the classification accuracy for each class showed particularly high accuracy for the \u0026ldquo;Forward\u0026rdquo; and \u0026ldquo;Back\u0026rdquo; classes in forward/backward manipulation, whereas accuracy for the remaining classes was only around 60%. However, the weights obtained from this training were adopted for use in the system given that correct predictions were achieved for the majority of the data, and that \u0026ldquo;Forward,\u0026rdquo; which accounts for the majority of endoscope insertion maneuvers, had especially high accuracy.\u003c/p\u003e\u003cp\u003eBased on the above, a proprietary AI model for autonomous insertion was successfully developed using datasets generated for YOLOv5 and DenseNet-121.\u003c/p\u003e\n\u003ch3\u003eMechanism of operation using the AI model for autonomous insertion\u003c/h3\u003e\n\u003cp\u003eTwo personal computers (PCs) were used: one PC ran the program for EOR ver.4 (EOR ver.4 PC), and the other ran the program for the AI model (AI PC). These two PCs communicated across the network via inter-process communication using named pipes.\u003c/p\u003e\u003cp\u003eFirst, the EOR ver.4 PC sends a message to the AI PC to predict the appropriate operation. Upon receiving the message, the AI PC acquires the endoscopic image output from the flexible endoscope system and uses it to predict the appropriate endoscope operation. Finally, the AI PC sends the result of its prediction back to the EOR ver.4 PC, which then operates the EOR Ver.4 system accordingly based on the received instruction. The above process, including communication and AI prediction, was executed within 0.05 seconds, and by repeating this cycle every 0.05 seconds, autonomous operation was achieved. The forward speed was set to 6.5 mm/s, the backward speed to 4.0 mm/s, and the angulation speed was determined based on the prediction by the AI. For safety, the upper limit of force applied in the forward/backward direction was set to 20 N, and the torque limits for angulation in all directions were set to 0.5 N\u0026middot;m.\u003c/p\u003e\u003cp\u003eWith these specifications, ACRS was completed.\u003c/p\u003e\n\u003ch3\u003eStudy design and protocol\u003c/h3\u003e\n\u003cdiv class=\"Heading\"\u003eStudy design and protocol\u003c/div\u003e\u003cp\u003eAutomated insertion was performed using the ACRS. The entire automated insertion process was recorded along with time measurements using a four-panel video configuration of the endoscope screen, master unit, slave unit, and colonoscope training model (Fig.\u0026nbsp;\u003cspan refid=\"Fig8\" class=\"InternalRef\"\u003e4\u003c/span\u003e, Supplementary Video 2). Based on the Level of Autonomy classification for medical robots\u003csup\u003e\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e\u003c/sup\u003e, trials in which completely automated insertion was successfully achieved were designated Level 4. Trials that were completed but required manual intervention to pass through the location where the AI failed to make a decision were classified as Level 3.\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003eThe success rate for Level 4 insertions and the mean insertion time (minutes) were determined. In addition, the mean insertion time (minutes) for Level 4 trials was compared with that of manual insertions performed by the expert (used as teaching data) and trainees. The trainees had experience performing fewer than 200 colonoscopy procedures, and they performed five consecutive insertion trials after completing two practice sessions to familiarize themselves with the operation of the EOR ver.4 master unit.\u003c/p\u003e\u003cdiv id=\"Sec7\" class=\"Section2\"\u003e\u003ch2\u003eStatistical analysis\u003c/h2\u003e\u003cp\u003eData are expressed as mean\u0026thinsp;\u0026plusmn;\u0026thinsp;standard error (SE) values. Since the Level 4 data did not follow a normal distribution on the Kolmogorov\u0026ndash;Smirnov test, comparisons with other data were conducted using the Mann\u0026ndash;Whitney U test with the Bonferroni correction. A p-value of \u0026lt;\u0026thinsp;0.05 was considered significant. All analyses were performed using EZR (version 1.68) \u003csup\u003e\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e\u003c/div\u003e"},{"header":"Results","content":"\u003cp\u003eA total of 87 consecutive automated insertion trials were performed using the system described above. Of these, 15 trials were excluded because image interpretation was not possible due to gel-like lubricant adhering to the tip of the endoscope. All 72 remaining trials achieved Level 3 or higher. Level 4 was attained in 62 trials, with a mean insertion time of 2.92\u0026thinsp;\u0026plusmn;\u0026thinsp;1.20 minutes; Level 3 was attained in 10 trials, with a mean insertion time of 5.22\u0026thinsp;\u0026plusmn;\u0026thinsp;1.67 minutes. The Level 4 success rate was 86.1%.\u003c/p\u003e\u003cp\u003eManual insertion by one expert over 100 trials yielded a mean insertion time of 1.42\u0026thinsp;\u0026plusmn;\u0026thinsp;0.31 minutes. Manual insertion by six trainees over 30 trials yielded a mean insertion time of 2.94\u0026thinsp;\u0026plusmn;\u0026thinsp;1.34 minutes.\u003c/p\u003e\u003cp\u003eAlthough automated insertion at Level 4 took significantly longer than insertion by the expert, it was comparable to that of the trainees (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e\u003cp\u003e\u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e\u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e\u003cdiv class=\"CaptionContent\"\u003e\u003cp\u003eComparison of outcomes between AI model: Level 4 and manual insertion (experts and trainees).\u003c/p\u003e\u003c/div\u003e\u003c/caption\u003e\u003ccolgroup cols=\"3\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\"\u0026plusmn;\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e\u003cp\u003eTime (mean\u0026thinsp;\u0026plusmn;\u0026thinsp;SD), min\u003c/p\u003e\u003c/th\u003e\u003cth align=\"left\" colname=\"c3\"\u003e\u003cp\u003eP value vs AI model: Level 4\u003c/p\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eAI model: Level 4\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e\u003cp\u003e2.92\u0026thinsp;\u0026plusmn;\u0026thinsp;1.20\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e-\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eExperts\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e\u003cp\u003e1.42\u0026thinsp;\u0026plusmn;\u0026thinsp;0.31\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e\u0026lt;\u0026thinsp;0.001\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e\u003cp\u003eTrainees\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"char\" char=\"\u0026plusmn;\" colname=\"c2\"\u003e\u003cp\u003e2.94\u0026thinsp;\u0026plusmn;\u0026thinsp;1.34\u003c/p\u003e\u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e\u003cp\u003e1\u003c/p\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/colgroup\u003e\u003ctfoot\u003e\u003ctr\u003e\u003ctd colspan=\"3\"\u003eMin; minutes. Significant at \u0026lt;\u0026thinsp;0.05.\u003c/td\u003e\u003c/tr\u003e\u003c/tfoot\u003e\u003c/table\u003e\u003c/div\u003e\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eThe following two principles guided the development of this system. First, when pursuing AI-driven autonomy, it is essential that the current level of care in colonoscopy be maintained, and this is the ultimate goal of the system\u0026rsquo;s development. Second, although the main focus of medical AI is currently diagnostic imaging\u003csup\u003e\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e,\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e\u003c/sup\u003e, physicians remain the principal agents of action in technical or therapeutic procedures, and the focus of development has therefore been on supporting practitioners, such as through 3D reconstruction of surgical organs or navigational guidance within the surgical site\u003csup\u003e\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e,\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e\u003c/sup\u003e. In considering medical procedures or treatments currently performed by the physician as the main agent that could be handled by an AI-driven autonomous robotic system, we focused on total colonoscopy as a procedure that demands a high level of skill but operates within limited control axes and degrees of freedom, making it a technically achievable target for development.\u003c/p\u003e\u003cp\u003eCurrent colonoscopy practice relies on both the colonoscope itself, which is designed to enable highly precise manipulation, and the skill of the endoscopist who operates it\u003csup\u003e10,11\u003c/sup\u003e. The establishment of axis-keeping shortening (AKS) and other techniques, insertion in patients with a history of abdominal surgery or complex bowel anatomy\u003csup\u003e\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e\u003c/sup\u003e, high adenoma detection rates (ADRs) through detailed observation\u003csup\u003e\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e\u003c/sup\u003e, and procedures such as endoscopic submucosal dissection (ESD)\u003csup\u003e\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e,\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e\u003c/sup\u003e for early-stage colorectal cancer would not be achievable without both of these elements. An attempt to achieve autonomous colonoscope insertion using AI has been reported by Hwang et al\u003csup\u003e27\u003c/sup\u003e. However, the platform used to mount the colonoscope had a similar configuration to the first-generation EOR system. In other words, because it lacked the haptic feedback functionality essential for precise manipulation, this system was limited to simple training models for the bowel and could not be applied to more complex situations. A platform that provides haptic feedback across all aspects of colonoscope manipulation (insertion and retraction, rotation, and angulation of the scope tip in all directions) and enables comprehensive monitoring of both manipulation and corresponding image data is a requirement for AI-driven autonomous insertion. To meet this requirement, EOR ver.4 was developed. However, several challenges were identified in the results of these autonomous insertion trials. To achieve completely automated insertion under Insertion Pattern 1 of the colonoscope training model, teaching data were collected using EOR ver.4 operated by an expert. EOR ver.4 is equipped with a two-stage foot switch for air feed and lens irrigation, as well as a separate foot switch for suction control. However, in the Insertion Pattern 1 procedures performed by the expert, insertion was successfully completed without the need to operate either of these foot switches due to the expert\u0026rsquo;s skill. As a result, there were no recorded instances in which the colonoscope tip became obscured by gel-like lubricant and prevented image interpretation, and thus no corresponding annotations or training data were available for that situation. Furthermore, when the forward direction becomes unclear during standard insertion, the endoscope is typically retracted to broaden the field of view and reassess the lumen before determining the next direction of advancement. However, because the expert was able to complete insertion with virtually no backward movement, there were insufficient annotations and training data for this scenario; therefore, some trials remained at Level 3. In addition, once the lumen was identified, the expert quickly adjusted the angulation to capture the forward path with minimal manipulation, whereas the AI model continuously performed incremental angulation adjustments. Combined with the decision to maintain constant speeds for insertion and retraction based on the processing capacity of the PC and the trained AI model, this resulted in longer insertion times. Although the teaching data were based on insertions performed by an expert with the aim of achieving ideal automated insertion, this AI model performed at the level of trainees. Just as endoscopists become experts through the acquisition of the various elements of insertion techniques, it is necessary to advance the system step by step towards greater proficiency by gradually incorporating these elements into the teaching data, introducing new learning challenges for the AI model to overcome, and, where necessary, integrating AI models other than YOLO or DenseNet and deploying higher-performance PCs.\u003c/p\u003e\u003cp\u003eIn conclusion, ACRS demonstrated a high success rate for fully automated colonoscope insertion. Although insertion times were at the level of trainees, the results suggest the potential for clinical application of completely automated colonoscope insertion.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDevelopment of EOR ver.4 and ACRS\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAfter an overview of the specifications of EOR ver.4, the AI model developed for autonomous insertion and the mechanism by which autonomous operation is achieved are described.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eDevelopment History and Specifications of EOR ver.4\u003c/em\u003e\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe EOR system has evolved progressively from version 1\u003csup\u003e12\u003c/sup\u003e, which was controlled via a joystick. In version 3, a proprietary master unit with haptic feedback was introduced, enabling intuitive one-handed insertion, retraction, and rotation operations while providing force sensation\u003csup\u003e13,14\u003c/sup\u003e. Building on this, version 4 incorporated haptic feedback into all operations, including vertical and horizontal angulation of the scope tip (Fig. 2, Supplementary VIDEO 2). EOR ver.4 is a system capable of complete monitoring of 16 parameters, including force sensation, angle, speed, and scope insertion length for all operations, recorded at a minimum resolution of 1/1000 second, together with synchronized endoscopic video image data recorded in high definition, and both data streams can be annotated. In this study, a single expert endoscopist used EOR ver.4 to perform total colonoscope insertion using Insertion Pattern 1 of a colonoscope training model (Kyoto Kagaku Co., Ltd.), generating a teaching dataset from 100 insertions recorded at 1/100-second intervals. Of these, 12 were used to develop AI models. The expert endoscopist had experience with over 10,000 colonoscopy procedures.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eDevelopment of an AI Model for Autonomous Insertion and Its Mechanism of Operation\u003c/em\u003e\u003c/strong\u003e\u003c/p\u003e\n\u003col\u003e\n \u003cli\u003eSelection and training of the AI models used to construct a proprietary AI model for autonomous insertion\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eTo construct a proprietary AI model capable of autonomous insertion, the existing AI models YOLOv5\u003csup\u003e25\u003c/sup\u003e and DenseNet-121 were used\u003csup\u003e26\u003c/sup\u003e. The former is an object detection model, which was used to predict the direction of angulation. The latter is a convolutional neural network (CNN) model for image recognition, which was used to predict the state of advancement (forward), retraction (backward), and angulation speed.\u003c/p\u003e\n\u003cp\u003eThe training dataset for YOLOv5 was created by annotating images extracted from the endoscopic video image data in the teaching dataset. Specifically, annotations were made by enclosing the target regions towards which the endoscope was to be angled within the endoscopic images. These enclosed regions are referred to as bounding boxes, and they are categorized into seven classes according to state of advancement on the image: \u0026ldquo;Hole,\u0026rdquo; \u0026ldquo;Gap,\u0026rdquo; \u0026ldquo;Wall,\u0026rdquo; \u0026ldquo;Fold (Right),\u0026rdquo; \u0026ldquo;Fold (Left),\u0026rdquo; \u0026ldquo;Fold (Up),\u0026rdquo; and \u0026ldquo;Fold (Down).\u0026rdquo; \u0026ldquo;Hole\u0026rdquo; refers to regions in which the colonic lumen is visible. \u0026ldquo;Gap\u0026rdquo; refers to regions in which the elongated, flattened lumen is visible. \u0026ldquo;Wall\u0026rdquo; refers to regions in which the intestinal wall is visible throughout the entire endoscopic image. For these three classes, the angulation direction was determined using the coordinates at the center of the bounding box. Specifically, the coordinates from the center of the image to the center of the bounding box\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003edefined the direction of angulation (Fig. 2A). \u0026ldquo;Fold\u0026rdquo; refers primarily to regions in which colonic folds such as those in the sigmoid colon are prominently visible. In the sigmoid colon, folds overlap, resulting in complex structures. For this reason, images of colonic folds such as those in the sigmoid colon were annotated as a separate class from other image types. Folds were categorized into four classes based on the direction from which they projected: \u0026ldquo;Right,\u0026rdquo; \u0026ldquo;Left,\u0026rdquo; \u0026ldquo;Up,\u0026rdquo; and \u0026ldquo;Down.\u0026rdquo; For example, in the case of a fold projecting from the right (Fold [Right]), the direction of angulation was determined using the coordinates at the center of the left edge of the bounding box. Specifically, the coordinates from the center of the image to the center of the bounding box edge defined the direction of angulation (Fig. 2B). For these four classes, the angulation direction was determined using the center of the bounding box edge opposite the direction in which the fold projects.\u003c/p\u003e\n\u003cp\u003eThe training dataset for DenseNet-121 was obtained using endoscopic images from the teaching dataset and their corresponding manipulation data. Five types of manipulation data were used: insertion length of the endoscope [mm]; torque applied during left\u0026ndash;right angulation [N\u0026middot;m]; torque applied during up\u0026ndash;down angulation [N\u0026middot;m]; torque applied during scope rotation [N\u0026middot;m]; and reactive force in the insertion direction during advancement [N]. For each paired set of images and manipulation data, the scope manipulation speed was calculated, and a one-hot vector label corresponding to the speed classification was assigned, thereby generating the final training dataset. For forward and backward manipulation, speeds were labelled in three categories: \u0026ldquo;Forward\u0026rdquo; for 15 mm/s and above; \u0026ldquo;Back\u0026rdquo; for \u0026ndash;15 mm/s and below; and \u0026ldquo;Stop\u0026rdquo; for all other speeds. For angulation, speeds were labelled in three categories: \u0026ldquo;Slow\u0026rdquo; for \u0026le;25 mm/s; \u0026ldquo;Normal\u0026rdquo; for 25 mm/s to \u0026lt;62.5 mm/s; and \u0026ldquo;Fast\u0026rdquo; for \u0026ge;62.5 mm/s.\u003c/p\u003e\n\u003cp\u003eTraining for the above two models was primarily conducted using PyTorch, a machine learning library for Python.\u003c/p\u003e\n\u003cp\u003eFor YOLOv5, the training dataset consisted of 2,100 images, the test dataset consisted of 470 images, and training was performed over 1,000 epochs with a batch size of 16. The confusion matrix resulting from this training process is shown in Fig. 4A. The classification accuracy exceeded 95%, except for the Fold class, where the accuracy was 74% to 96%. It was considered that the situation in the direction of scope advancement could be predected using the model weights obtained from the results of this training. Misclassifications were more frequent in the Fold class than in other classes, particularly between the Fold subclasses, but it was decided to adopt the weights obtained from this training in light of the fact that even manual insertions are conducted while continuously correcting decisions.\u003c/p\u003e\n\u003cp\u003eThe training data for DenseNet-121 included 1,745 training images and 300 test images for forward/backward manipulation, together with 1,443 training images and 187 test images for angulation. To ensure a sufficient number of training images, several image transformations were randomly combined and applied to the model input data. (The optimization method used was Adam\u003csup\u003e27\u003c/sup\u003e, with a learning rate set to 0.0005. Mean squared error was used as the loss function for both forward/backward manipulation and angulation. After calculating the individual losses, a weighted sum was calculated for each batch, with the loss for forward/backward manipulation weighted at 0.7, and the loss for angulation weighted at 0.3.) Training was conducted over 5,000 epochs with a batch size of 128. The resulting confusion matrices for forward/backward manipulation and angulation are shown in Fig. 4B and Fig. 4C, respectively. The average per-batch classification accuracy was 95.3% for forward/backward manipulation and 66.9% for angulation. An analysis of the classification accuracy for each class showed particularly high accuracy for the \u0026ldquo;Forward\u0026rdquo; and \u0026ldquo;Back\u0026rdquo; classes in forward/backward manipulation, whereas accuracy for the remaining classes was only around 60%. However, the weights obtained from this training were adopted for use in the system given that correct predictions were achieved for the majority of the data, and that \u0026ldquo;Forward,\u0026rdquo; which accounts for the majority of endoscope insertion maneuvers, had especially high accuracy.\u003c/p\u003e\n\u003cp\u003eBased on the above, a proprietary AI model for autonomous insertion was successfully developed using datasets generated for YOLOv5 and DenseNet-121.\u003c/p\u003e\n\u003col start=\"2\"\u003e\n \u003cli\u003eMechanism of Operation using the AI Model for Autonomous Insertion\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eTwo personal computers (PCs) were used: one PC ran the program for EOR ver.4 (EOR ver.4 PC), and the other ran the program for the AI model (AI PC). These two PCs communicated across the network via inter-process communication using named pipes.\u003c/p\u003e\n\u003cp\u003eFirst, the EOR ver.4 PC sends a message to the AI PC to predict the appropriate operation. Upon receiving the message, the AI PC acquires the endoscopic image output from the flexible endoscope system and uses it to predict the appropriate endoscope operation. Finally, the AI PC sends the result of its prediction back to the EOR ver.4 PC, which then operates the EOR Ver.4 system accordingly based on the received instruction. The above process, including communication and AI prediction, was executed within 0.05 seconds, and by repeating this cycle every 0.05 seconds, autonomous operation was achieved. The forward speed was set to 6.5 mm/s, the backward speed to 4.0 mm/s, and the angulation speed was determined based on the prediction by the AI. For safety, the upper limit of force applied in the forward/backward direction was set to 20 N, and the torque limits for angulation in all directions were set to 0.5 N\u0026middot;m.\u003c/p\u003e\n\u003cp\u003eWith these specifications, ACRS was completed.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eCompeting Interests\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eNone\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAcknowledgments\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis research was supported by AMED under Grant Number JP18lm0203009.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthor Contributions\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eK. KUME.: Planning and development of EOR ver.4 and the AI model; overall planning and supervision of the research; acquisition of training data and manual data; statistical analysis; drafting of the manuscript. R. TAIRA: Central role in development and explanation of the AI model; trial insertions using the AI model. S. TERAO: Development of the AI model; trial insertions using the AI model. N. SAKAI: Development of EOR ver.4; development and supervision of the AI model; trial insertions using the AI model; technical oversight.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eSung, H. et al. Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. \u003cem\u003eCA Cancer J. Clin.\u003c/em\u003e 71, 209-249 (2021).\u003c/li\u003e\n\u003cli\u003eJoseph, D. A. et al. Colorectal cancer screening: Estimated future colonoscopy need and current volume and capacity. \u003cem\u003eCancer\u003c/em\u003e 122, 2479-2486 (2016). \u003c/li\u003e\n\u003cli\u003eBretthauer, M. et al. Effect of colonoscopy screening on risks of colorectal cancer and related death. \u003cem\u003eN. Engl. J. Med.\u003c/em\u003e 387, 1547-1556 (2022).\u003c/li\u003e\n\u003cli\u003eSpier, B. J. et al. Colonoscopy training in gastroenterology fellowships: Determining competence. \u003cem\u003eGastrointest. Endosc.\u003c/em\u003e 71, 319-324 (2010). \u003c/li\u003e\n\u003cli\u003eLee, S.-H., Park, Y.-K., Lee, D.-J. \u0026amp; Kim, K.-M. Colonoscopy procedural skills and training for new beginners. \u003cem\u003eWorld J. Gastroenterol.\u003c/em\u003e 20, 16984-16995 (2014).\u003c/li\u003e\n\u003cli\u003eWinters, C., Subramanian, V. \u0026amp; Valdastri, P. Robotic, self-propelled, self-steerable, and disposable colonoscopes: Reality or pipe dream? A state of the art review. \u003cem\u003eWorld J. Gastroenterol.\u003c/em\u003e 28, 5093-5110 (2022).\u003c/li\u003e\n\u003cli\u003eCiuti, G. et al. Frontiers of robotic colonoscopy: A comprehensive review of robotic colonoscopes and technologies. \u003cem\u003eJ. Clin. Med.\u003c/em\u003e 9, 1648 (2020).\u003c/li\u003e\n\u003cli\u003eVucelic, B. et al. The aer-o-scope: Proof of concept of a pneumatic, skill-independent, self-propelling, self-navigating colonoscope. \u003cem\u003eGastroenterology\u003c/em\u003e 130, 672-677 (2006).\u003c/li\u003e\n\u003cli\u003eSlawinski, P. R. et al. Autonomously controlled magnetic flexible endoscope for colon exploration. \u003cem\u003eGastroenterology\u003c/em\u003e 154, 1577-1579.e1 (2018).\u003c/li\u003e\n\u003cli\u003eMazurek, M. et al. Association between endoscopist specialty and colonoscopy quality: A systematic review and meta-analysis. \u003cem\u003eClin. Gastroenterol. Hepatol.\u003c/em\u003e 20, 1931-1946 (2022).\u003c/li\u003e\n\u003cli\u003eHsu, W. F. \u0026amp; Chiu, H. M. Optimization of colonoscopy quality: Comprehensive review of the literature and future perspectives. \u003cem\u003eDig. Endosc.\u003c/em\u003e 35, 822-834 (2023).\u003c/li\u003e\n\u003cli\u003eKume, K., Kuroki, T., Sugihara, T. \u0026amp; Shinngai, M. Development of a novel endoscopic manipulation system: The endoscopic operation robot. \u003cem\u003eWorld J. Gastrointest.\u003c/em\u003e\u003cem\u003e \u003c/em\u003e\u003cem\u003eEndosc.\u003c/em\u003e 3, 145-150 (2011).\u003c/li\u003e\n\u003cli\u003eKume, K., Sakai, N. \u0026amp; Goto, T. Development of a novel endoscopic manipulation system: The endoscopic operation robot ver.3. \u003cem\u003eEndoscopy\u003c/em\u003e 47, 815-819 (2015).\u003c/li\u003e\n\u003cli\u003eKume, K., Sakai, N. \u0026amp; Goto, T. Haptic feedback is useful in remote manipulation of flexible endoscopes. \u003cem\u003eEndosc. Int. Open\u003c/em\u003e 6, E1134-E1139 (2018).\u003c/li\u003e\n\u003cli\u003eYang, G. Z. et al. Medical robotics - Regulatory, ethical, and legal considerations for increasing levels of autonomy. \u003cem\u003eSci. Robot.\u003c/em\u003e 2, eaam8638 (2017).\u003c/li\u003e\n\u003cli\u003eKanda, Y. Investigation of the freely available easy-to-use software \u0026ldquo;EZR\u0026rdquo; for medical statistics. \u003cem\u003eBone Marrow Transplant.\u003c/em\u003e 48, 452-458 (2013).\u003c/li\u003e\n\u003cli\u003eZhou, S. K. et al. A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises. \u003cem\u003eProc. IEEE Inst. Electr. Electron. Eng.\u003c/em\u003e 109, 820-838 (2021)..\u003c/li\u003e\n\u003cli\u003eOkagawa, Y., Abe, S., Yamada, M., Oda, I. \u0026amp; Saito, Y. Artificial intelligence in endoscopy. \u003cem\u003eDig. Dis. Sci.\u003c/em\u003e 67, 1553-1572 (2022).\u003c/li\u003e\n\u003cli\u003eMoglia, A., Georgiou, K., Georgiou, E., Satava, R. M. \u0026amp; Cuschieri, A. A systematic review on artificial intelligence in robot-assisted surgery. \u003cem\u003eInt. J. Surg.\u003c/em\u003e 95, 106151 (2021)\u003c/li\u003e\n\u003cli\u003eG\u0026ouml;bel, B., Reiterer, A. \u0026amp; M\u0026ouml;ller, K. Image-based 3D reconstruction in laparoscopy: A review focusing on the quantitative evaluation by applying the reconstruction error. \u003cem\u003eJ. Imaging\u003c/em\u003e \u003cstrong\u003e10\u003c/strong\u003e, 180 (2024).\u003c/li\u003e\n\u003cli\u003eCorley, D. A. et al. Adenoma detection rate and risk of colorectal cancer and death. \u003cem\u003eN. Engl. J. Med.\u003c/em\u003e 370, 1298-1306 (2014).\u003c/li\u003e\n\u003cli\u003eOhata, K. et al. Long-term outcomes after endoscopic submucosal dissection for large colorectal epithelial neoplasms: A prospective, multicenter, cohort trial from Japan. \u003cem\u003eGastroenterology\u003c/em\u003e 163, 1423-1434.e2 (2022).\u003c/li\u003e\n\u003cli\u003eKume, K. Endoscopic therapy for early gastric cancer: Standard techniques and recent advances in ESD. \u003cem\u003eWorld J. Gastroenterol.\u003c/em\u003e 20, 6425-6432 (2014).\u003c/li\u003e\n\u003cli\u003eHwang, B., Changbeom, S., Seonggun, J. et al. A preliminary study on autonomous robotic colonoscopy via deep neural network. \u003cem\u003e13th International Conference on Control, Automation and Information Sciences\u003c/em\u003e\u003cem\u003e(ICCAIS)\u003c/em\u003e (2024).\u003c/li\u003e\n\u003cli\u003eRedmon, J., Divvala, S., Girshick, R. \u0026amp; Farhadi, A. You only look once: Unified, real-time object detection. In \u003cem\u003eproceedings of the IEEE conference on computer vision and pattern recognition\u003c/em\u003e 779-788 (2016).\u003c/li\u003e\n\u003cli\u003eHuang, G., Liu, Z., Maaten, L. \u0026amp; Weinberger, K. Q. Densely connected convolutional networks. In \u003cem\u003eProceedings of the IEEE conference on computer vision and pattern recognition\u003c/em\u003e 4700-4708 (2017).\u003c/li\u003e\n\u003cli\u003eKingma, D. P. \u0026amp; Ba, J. Adam: A method for stochastic optimization. ArXiv:1412.6980 (2014).\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"nature-portfolio","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"","title":"Nature Portfolio","twitterHandle":"","acdcEnabled":false,"dfaEnabled":false,"editorialSystem":"ejp","reportingPortfolio":"","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-7769716/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7769716/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eBackground\u003c/h2\u003e\u003cp\u003eIn the field of colonoscopy, robotic systems have been developed to support or replace human operators due to a shortage of trained endoscopists. A master-slave robotic system, the Endoscopic Operation Robot version 4 (EOR ver.4), in which a colonoscope is mounted on the slave unit, and tactile manipulation is enabled via the master unit, was developed. Using teaching data obtained from full colonoscope insertions performed on a colonoscopy training model by expert endoscopists, an artificial intelligence (AI) model was constructed to achieve autonomous insertion. Using this model, EOR ver.4 was developed as an autonomous system, the Autonomous Colonoscope Robot System (ACRS), and its ability to perform completely automated insertion was evaluated.\u003c/p\u003e\u003ch2\u003eMethods\u003c/h2\u003e\u003cp\u003eACRS performed fully automated insertions during which insertion images and insertion times were recorded. Completely automated insertions were designated Level 4, whereas insertions requiring some manual assistance were designated Level 3. Separately from the expert who provided the training data, insertion images and times in manual insertions performed by trainees were also recorded, and they were compared with completely automated insertions (Level 4). A colonoscopy training model was used for these insertions.\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e\u003cp\u003eOf the 72 automated insertions at Level 3 or higher, 62 were classified as Level 4, giving a success rate of 86.1%. The average insertion time for Level 4 procedures was 2.92\u0026thinsp;\u0026plusmn;\u0026thinsp;1.20 minutes, significantly longer than that of the expert (1.42\u0026thinsp;\u0026plusmn;\u0026thinsp;0.31 minutes), but comparable to the time taken by trainees (2.94\u0026thinsp;\u0026plusmn;\u0026thinsp;1.34 minutes).\u003c/p\u003e\u003ch2\u003eConclusion\u003c/h2\u003e\u003cp\u003eACRS demonstrated a high success rate for fully automated colonoscope insertion.\u003c/p\u003e","manuscriptTitle":"Development of an Autonomous Robot for Fully Automated Colonoscope Insertion: The Autonomous Colonoscope Robot System (ACRS)","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-11-24 16:44:36","doi":"10.21203/rs.3.rs-7769716/v1","editorialEvents":[],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"communications-medicine","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"commsmed","sideBox":"Learn more about [Communications Medicine](http://www.nature.com/commsmed)","snPcode":"43856","submissionUrl":"https://mts-commsmed.nature.com/cgi-bin/main.plex","title":"Communications Medicine","twitterHandle":"@commsmedicine","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"ejp","reportingPortfolio":"Communications Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"a0898939-5215-4d3a-b5da-9701e35d8123","owner":[],"postedDate":"November 24th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[{"id":56258017,"name":"Health sciences/Gastroenterology/Colonoscopy"},{"id":56258018,"name":"Physical sciences/Engineering/Mechanical engineering"}],"tags":[],"updatedAt":"2025-11-24T16:44:36+00:00","versionOfRecord":[],"versionCreatedAt":"2025-11-24 16:44:36","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7769716","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7769716","identity":"rs-7769716","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}