An Intelligent Approach for Automating Robotic Arm Maneuvering in Endometriosis Surgery

This section outlines the proposed method for automating the maneuvering of robotic arms within the patient’s body in an endometriosis surgery. A decision-tree model is introduced, designed specifically to guide robotic systems step-by-step through typical endometriosis surgery procedures. The proposed tree comprises decision-making nodes, where data from the camera and other metadata inputs are analyzed to determine appropriate actions, such as proceeding with the surgery or halting it in response to an identified error. Additionally, the tree includes action nodes that dictate specific tasks, such as robotic arm and camera movements. Furthermore, advanced ensemble neural network frameworks have been developed and integrated into the decision-making nodes to enhance the system’s analytical and operational capabilities. In the first step of the proposed decision-tree, the system evaluates whether the camera captures a high-quality image. In certain cases, visual obstructions, such as blood particles or water vapor, can severely impair the robot's vision by introducing noise into the images. Such noisy images can lead to significant deviations from the robot’s intended operational strategy. Therefore, as a preliminary measure, the robot must assess the quality of the visual data received from the camera. If the image is excessively noisy, the system should notify clinical staff to address the issue. Alternatively, the robot can be equipped with mechanisms to autonomously clean the camera’s lens, thereby reducing reliance on clinical personnel and facilitating the surgical process. In some patients, adhesions may be present, posing obstacles during surgery. To address this, surgeons typically use a CO2 laser to vaporize the adhesions, thereby clearing the surgical field and facilitating the operation. Accordingly, in the second step of the proposed decision-tree, following confirmation that the images are free of noise, the robot must determine whether adhesions are present within the patient’s body. If adhesions are detected, the robot should employ the CO2 laser instrument to eliminate them and streamline the surgical process. Conversely, if no adhesions are found, the robot should proceed to the third step, wherein it determines whether to examine gynecological positions (the uterus and ovaries) or the peritoneum. In the third step of the proposed method, the robot confirms that the visual data is noise-free and that no adhesions are present within the body. Following this, the robot proceeds to survey the ovaries, uterus, or peritoneum to identify potential ovarian, uterine, or peritoneal endometriosis lesions. To achieve this, the robotic camera must rotate toward the upper region (the peritoneum) or the uterus. After performing this rotation, the system verifies whether the peritoneum or uterus is visible through the camera in order to certify the rotation. This process mimics the cognitive approach of laparoscopic surgeons, who rotate the laparoscope and observe the monitor to determine whether the peritoneum or uterus is within the field of view. However, there is a minimal risk of rotation failure during this step. If such an issue arises, the system should alert nurses and clinical experts to intervene and resolve the problem. While the probability of rotation failure is minimal, it must still be considered, given the critical importance of achieving fully autonomous surgery. Addressing even low-probability risks ensures maximum reliability of the system. If the peritoneum is selected for investigation, the robot must systematically survey its various regions to identify potential endometriosis lesions. Upon detecting such lesions, the robot can utilize a CO2 laser to effectively vaporize them. Using a CO2 laser is considered effective for eliminating pathological tissue and offers distinct advantages over traditional surgical approaches [8]. Alternatively, if the uterus or ovaries are chosen for examination, the robotic arms must advance toward the uterus. At this stage, the robot is required to detect, semantically segment, and localize the uterus. To achieve this, a uterus semantic segmentation framework is employed. This framework not only enables the robot to maneuver its arms with precision but also assists in calculating the proximity of the robotic arms and camera to the uterus. Specifically, a larger segmented surface area of the uterus indicates a closer proximity of the camera to the uterus. To ensure safe movement of the camera toward the uterus, the distance between the uterus and the camera must be accurately calculated. This calculation is crucial for guiding the camera forward while preventing any potential collision with the uterus or surrounding tissues. After reaching the uterus, it must be held and stabilized to allow for meticulous and careful examination. We propose the use of suction instruments instead of traditional grasping tools, as suction instruments not only simplify surgical procedures but also minimize potential damage to the uterine tissue. To achieve this, the uterus semantic segmentation framework is essential. This framework facilitates the process of stabilizing the uterus for the investigation of uterine endometriosis and assists in positioning and holding the uterus to access the ovaries for the identification of ovarian endometriosis. For cases of ovarian endometriosis, a similar strategy should be employed following the stabilization of the ovaries using either grasping instruments or suction instruments. Subsequently, the ovary must be carefully detected and localized. To achieve this, an ovary semantic segmentation framework is required, enabling the simultaneous detection and localization of both the ovary and uterus for the purpose of identifying ovarian endometriosis. Finally, upon accurately detecting, localizing, and segmenting endometriosis lesions—whether on the peritoneum, uterus, or ovaries—the robot can utilize its CO2 laser to vaporize the pathological tissue. The proposed decision-tree model is illustrated in Fig. 1. For the endometriosis lesions detection and localization, the ensemble framework introduced in [8] can be used. In the proposed decision-tree framework, four decision-making nodes are designed to address noise detection, adhesion detection, peritoneum or uterus classification, and ovary detection, respectively. For each decision-making node, an advanced ensemble neural network framework has been developed to perform binary classification. Additionally, two semantic segmentation and localization frameworks are implemented to identify the uterus and ovaries during surgery, which are essential for determining organ positions and enabling appropriate surgical actions. The ensemble frameworks enhance the reliability of the proposed system by integrating multiple AI models, thereby yielding more precise decisions. This approach can be compared to a scenario where a panel of experts deliberates on a problem, resulting in more reliable and trustworthy outcomes. Furthermore, as the neural networks within the ensemble framework operate in parallel, there is no significant impediment to the overall execution speed. The structures of the proposed ensemble neural network classifiers are designed with an initial layer that extracts visual features from the input image. This layer employs various architectures, such as ResNet and VGG, among others. The extracted features are then passed to a fully connected layer responsible for determining the class to which the input image belongs. Based on this classification, the network generates a vote. Simultaneously, additional neural networks employing different architectures process the input image, each producing its own votes. Finally, a collective decision is made by aggregating the votes from the individual networks. This final decision guides the robot through the decision-making nodes in the proposed decision-tree model. The structure of the ensemble neural network classifiers, incorporating diverse architectures as feature extractors, is illustrated in Fig. 2. The architecture of the proposed ensemble neural network classifier, which is designed to determine whether the input image is noisy or of high quality, is illustrated in Fig. 2.a. In each neural network, the image or frame from the real-time video is initially processed by a feature extractor layer. This layer can be implemented using various CNN architectures, such as ResNet, VGG, and others. The extracted visual features are subsequently passed to a fully connected layer (FCL), which determines the category to which the input image belongs. This methodology is uniformly applied across all proposed ensemble classification frameworks. However, the frameworks differ in the number of feature extractor layers and the specific architectures utilized. To design the ensemble neural network frameworks, a systematic approach was employed. First, various architectures were examined for each classification task. Subsequently, the outputs of all neural networks for the same test set of images were recorded. Finally, a search algorithm was utilized to evaluate all possible designs for the ensemble framework. The ensemble with the highest accuracy was ultimately selected for inclusion in the proposed system. In cases where multiple ensemble frameworks achieved the same accuracy, the framework with a greater number of networks was chosen to enhance the system's reliability. Since the output of the ensemble framework is generated by a larger number of advanced networks, the results are anticipated to be more robust and dependable. Our findings demonstrate that the ensemble neural network framework, incorporating feature extractor architectures such as ResNet50, ResNet50V2, VGG16, VGG19, ResNet101, ResNet101V2, InceptionResNetV2, DenseNet169, DenseNet201, and NASNetLarge, achieved the highest reliability for the first decision-making node of the proposed tree. This node is responsible for classifying noisy images from high-quality ones. In this design, the threshold value, set to 2, determines that if the number of votes within an ensemble framework meets or exceeds this value, the classifier produces a positive output. For the first ensemble framework, a positive output corresponds to the noisy image class. The architecture of this framework is depicted in Fig. 2.a. Similarly, the proposed ensemble framework for the second decision-making node in the decision tree comprises 14 independent neural networks employing various feature extractor architectures, including InceptionV3, MobileNet, ResNet50, ResNet50V2, VGG16, VGG19, ResNet101, ResNet101V2, InceptionResNetV2, DenseNet121, DenseNet169, DenseNet201, NASNetLarge, and ConvNeXtXLarge. The threshold value for the second ensemble framework is set to 10. This framework is designed to classify images containing adhesions from normal images, with its architecture illustrated in Fig. 2.b. For the third decision-making node in the proposed tree, an ensemble framework was developed to classify whether the input image contains the uterus or peritoneum. Our findings indicate that the ensemble framework incorporating feature extractor layers with architectures such as InceptionV3, ResNet50, ResNet50V2, VGG16, VGG19, ResNet101, ResNet101V2, InceptionResNetV2, DenseNet169, DenseNet201, MobileNet, and NASNetLarge demonstrates sufficient reliability. The threshold value for this ensemble framework is set to 1. The architecture of this framework is illustrated in Fig. 2.c. Similarly, for the fourth decision-making node in the proposed tree, the ensemble classifier utilizes InceptionV3, ResNet50, ResNet50V2, ResNet101, ResNet101V2, InceptionResNetV2, VGG16, VGG19, MobileNet, DenseNet121, DenseNet169, ConvNeXtXLarge, and NASNetLarge as the feature extractor architecture. The threshold value for this framework is set to 10. The architecture of the proposed ensemble framework for the fourth decision-making node is depicted in Fig. 2.d. In the proposed decision tree framework, semantic segmentation and localization tasks are handled using two ensemble frameworks—one dedicated to uterus detection and the other to ovary detection. Each ensemble framework integrates three U-Net models, employing ResNet50, ResNet101, and VGG19 architectures as backbone networks. Within the decision tree, the segmentation and localization frameworks for the uterus and ovary are executed following classification at the third and fourth decision-making nodes. While ovary detection can be performed without a classification framework, this approach has been designed to minimize noise-related risks, thereby improving the overall reliability of the system. The architectures of the proposed semantic segmentation frameworks for uterus and ovary detection are depicted in Figs. 3.a and 3.b, respectively. In ensemble frameworks for classification, each neural network generates a binary decision that identifies the category to which the input image belongs. Similarly, in semantic segmentation frameworks, each neural network produces binary decisions for individual pixels within the input image. To achieve accurate semantic segmentation and localization of the uterus and ovary during robotic and laparoscopic surgery, the votes from all neural networks are aggregated for each pixel. As detailed in Section 4, our findings reveal that the proposed neural network frameworks for both classification and semantic segmentation tasks exhibit robustness and reliability. Moreover, the decision tree, developed based on medical methodologies for endometriosis surgery and integrated with advanced deep learning frameworks, enhances system reliability. The interpretability of the decision tree further supports its adaptability and ease of maintenance for future applications. By presenting a dependable approach for maneuvering robotic arms within a patient's body during surgery, the proposed method significantly improves the efficiency and precision of surgeries. When combined with advanced detection techniques, as referenced in [8], this approach enables fully automated surgical procedures powered by reliable AI systems. This innovation not only enhances surgical accuracy and reliability but also reduces the risks of post-operative infections, decreases economic costs, and mitigates intraoperative bleeding.