Knowledge-Guided Surgery: Augmented Reality based Registration and Projection for Anatomical Tissues and Preoperative Planned Approaches

To address the challenges of providing real-time guidance during surgical procedures and fully utilizing preoperative lesion data information intraoperatively, we propose an augmented reality-based, knowledge-guided method for surgical procedures. Our method involves designing a semantic segmentation algorithm for medical images to achieve accurate lesion tissues segmentation. With the segmentation results, we localize the tissues and organs, plan the surgical approaches based on safety constraints, and generate a virtual knowledge model containing semantic information of the lesion and the preoperative planned approaches through 3D visualization operations. To align the virtual surgical knowledge model with the patient's lesion site in the real surgical environment, we establish a portable high-precision augmented reality display system with external depth cameras. We use joint calibration, point cloud alignment, and wireless communication transmission to improve the augmented reality device's scene perception capability and achieve the precise integration of virtual and real objects in holographic projection. Our proposed method achieves the goal of knowledge-guided surgery process. This paper presents the personalized 3D representation of medical images and demonstrates the precise application of augmented reality technology in surgery, which provides solutions to the problems of surgery planning and navigation.