Intraoperative Patient Specific Volumetric Reconstruction and 3D Visualization for Laparoscopic Liver Surgery

Despite the benefits of minimally invasive surgery, interventions such as laparoscopic liver surgery present unique challenges, like the significant anatomical differences between preoperative images and intraoperative scenes due to pneumoperitoneum, patient pose, and organ manipulation by surgical instruments. To address these challenges, we propose a method for intraoperative 3D reconstruction of the surgical scene, including vessels and tumors, without altering the surgical workflow. The technique combines Neural Radiance Field (NeRF) reconstructions from tracked laparoscopic videos with ultrasound 3D compounding. We evaluate the accuracy of our reconstructions on a clinical laparoscopic liver ablation dataset, consisting of laparoscope and patient reference poses from optical tracking, laparoscopic and ultrasound videos, as well as preoperative and intraoperative CTs. We propose a solution to compensate for liver deformations due to pressure applied during ultrasound acquisitions, improving the overall accuracy of the 3D reconstructions compared to the ground truth intraoperative CT with pneumoperitoneum. We train a unified NeRF from the ultrasound and laparoscope data, which allows real-time view synthesis providing surgeons with comprehensive intraoperative visual information for laparoscopic liver surgery.