High-Fidelity Pose Estimation for Real-Time Extended Reality (XR) Visualization for Cardiac Catheterization

High-Fidelity Pose Estimation for Real-Time Extended Reality (XR) Visualization for Cardiac Catheterization | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article High-Fidelity Pose Estimation for Real-Time Extended Reality (XR) Visualization for Cardiac Catheterization Bobak Mosadegh, Mohsen Annabestani, Sandhya Sriram, Alexandre Caprio, and 3 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4645065/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 06 Nov, 2024 Read the published version in Scientific Reports → Version 1 posted 10 You are reading this latest preprint version Abstract Extended reality (XR) technologies are emerging as promising platforms for medical training and procedural guidance, particularly in complex cardiac interventions. This paper presents a high-fidelity methodology to perform real-time 3D catheter tracking and visualization during simulated cardiac interventions. A custom 3D-printed setup with mounted cameras enables biplane video capture of a catheter. A computer vision algorithm processes the biplane images in real-time to reconstruct the 3D catheter trajectory represented by any designated number of points along its length. This method accurately localizes the catheter tip within 1 mm and can reconstruct any arbitrary catheter configuration. The tracked catheter data is integrated into an interactive Unity-based scene rendered on the Meta Quest 3 headset. The visualization seamlessly combines a reconstructed 3D patient-specific heart model with the dynamically tracked catheter, creating an immersive extended reality training environment. Our experimental study, involving six participants, demonstrated that the 3D visualization provided by the proposed XR system significantly outperformed 2D visualization in terms of speed and user experience. This suggests that the XR system has the potential to enhance catheterization training by improving spatial comprehension and procedural skills. The proposed system demonstrates the potential of XR technologies to transform percutaneous cardiac interventions through improved visualization and interactivity. Physical sciences/Engineering/Biomedical engineering Health sciences/Cardiology/Interventional cardiology Percutaneous cardiac intervention Extended Reality Mixed Reality Computer Vision 3D visualization Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 Figure 8 Figure 9 Figure 10 Figure 11 1. Introduction Percutaneous cardiac procedures demand precision in manipulating catheters through blood vessels and heart chambers for access in many cardiovascular conditions such as coronary artery disease, valvular heart disease, and structural heart defects 1–6 . The current methods, relying on fluoroscopy and echocardiography imaging, fall short in providing a comprehensive view of the catheter's dynamic position and orientation within the patient, necessitating significant mental reconstruction of the anatomical context 7,8 . As the field of complex catheter-based structural heart interventions advances, there is a pressing need for systems that offer enhanced visualization and guidance. This paper delves into the technical challenges required to provide real-time catheter tracking to an extended reality (XR) system with high speed and accuracy. Such capabilities will have applications for various use in percutaneous transcatheter cardiac interventions training 9 , pre-procedural planning, intra-operative guidance 10 , and Telesurgery to enable physicians to remotely maneuver catheters 11 . Conventional training techniques, anchored in fluoroscopy-guided observations and hands-on practice, pose challenges such as high costs, radiation exposure, and limited spatial comprehension for trainees 12–14 . The limitation of 2D visualization in representing the catheter's 3D location prompts the exploration of innovative solutions. XR, by combining 3D visualizations with real procedural tools, such as the actual catheter used in clinical practice, offers an unparalleled advantage in providing trainees with natural interactions and psychomotor skill development closer to live cases 15–17 , albeit conventional system have yet to provide XR visualization. Our work focuses on introducing a cutting-edge XR-based 3D visualization system centered on the Meta Quest 3 platform. By leveraging dual-cameras and video processing to track the catheter's 3D position and shape, our system addresses the limitations in depth perception of 2D fluoroscopic views. The real-time 3D catheter reconstruction, seamlessly integrates into an interactive environment, designed to enhance visualization, and offers a more immersive training experience compared to existing methods 18 . The use of XR, with its ability to replicate real procedural tools, ensures that operators can develop psychomotor skills in a manner closely resembling live cases, a feat not achievable with virtual reality (VR) alone. This study aligns with the broader trend of immersive technologies gaining prominence in medical training, particularly in the realm of cardiac interventions 19 . These advancements go beyond mere technical innovation, addressing critical concerns such as radiation exposure, cost-effectiveness, and the need for standardized training environments 20 . The integration of XR technologies into percutaneous transcatheter training not only presents a leap forward in skill acquisition but also offers a safer and more accessible approach for trainees. The outcomes of this work hold implications not only for the field of cardiac interventions, but also for medical training as a whole. The establishment of a comprehensive understanding of the benefits and challenges associated with 3D visualization in MR-based systems contributes to the ongoing discourse on optimizing simulation technologies. Our innovation introduces a novel approach to an XR-based cardiac catheterization system, incorporating the Meta Quest 3 and real-time 3D catheter reconstruction derived from a physical catheter manipulated by the user. By generating a precise 3D model of the catheter seamlessly within a patient-specific rendering of the heart within the Meta Quest 3, our system strives to offer enhanced visualization and immersive training experiences compared to existing methods. The unique multi-view perspectives provided by our system has the potential to lower learning curved for developing dexterity for new catheter systems and to prepare for specific cases with complex anatomies. 2. METHODS The proposed method involves designing and building a real-time catheter tracking system compatible with the commercial Meta Quest 3 XR headset. This system will be suitable for use with a variety of commercially available catheters (Fig. 1 -a). The hardware component utilizes a 3D-printed frame to secure the catheter within a region of interest (boundary of heart), simultaneously recorded by two orthogonally positioned cameras (Fig. 1 -b). A custom-designed computer vision (CV) algorithm (Fig. 1 -c) infers the catheter's shape and orientation from the biplane views (Fig. 1 -d, e). This data is used to reconstruct the catheter's 3D shape (Fig. 1 -f) and transmitted in real-time to the XR headset. The headset leverages the Unity game engine to provide a stand-alone rendering environment (Fig. 1 -i). This system will facilitate a specific set of tasks related to precise catheter positioning. The reconstructed catheter data is co-registered and visualized within a patient-specific anatomical heart model (Fig. 1 -h) generated from a cardiac computed tomography (CT) scan acquired at end-diastole DICOM format (Fig. 1 -g). This integration occurs within the Meta Quest 3 (Fig. 1 -j). Users can physically manipulate a real commercial catheter (identical to those used in a cath lab) and maneuver it in real-time within the 3D patient heart model, observing the process via the Meta Quest 3 XR headset. The following sections provide a detailed technical description of each development stage. 2.1. Design and Fabrication of the 3D Printed Setup To facilitate tracking of the catheter and establish a physical platform for controlled catheter manipulation, we designed a cubic 3D model using Dassault Systems SolidWorks 2022 and fabricated it using a 3D printer (Fig. 2 ). As illustrated in Fig. 2 , several components were integrated into the 3D setup to enable accurate computer vision tracking of the catheter. A removable inlet with a 4 mm diameter, centered at (0,0,0) in our 3D Cartesian coordinate system, is included for catheter insertion and articulation into the central open space within the cube. This inlet can be easily exchanged to accommodate catheters of varying thickness. The setup contains mounts designed to securely hold two cameras at specific locations along two sides of the cube, forming an orthogonal biplane imaging system. From these fixed positions, the two cameras capture video of the catheter movement through the central tracking region of interest (ROI). To specify the ROI, eight fiducial markers ( m Ti and m Fi for i = 1,2,3, and 4, T and F denotes Top, and Front respectively) were 3D printed onto specific locations within the model on two cross-shaped pillars, ensuring that the ROI can accommodate the size of a human heart. The fiducial markers are in two orthogonal planes, allowing the transformation of each camera’s perspective to a global coordinate system. This coordinate transformation is essential in reconstructing the 3D catheter trajectory within the 3D model workspace, ensuring that heart is centered in the ROI and the inlet of both the cube and heart are aligned. The cubic planes are printed using a white color to provide a high contrast background against the catheter when viewed by the cameras. This design choice helps with more accurate isolation and segmentation of the catheter from the background by the computer vision tracking algorithm. The setup, including all sections and backgrounds, is 3D printed from rigid materials (VeroClear and VeroUltraWhite for backgrounds) on a Stratasys J826 PolyJet printer. 2.2. Computer Vision-Based Catheter Segmentation and 3D Trajectory Extraction The main component enabling real-time tracking of the catheter position is a computer vision segmentation algorithm that first identifies and segments the catheter and then reconstructs the full 3D trajectory from synchronized orthogonal camera footage captured from the workspace. The processing pipeline operates on a per-frame basis, taking in video streams from two cameras positioned orthogonally around the 3D cube model. The cameras are intrinsically calibrated and have known, fixed poses relative to the prespecified coordinate space. The proposed vision algorithm has nine steps, as illustrated in Fig. 3 . Each frame of the real-time video is denoted by I(r, c, t) , where r , c , and t represent the pixel row, pixel column, and time dimensions, respectively. To start, every frame undergoes a perspective transformation using a perspective transformation matrix (Eq. 1 ) mapping the front and the top cameras to the same global coordinate space, based on the known fiducial points (denoted by m T , and m F ). Vectors m T and m F , each consisting of eight coordinate values ( m Ti and m Fi for i = 1,2,3, and 4 as depicted in Fig. 1 ), are used to specify the four corners of the ROI for the top and front planes, respectively. These vectors are marked on the two printed crosses within the setup (Fig. 1 ). The perspective transformation facilitates consistent image processing in the unified coordinate space and cancels out the distortion. As expressed in Eq. 1 , the transformed coordinate points (x', y') are obtained from the local camera-based coordinates (x, y). The coordinate transformation matrix implements an affine transformation using rotation and scaling (a1 to a4), translation (b1 and b2), and projection (c1 and c2). The parameters of the transformation matrix are initially unknown. To obtain these parameters, a set of eight equations need to be solved. To do so, as an initial step, four fiducial points are selected within the input image. Subsequently, these predefined points are mapped to predetermined locations based on the known dimensions and positions of ROIs. This procedure yields a system of eight equations and eight unknowns, allowing for a solvable configuration. We can then compute the resulting perspective transformation matrix. After the acquisition of the transformation matrix, all the input frames undergo perspective transformation, yielding coordinate-normalized images. Matrix parameters are acquired using the initial frame of each video, enabling seamless application across subsequent frames in real-time scenarios. $$\left[\begin{array}{c}X{\prime }\\ Y{\prime }\\ 1\end{array}\right]=\left[\begin{array}{ccc}{a}_{1}& {a}_{2}& {b}_{1}\\ {a}_{3}& {a}_{4}& {b}_{2}\\ {c}_{1}& {c}_{2}& 1\end{array}\right]\left[\begin{array}{c}X\\ Y\\ 1\end{array}\right]$$ 1 With the frames aligned, preprocessing steps are applied including Gaussian smoothing to reduce background noise and enable robust detection; a larger Gaussian kernel size increases the smoothness but can also degrade localization precision. Next, contrast/brightness adjustments enhance the visibility of the catheter against the white background. To isolate the catheter, an adaptive thresholding operation is applied, which converts the grayscale frame into a binary image based on dynamic local thresholding. The adaptive thresholding algorithm computes individualized threshold value, denoted by T(x, y) , for each pixel (x, y) , by averaging the pixel values present in a localized neighborhood window centered on the target pixel. In other words, Adaptive thresholding is as follows: T(x,y) = mean(neighborhood) – C ( 2 ) Where mean(neighborhood) is the mean of the pixel values within the neighborhood proximity window. To further normalize the pixel intensity threshold, the constant C, is subsequently subtracted from this mean value. C is a customizable constant that can be positive, negative, or zero (but is typically positive). A positive C raises the threshold leading to a darker image, a negative C lowers it leading to a brighter image, and zero uses the mean directly. The optimal value for C depends on the specific image and desired outcome. It's generally chosen through experimentation to achieve the best possible segmentation or object detection results. Each pixel's final value, represented by g(x, y), is then obtained by masking the input frames using the computed threshold as in: $$g\left(x,y\right)=\left\{\begin{array}{c}1, PV(x,y) > T(x,y) \\ 0, PV(x,y) \le T(x,y)\end{array}\right.$$ 3 Where the variable PV(x, y) denotes the pixel value within the input image. Consequently, the process involves binary classification of each pixel, wherein the adaptation to a locally determined threshold takes precedence over a globally assigned value for the entire image. The configurability of this method is manifested through the adjustable parameters of the neighborhood block size and the constant C . While larger block sizes serve to mitigate the impact of noisy artifacts, it is imperative to acknowledge that they concurrently introduce a trade-off by diminishing the precision of localization. In the subsequent stage of the proposed vision algorithm, a morphological operation for skeletonization is introduced to attenuate the binary representation of the catheter object, shaping it into a central, pixel-wide arc that characterizes the medial axis trajectory. This transformation yields a concise portrayal, streamlining the subsequent tracking process. Distinct skeletonization methods are available, such as Zhang's method 21 and Lee’s method 22 . In this instance, Zhang’s method has been implemented, which involves a series of sequential passes across the image, systematically eliminating pixels situated on the periphery of the object. This iterative process persists until further removal of pixels becomes unattainable. Initially, the tip location is determined by identifying the point with the maximum number of neighboring true values (white pixels) 23 . The coordinates of the inferred tip are then recorded as the first data point within an array, with the corresponding pixels being set to true values in the skeletonized binary catheter image. This procedure is repeated iteratively until all pixel’s transition to true values, thereby documenting the coordinates of the entire catheter trajectory from its tip to its entry point. This yields a two-dimensional directional array (vector) that succinctly encapsulates the spatial trajectory of the catheter for each frame—from its tip to its entry point. These pixel coordinates are subsequently mapped onto a real-world 3D coordinate system in millimeters, utilizing known phantom dimensions and camera intrinsic parameters. The coordinates are further down sampled to generate a subset of K points, where the initial point denotes the catheter's tip, the concluding point designates the entry point, and the intervening K-2 points are evenly distributed to delineate the catheter's curvature. The computer vision algorithm is implemented in Python using the OpenCV library and operates in real-time on both planes. The resulting K points infer the X, Y coordinates from the top plane and Z coordinate from the front plane. Combining these points results in a real-time 3D K-point tracking system, providing a holistic representation of the catheter's spatial configuration and its curvature. 2.3. Patient-Specific 3D Heart Model Generation To generate a patient-specific 3D model of the heart, we utilized Materialize Mimics Research software version 21.0 for 3D image processing. The initial step involved importing a cardiac computed tomography (CT) scan acquired at end-diastole in the DICOM format, as demonstrated in Fig. 4 -a. Subsequently, this data underwent image segmentation within Mimics Software to delineate the heart and spine, forming a unified 3D mask while preserving their respective spatial relationships. The resulting 3D segmentation was saved as an STL file. To further refine the model and eliminate extraneous components, such as vessels and ribs, we employed Geomagic Wrap software developed by 3D Systems Geomagic Corporation. This refinement process effectively removed artifacts and smoothed the mesh, as illustrated in Fig. 4 -b and 4 -c. The generated 3D heart is used in the proposed XR scene. The initial step involved exporting the model from Geomagic as an STL file, which was then imported into the CAD software SolidWorks. Within SolidWorks, meticulous attention was given to the positioning of six posts inside the Right Atrium, ensuring optimal accessibility from the inferior vena cava using a catheter (Fig. 4 -d). Each post was constructed from an extrusion of a 2.9 mm diameter circle, resulting in a 4 mm long cylinder. The xyz coordinates were precisely measured at the center of each post (Fig. 4 -e). Upon finalizing the model, four spherical markers, each with a diameter of 3 mm, were added parallel to the spine base (Fig. 4 -f). These markers will serve as reference points for orientation and integration within the Unity scene system. Finally, the entire model was exported as a single GLTF file, with a bin file associated with each individual element (spine, heart, post 1, post 2, etc.), facilitating its seamless integration into the XR scene. 2.4. Mixed Reality Rendering Employing the Unity game engine, we have integrated the inferred tracking data of a catheter and a patient-specific cardiac model into a mixed reality application tailored for the Meta Quest 3 headset. Based on the steps explained in section 2.3 , a 3D model of the patient's heart, reconstructed from a CT scan 9,10,24 , is rendered using the Unity game engine. Furthermore, the catheter's spline obtained from the inferred K points, acquired in real-time by our proposed computer vision algorithm, are also rendered in the scene. For this mapping to be accurate, alignment of fiducial points on the 3D printed model with corresponding anatomical landmarks is essential. The communication between the output of the Python-based computer vision algorithm and the 3D rendering in Unity, is implemented by using the web-based library Flask, which is a common choice for efficient real-time data transfer. More specifically, inferred tracking data comprised of the position of each of the K points on the catheter, as well as the rotation of the tip is saved using a JSON file and then transmitted to Unity through a Flask-based WebSocket connection. The Flask server can be run on any IP address and port that the user specifies, with the default being localhost at port 5000. To connect this data to the Unity application, the user is prompted to enter this IP address and port when they launch the app and connect to the server using the SocketIO Unity package. After receiving the transmitted tracking data, Unity renders dynamic updates to both the catheter location and curvature in the 3D scene. Using the fiducial markers, an affine transformation is used to correctly map the catheter placement and movement with respect to the 3D model of the heart. The catheter rendering is superimposed within the patient-specific heart rendering and surrounding scene within the Quest 3 headset, creating a cohesive XR environment. This integrated platform offers an interactive experience that goes beyond mere visualization, allowing users to observe and analyze the catheter's movements in real time with quantitative feedback based on tracking relative to pre-defined target positions. The visual representation is contextualized within the reconstructed cardiac anatomy, providing a valuable tool for training simulations. Moreover, this innovative approach holds considerable potential for real-life applications in clinical settings, where the interactive XR platform could contribute to improved catheterization procedures through enhanced training and procedural guidance. 3. RESULTS AND DISCUSSION Real-time pose detection of a catheter is a critical need for a XR system to perform accurate three-dimensional tracking to provide guidance and navigation for clinical procedures. Integrating this capability into a XR system can enhance medical training, allowing practitioners to refine dexterity, spatial awareness, and procedural skills. Moreover, the detailed 3D representation of the catheter aids in preoperative planning, enabling clinicians to strategize and optimize their approach, thereby reducing errors during actual interventions. This approach, complemented by the proposed K-points tracking algorithm (i.e., arbitrary number of pre-defined points tracked along the length of a catheter), was evaluated through comprehensive experiments using both a commercially available catheter and 3D-printed catheter phantoms characterized by predetermined angles and curvatures. To evaluate the system's adaptability to diverse commercial catheter types, we conducted initial tests with two specific catheters: the OSCOR Sheath and the PHILIPS Intracardiac Echocardiography (ICE) catheter. Samples of each catheter are depicted in Fig. 5 . For a more thorough demonstration of the system's functionality with these catheters, additional videos can be found in the Supplementary materials ( Supplementary Videos Sa, and Sb ). As another evaluation, our assessment encompassed a dual-focused approach, including the analysis of tip tracking accuracy with the OSCOR Sheath and a scrutiny of the algorithm's capacity for 3D pose reconstruction across the entire catheter length using a mock 3D-printed catheter. The outcomes of these experiments will yield valuable insights into the algorithm's potential application and performance in practical clinical scenarios, such as cardiac interventions. This, in turn, aims to enhance procedural precision, efficiency, and overall patient care by minimizing complications in catheter-based procedures. 3.1 Tip Localization Accuracy As an initial test, the accuracy of localizing the catheter tip position was quantified. Five fiducial points (P1-P5 in Fig. 6 ) with known 3D coordinates were 3D printed onto the boundary of interest. The commercial catheter was maneuvered such that its tip touched each fiducial point, with the contact captured by the biplane cameras. The absolute difference of 3D coordinates between the reconstructed tip position (measured) from the computer vision algorithm and the actual fiducial coordinates was computed as shown in the table in Fig. 6 -d. As you can see in this table, the average tip localization error is less than 1 mm in 3D coordinates. This level of accuracy provides confidence in the ability to precisely reconstruct the catheter's position. 3.2 3D Pose Reconstruction of different Catheter Configurations To evaluate the reconstruction of full catheter trajectories, experiments were conducted with 3D printed catheter phantoms designed with three different fixed angles: 30°, 45°, and 60° (Fig. 7 -a). Each of the 3D printed and fixed angles catheter phantoms was positioned at four different locations within the workspace, aligned to the inlet hole. At each position, the catheter phantom was rotated manually at various speeds while both cameras captured video for processing. The K-points tracking algorithm processed the biplane videos, reconstructing the full 3D trajectory over time. Qualitative results depicting the reconstructed 3D trajectories across different angles and positions are shown in Fig. 7 -b-d for 30°, 45°, and 60° fixed angle catheters, respectively. For quantitative assessment, the catheter's shape was isolated from specific frames and the angles were calculated from the coordinates of each of the K-points. Results showed an average reconstruction error of ~ 0.6°, ~ 0.37°, and ~ 0.1° for the 30°, 45°, and 60° catheters, respectively, across all k-points. Representative 2D projections of the reconstructed shapes are provided in Fig. 7 -e-g. The low reconstruction error highlights the capability to reliably reconstruct diverse catheter configurations. In Supplementary Videos Sc to Se , you can see several sample videos for this test. To subject the system to a more challenging test, we fabricated three distinct catheters with arbitrary curves using 3D printing, as illustrated in Fig. 8 -a. Each 3D-printed catheter was securely positioned at a fixed location within the workspace, aligning with the inlet hole. Subsequently, the catheter phantom underwent manual rotation at various speeds while cameras recorded footage for subsequent processing. The K-points tracking algorithm analyzed the biplane videos, reconstructing the complete 3D trajectory over time. Qualitative results, showcasing the reconstructed 3D trajectories across different curves. To validate the fidelity of the reconstructed shape against the designed curved catheter form, we extracted the 2D view of the 3D reconstructed trajectory. Simultaneously, we extracted the 2D view of the 3D designed curved catheters from SolidWorks, which was used to design the curved catheters. We manually superimposed the designed catheters onto the 2D view of the reconstructed data. As depicted in Figs. 8 -b-d, a high degree of shape matching is evident, confirming that the reconstructed catheter shape precisely mirrors the design of the test catheter. In Supplementary Videos Sf to Sh you can see several sample videos for this test. 3.3 Extended Reality (XR) Results As shown in our previous work, the Unity engine's rendering capabilities enabled the presentation of a patient-specific heart, overlaid with real-time tracking of the catheter position and shape 9 . This work improved upon that XR scene with several improvements: i) No embedded wires are required in the catheter for EM sensor tracking, making it compatible with various commercially available catheters or sheaths. In contrast, previous methods necessitated embedding at least three wires into the catheter, which proved to be expensive, time-consuming, and unfeasible for a variety of catheters, such as ICE catheters. ii) The catheter is rendered using the new spline function that is composed of k-points (instead of just 3 that were tracked using the EM sensors) improving the fidelity of the rendered catheter shape, iii) The software was made compatible with the Meta Quest 3 headset, instead of just the Hololens 2, and the scene was programmed to allow for 6 pre-defined targets within the heart, instead of just 3 that were hardcoded to recapitulate a transseptal puncture. As shown in Fig. 9 , the visualization includes a central 3D view of the heart, with additional 2D view on the left, as well as endoscopic view and a close-up target view on the right side of the scene. This interactive XR platform allowed users to explore catheter navigation in relation to the intricate cardiac anatomy reconstructed from medical scans. The immersive experience allows the user to move their head or walk around while the headset maintains the rendered scene in the exact position it was set, preventing the scene from blocking the view of the user to see anything else in the room (i.e., patient, other imaging). Additionally, using the voice command “Reset View,” the user can move the scene directly in front of their current head position, allowing for additional flexibility This XR system serves as a versatile tool for visualizing catheterization procedures, providing users with real-time feedback on the catheter's motion relative to pre-defined targets. 3.3.1 Functionality and User Experience Assessment To evaluate the functionality and user experience of the proposed XR system for guiding a percutaneous transcatheter procedure, we conducted an experimental study involving six participants. This study was carried out in accordance with the guidelines and regulations of the Institutional Review Board (IRB) at Weill Cornell Medicine, which gave approval for this study to be conducted. Participants were employees at Weill Cornell Medicine, but were not authors of this paper, and all provided informed consent for participation in the study. It should be noted that participants were not cardiologists or interventionalists with prior experience with delivery catheters. The primary objective was to assess both the quantitative performance metrics and qualitative feedback regarding the system's efficacy in navigating a catheter through predetermined targets within the heart using different visualization modes, allowing us to assess the significance of 3D depth perception in localizing a catheter. The experimental setup began with a structured familiarization period, allowing participants a maximum of 10 minutes to acquaint themselves with the XR system and the catheter. This period was crucial to ensure that participants had a basic understanding of the system's operation and functionality before engaging in the experimental sessions. Following the familiarization period, participants engaged in three distinct sessions, each designed to evaluate a specific visualization mode: 2D-View In this session, participants were presented with two-dimensional representations (Fig. 10 -Left ) as would be seen in standard 2D monitors. The objective was to navigate the catheter to predetermined targets without any depth perception, such that mental coordination between multiple views is needed to triangulate a 3D position in space. 2D3D-View In this session, participants were presented with the same two-dimensional views as in Session 1, alongside a 3D mixed reality representation of the heart in the center of the display (Fig. 10 -Middle ). This hybrid visualization mode aimed to provide participants with additional spatial context to aid in target navigation utilizing the immersive nature of mixed reality headsets. 3D-View In this session, participants solely engaged with the 3D representation of the heart (Fig. 10 -Right ). This visualization mode aimed to simplify the attention of the user to only the 3D rendering to determine if the additional views from the 2D modes are beneficial, distracting, or inconsequential. The order of sessions was randomized to mitigate potential order effects and ensure the validity of the experimental findings. Throughout each session, meticulous records were kept, capturing the following key metrics: t ( Time ) : This metric represents the time duration (in seconds) taken by participants to complete the task within each session. It serves as a quantitative measure of procedural performance. nT ( number of Targets ) : This metric denotes the number of predetermined targets successfully reached by participants during each session. It serves as a quantitative measure of procedural accuracy and success rate. tT ( time per Target ) : This metric calculates the average time taken by participants to reach each individual target within a session. %tT/2D ( percentage of tT relative to tT of 2D-View ) : This metric compares the time taken by participants to reach targets in alternative visualization modes (2D3D-View and 3D-View) relative to the baseline 2D-View mode. It quantifies the efficiency gains achieved with the alternative visualization modes and serves as a comparative measure of procedural speed. The quantitative results of this experiment, as shown in Fig. 11 (see Table S1 for all results ) , underscore the significant advantages of a 3D-View for guiding catheterization procedures over 2D-View and even the 2D3D-View. The 3D-View consistently yielded the fastest task completion times and the highest number of successfully reached targets by all participants, irrespective of age group, indicating its clear performance superiority. Participants unanimously reported a superior experience with the 3D-View compared to other visual representations, a preference echoed across age groups. Despite varying levels of difficulty encountered, the 3D-View markedly facilitated more efficient performance compared to 2D and 2D3D views, as evidenced by time per target (tT) and percentage of tT relative to 2D-View session (%tT/2D). The %tT/2D metric illustrates that 2D3D-View and 3D-View allowed participants to reach targets approximately 3.1 (%tT/2D = %32.5) and 7.3 (%tT/2D = %13.6) times faster, respectively, than with 2D-View alone, signifying substantial speed and efficiency improvements with important implications for real-world interventional procedures. While 2D3D-View provided all information available in 3D-View, the additional 2D views appeared to distract from user performance (although not statistically significant with this low sample size), suggesting that the 3D-View alone is optimal for this procedure. To conduct a qualitative assessment, we have developed a survey comprising eight questions that explore various aspects of the participants' experiences. When asked to rate their experience on a scale of 1 to 5 (with 5 being the highest), the subjects unanimously agreed that the 3D view mode was significantly more helpful than the 2D view mode for improving the accuracy and speed of catheter placement (almost all subjects rated 5). This suggests that the 3D visualization provided by the XR system greatly enhanced the subjects' spatial comprehension and procedural skills during the simulated cardiac intervention. Furthermore, the subjects found the digital interface of the XR image-guidance system to be generally user-friendly (average rating of 4.2). They also expressed a high degree of confidence in the system's utility for both pre-operative training/planning and intra-operative procedures (average ratings of 5 and 4.5, respectively). The overall experience with the system was rated very positively, with an average score of 4.8 out of 5. When asked about their experience with video games, the subjects reported a wide range of experience levels, with an average rating of 2.7 out of 5. However, their experience with XR headsets was relatively low, with an average rating of 1.5 out of 5. These survey results demonstrate the potential of XR technologies to transform medical training and real-world procedures through enhanced visualization and interactivity. The subjects' feedback suggests that the developed XR image-guidance system could significantly improve spatial comprehension and procedural skills for complex cardiac interventions. 4. CONCLUSION This paper presents a novel XR platform for real-time 3D catheter tracking and visualization during simulated percutaneous cardiac procedures. The proposed computer vision algorithm leverages biplane imaging to accurately reconstruct varied catheter configurations within a 3D printed setup. Integration with the Meta Quest 3 headset enabled immersive visualization of the tracked catheter trajectory of a reconstructed patient-specific heart model. The comprehensive experiments showcase precise tip localization of the catheter within approximately 1 mm of error, coupled with a high fidelity of shape reconstruction along its length. Our experimental study, involving six participants, revealed that the 3D visualization provided by the proposed XR system significantly outperformed 2D visualization, both in terms of speed and user experience. These findings strongly suggest that the XR system holds promise for enhancing catheterization training by bolstering spatial comprehension and procedural skills. Such results underscore the system's potential to revolutionize transcatheter cardiac procedures training through enhanced spatial comprehension. The XR environment offers an interactive and intuitive platform for honing the intricate psychomotor skills essential for catheter manipulation within patient anatomy. This study, however, has two primary limitations that should be further explored in future work. These two limitations are: i) Small sample size . Although the results show statistically significant differences, the sample size is still small and therefore a larger study should be performed to show generalization of the impact of the differences seen between three viewing modes. ii) Subjects are non-cardiologists. All participants in this study were non-cardiologists or even interventionalists and therefore the conclusions can’t necessarily be generalized to the performance of trained individuals. However, this study does imply great impact for the learning curve of untrained individuals. Overall, this innovative approach shows promise in transforming training and future percutaneous transcatheter cardiac procedures through the combined benefits of enhanced 3D visualization, immersive interactions, and patient-specific reconstruction. While the current results validate the technical feasibility of the approach, further research can help translate this technology to practical clinical deployment. Through progressive research to enhance the visualization, expanding the clinical integration, and validating the training outcomes, this XR catheter tracking platform aims to effectively prepare next-generation cardiologists to master this vital and complex procedural skill. Declarations Competing Interests The authors declare no competing interests. Author Contribution Methodology and Machine vision algorithm design, M.A.; Coding and implementation of the algorithm, M.A.; Design and implementation of the 3D setup, M.A.; Testing and investigation, M.A.; Experimental study , and Data analysis, M.A.; Front-end development and coding, S.S.; Unity implementation and mixed reality, S.S.; Testing and investigation, S.S.; Code repository handling, S.J.; Testing and investigation, S.J.; Heart model and target design, A.C.; Resources, S.C.W.; Front-end project supervision, A.S.; Conceptualization, B.M.; Project administration, B.M.; Supervision, B.M.; Funding acquisition, B.M. All authors have read and agreed to the published version of the manuscript. Acknowledgments We thank Philips for providing the VeriSight ICE catheter supported by a research grant. Data Availability Data is provided within the manuscript or supplementary information files. References Bashore, T. M. et al. 2012 American College of Cardiology Foundation/Society for Cardiovascular Angiography and Interventions expert consensus document on cardiac catheterization laboratory standards update: A report of the American College of Cardiology Foundation Task Force on Expert Consensus documents developed in collaboration with the Society of Thoracic Surgeons and Society for Vascular Medicine. J Am Coll Cardiol 59, 2221–2305 (2012). https://doi.org:10.1016/j.jacc.2012.02.010 Malik, P. Grossman’s Cardiac Catheterization, Angiography, and Intervention . 7 edn, Vol. 23 (2007). Gaba, P. et al. Percutaneous Coronary Intervention vs Coronary Artery Bypass Graft Surgery for Left Main Disease in Patients With and Without Acute Coronary Syndromes: A Pooled Analysis of 4 Randomized Clinical Trials. JAMA Cardiology 8, 631–639 (2023). https://doi.org:10.1001/jamacardio.2023.1177 Okumus, N., Abraham, S., Puri, R. & Tang, W. H. W. Aortic Valve Disease, Transcatheter Aortic Valve Replacement, and the Heart Failure Patient. JACC: Heart Failure 11, 1070–1083 (2023). https://doi.org:doi:10.1016/j.jchf.2023.07.003 Bangalore, S. et al. Evidence-Based Practices in the Cardiac Catheterization Laboratory: A Scientific Statement From the American Heart Association. Circulation 144, e107-e119 (2021). https://doi.org:10.1161/cir.0000000000000996 Yoon, S.-H. et al. Outcomes After Transcatheter Edge-to-Edge Mitral Valve Repair According to Mitral Regurgitation Etiology and Cardiac Remodeling. JACC: Cardiovascular Interventions 15, 1711–1722 (2022). https://doi.org:doi:10.1016/j.jcin.2022.07.004 Holmvang, L., Lüscher, M. S., Clemmensen, P., Thygesen, K. & Grande, P. Very Early Risk Stratification Using Combined ECG and Biochemical Assessment in Patients With Unstable Coronary Artery Disease (A Thrombin Inhibition in Myocardial Ischemia [TRIM] Substudy). Circulation 98, 2004–2009 (1998). https://doi.org:doi:10.1161/01.CIR.98.19.2004 Sorajja, P., Michael, J. L. & Morton, J. K. Cardiac Catheterization Handbook E-Book . (Elsevier Health Sciences, 2021). Jang, S.-J. et al. Development of a Hybrid Training Simulator for Structural Heart Disease Interventions. 2, 2000109 (2020). https://doi.org:https://doi.org/10.1002/aisy.202000109 Liu, J. et al. An augmented reality system for image guidance of transcatheter procedures for structural heart disease. PloS one 14, e0219174 (2019). https://doi.org:10.1371/journal.pone.0219174 Wu, D. et al. Comparative Analysis of Interactive Modalities for Intuitive Endovascular Interventions. IEEE Transactions on Visualization & Computer Graphics, 1–18 (5555). https://doi.org:10.1109/tvcg.2024.3362628 Joshi, A. & Wragg, A. Simulator Training in Interventional Cardiology. Interv Cardiol 11, 70–73 (2016). https://doi.org:10.15420/icr.2016.11.1.70 Mercuri, M., Sheth, T. & Natarajan, M. K. Radiation exposure from medical imaging: a silent harm? Cmaj 183, 413–414 (2011). https://doi.org:10.1503/cmaj.101885 Stahl, C. M., Meisinger, Q. C., Andre, M. P., Kinney, T. B. & Newton, I. G. Radiation Risk to the Fluoroscopy Operator and Staff. AJR Am J Roentgenol 207, 737–744 (2016). https://doi.org:10.2214/ajr.16.16555 De Ponti, R. et al. Superiority of simulator-based training compared with conventional training methodologies in the performance of transseptal catheterization. J Am Coll Cardiol 58, 359–363 (2011). https://doi.org:10.1016/j.jacc.2011.02.063 Viglialoro, R. M. et al. Augmented Reality, Mixed Reality, and Hybrid Approach in Healthcare Simulation: A Systematic Review. Applied Sciences 11, 2338 (2021). Gallagher, A., McClure, N., McGuigan, J., Crothers, I. & Browning, J. Virtual reality training in laparoscopic surgery: a preliminary assessment of minimally invasive surgical trainer virtual reality (MIST VR). Endoscopy 31, 310–313 (1999). Linte, C. A., Moore, J., Wiles, A. D., Wedlake, C. & Peters, T. M. Virtual reality-enhanced ultrasound guidance: a novel technique for intracardiac interventions. Comput Aided Surg 13, 82–94 (2008). https://doi.org:10.3109/10929080801951160 Barsom, E. Z., Graafland, M. & Schijven, M. P. Systematic review on the effectiveness of augmented reality applications in medical training. Surgical endoscopy 30, 4174–4183 (2016). Klein, L. W. et al. Occupational health hazards in the interventional laboratory: time for a safer environment. Radiology 250, 538–544 (2009). https://doi.org:10.1148/radiol.2502082558 Zhang, T. Y. & Suen, C. Y. A fast parallel algorithm for thinning digital patterns. Commun. ACM 27, 236–239 (1984). https://doi.org:10.1145/357994.358023 Lee, T. C., Kashyap, R. L. & Chu, C. N. Building Skeleton Models via 3-D Medial Surface Axis Thinning Algorithms. CVGIP: Graphical Models and Image Processing 56, 462–478 (1994). https://doi.org:https://doi.org/10.1006/cgip.1994.1042 Annabestani, M. & Naghavi, N. Non-uniform deformation and curvature identification of ionic polymer metal composite actuators. Journal of Intelligent Material Systems and Structures 26, 582–598 (2014). https://doi.org:10.1177/1045389X14538535 Torabinia, M. et al. Deep learning-driven catheter tracking from bi-plane X-ray fluoroscopy of 3D printed heart phantoms. Mini-invasive Surgery 5, 32 (2021). https://doi.org:10.20517/2574-1225.2021.63 Additional Declarations No competing interests reported. Cite Share Download PDF Status: Published Journal Publication published 06 Nov, 2024 Read the published version in Scientific Reports → Version 1 posted Editorial decision: Revision requested 26 Aug, 2024 Reviews received at journal 07 Aug, 2024 Reviews received at journal 04 Aug, 2024 Reviewers agreed at journal 28 Jul, 2024 Reviewers agreed at journal 16 Jul, 2024 Reviewers invited by journal 06 Jul, 2024 Editor assigned by journal 06 Jul, 2024 Editor invited by journal 02 Jul, 2024 Submission checks completed at journal 02 Jul, 2024 First submitted to journal 26 Jun, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4645065","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":329016421,"identity":"90a6e8f1-0dd9-447a-b7a1-0025b06e96c5","order_by":0,"name":"Bobak Mosadegh","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA8klEQVRIiWNgGAWjYBACAwYGxgcJBjZyEDYIsDM3MDCw4dXCbPChIs0YoYWZkaAWNskZZw4nNhCtxZy9O0Gaty0tfW178waGHxX38vhBWj6UHcapxbLn7AZj3jab3G1njhUw9pwpLpZsZmxgnHEOtxaDG7kbkoG25G67kWPAzNiWkLjhMGMDM28bfi2HgQrSze6/gWjZD9LyF7+WjY1A7yeY3eCB2gL0C5CB1y+bGYCBbLjtTFrBwZ4zCYkzgLYc7DmXjlOLOXvv9h/AqJQ3O35444MfFQmJ/e3NBx/8KLPGqQUFHMBgjIJRMApGwSggDwAAfP1e3MRKA8AAAAAASUVORK5CYII=","orcid":"","institution":"Weill Cornell Medicine","correspondingAuthor":true,"prefix":"","firstName":"Bobak","middleName":"","lastName":"Mosadegh","suffix":""},{"id":329016422,"identity":"e656c385-855c-4ed2-b30e-e0eff591d382","order_by":1,"name":"Mohsen Annabestani","email":"","orcid":"","institution":"Weill Cornell Medicine","correspondingAuthor":false,"prefix":"","firstName":"Mohsen","middleName":"","lastName":"Annabestani","suffix":""},{"id":329016423,"identity":"56352ee4-9eb0-49ba-bc4a-6b01b3e3487c","order_by":2,"name":"Sandhya Sriram","email":"","orcid":"","institution":"Weill Cornell Medicine","correspondingAuthor":false,"prefix":"","firstName":"Sandhya","middleName":"","lastName":"Sriram","suffix":""},{"id":329016424,"identity":"44beb6f0-77f8-4d90-8b9c-487af1d4ca3a","order_by":3,"name":"Alexandre Caprio","email":"","orcid":"","institution":"Weill Cornell Medicine","correspondingAuthor":false,"prefix":"","firstName":"Alexandre","middleName":"","lastName":"Caprio","suffix":""},{"id":329016426,"identity":"84a33f5a-1bc0-4800-944f-b6245d820f80","order_by":4,"name":"Sepehr Janghorbani","email":"","orcid":"","institution":"Weill Cornell Medicine","correspondingAuthor":false,"prefix":"","firstName":"Sepehr","middleName":"","lastName":"Janghorbani","suffix":""},{"id":329016427,"identity":"cbeb8cc7-9f21-4d98-8498-86ed6d3f418a","order_by":5,"name":"S. Chiu Wong","email":"","orcid":"","institution":"Weill Cornell Medicine","correspondingAuthor":false,"prefix":"","firstName":"S.","middleName":"Chiu","lastName":"Wong","suffix":""},{"id":329016428,"identity":"f47aa7ef-9319-410d-9990-5da321bb0bd0","order_by":6,"name":"Alexandros Sigaras","email":"","orcid":"","institution":"Caryl and Israel Englander Institute for Precision Medicine","correspondingAuthor":false,"prefix":"","firstName":"Alexandros","middleName":"","lastName":"Sigaras","suffix":""}],"badges":[],"createdAt":"2024-06-26 23:02:12","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4645065/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4645065/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1038/s41598-024-76384-z","type":"published","date":"2024-11-06T15:56:51+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":61176829,"identity":"c60857e2-42c5-4c08-a016-614ff1d87627","added_by":"auto","created_at":"2024-07-26 15:48:27","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":287128,"visible":true,"origin":"","legend":"\u003cp\u003eProposed XR-based system: (a) Commercially available catheters that were used in the system. (b) 3D-printed frame holding a catheter positioned in the virtual boundary of a heart within view of two cameras. (c) A custom computer vision algorithm. (d \u0026amp; e) Inferred catheter shape from biplane views. (f) Reconstructed 3D shape of catheter. (g) segmented blood volume from deidentified CT scan (DICOM), (h) Patient-specific 3D heart mesh model, (i) Unity-powered rendering environment, (j) Meta Quest 3 XR headset. (k) User manipulating a real catheter in the 3D model of real patient’s heart and observing the process in the headset.\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-4645065/v1/19869f25fd2ac99e4c08f4b7.png"},{"id":61176831,"identity":"4d8998e5-258c-4604-a8e6-ac57fc8ee442","added_by":"auto","created_at":"2024-07-26 15:48:28","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":249616,"visible":true,"origin":"","legend":"\u003cp\u003eProposed 3D-printed frame. The schematic design of the 3D model (Left) and image of the printed version (Right). The frame features a removable inlet fixture for catheter insertion at the center of the region of interest (ROI). It incorporates two cross-shaped pillars, each marked with four color-coded points (mTi and mFi for i=1,2,3, and 4, where T and F denote Top and Front), defining the cubic boundary of the heart. Additionally, two mounts are placed on two sides of the cube to securely accommodate two cameras, forming an orthogonal biplane imaging system.\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-4645065/v1/b937a13179865c8d282f0fd9.png"},{"id":61176828,"identity":"b4a08298-bcfb-49a1-a95e-8aa73bfe23a2","added_by":"auto","created_at":"2024-07-26 15:48:27","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":27855,"visible":true,"origin":"","legend":"\u003cp\u003eBlock diagram of\u003cstrong\u003e \u003c/strong\u003eproposed vision algorithm.\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-4645065/v1/467bef79b8c001c317a68267.png"},{"id":61176830,"identity":"a5203d72-018d-4db3-8e53-ac6e38406d17","added_by":"auto","created_at":"2024-07-26 15:48:28","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":349310,"visible":true,"origin":"","legend":"\u003cp\u003ePatient-Specific 3D Heart Model Generation steps. (a) Screenshot captured from Materialize Mimics Research software demonstrating the segmentation process of a cardiac CT scan to isolate the heart and the spine into a 3D object. (b) An initial screenshot from Geomagic Wrap software before the cleaning and refinement stages of the 3D volume representing the heart and spine. (c) Post-refinement, a screenshot from Geomagic Wrap software displaying a smooth 3D volume of the spine and heart. (d) Screenshot of the software Solidworks presenting the entire spine and a sectional view of the right atrium, highlighting the incorporation of six posts within the heart. (e) A cross-sectional view of the right atrium, highlighting the design and measurement of the posts and their coordinates. (f) the full spine and the heart with the four fiducial markers.\u003c/p\u003e","description":"","filename":"4.png","url":"https://assets-eu.researchsquare.com/files/rs-4645065/v1/f04ad97608f197efe2cd49a1.png"},{"id":61178040,"identity":"f165b217-315e-42d5-98f2-f20ec6281990","added_by":"auto","created_at":"2024-07-26 15:56:28","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":197555,"visible":true,"origin":"","legend":"\u003cp\u003eExemplary frame showcasing 3D tracking of two distinct catheters using the proposed method. OSCOR Sheath (Left), Philips ICE catheter (Right).\u003c/p\u003e","description":"","filename":"5.png","url":"https://assets-eu.researchsquare.com/files/rs-4645065/v1/fe0d7a1fe63851b97ece693d.png"},{"id":61176836,"identity":"905bf6f5-b473-4d99-8d70-4535c31ecb6d","added_by":"auto","created_at":"2024-07-26 15:48:28","extension":"png","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":152774,"visible":true,"origin":"","legend":"\u003cp\u003eFiducial points for validation of tip localization accuracy. (a) 3D printed setup. (b) Five fiducial points (designed version), (c) Five fiducial points (real version), (d) Values of the 3D coordinates for the fiducial points shown as the actual coordinates by design, experimentally measured coordinates, and the absolute difference between the two to show accuracy.\u003c/p\u003e","description":"","filename":"6.png","url":"https://assets-eu.researchsquare.com/files/rs-4645065/v1/9bf14d363d074cba10aa1530.png"},{"id":61178914,"identity":"61faea0e-5dd2-4e13-8b85-5fde3bdef88a","added_by":"auto","created_at":"2024-07-26 16:04:28","extension":"png","order_by":7,"title":"Figure 7","display":"","copyAsset":false,"role":"figure","size":309647,"visible":true,"origin":"","legend":"\u003cp\u003e(a)\u003cstrong\u003e \u003c/strong\u003e3D printed catheter phantoms for three different fixed angles (30°, 45°, and 60°). Reconstructed 3D trajectories of the 3D printed fixed angled catheters at four different locations for the three fixed angled catheters (b) 30°, (c) 45°, and (d) 60°. 2D projections and measured angles of the reconstructed trajectory of a 3D printed catheter for four different locations with a (e) 30° bend, (f) 45° bend, and (g) 60° bend.\u003c/p\u003e","description":"","filename":"7.png","url":"https://assets-eu.researchsquare.com/files/rs-4645065/v1/e843769cd18b3476752b7199.png"},{"id":61178037,"identity":"dd1e1fe2-54a6-4311-8b6d-b084d86da5dc","added_by":"auto","created_at":"2024-07-26 15:56:28","extension":"png","order_by":8,"title":"Figure 8","display":"","copyAsset":false,"role":"figure","size":200424,"visible":true,"origin":"","legend":"\u003cp\u003e(a)\u003cstrong\u003e \u003c/strong\u003e3D printed mock catheters for three different arbitrary curves. 2D projections of the designed and reconstructed trajectory of the 3D printed catheters with a design for (b) Curve1, (c) Curve2, and (d) Curve3.\u003c/p\u003e","description":"","filename":"8.png","url":"https://assets-eu.researchsquare.com/files/rs-4645065/v1/917ba8aa31ae9e40e8f6f4a3.png"},{"id":61178038,"identity":"7dd830ea-83b7-4598-9b8a-edca0798b272","added_by":"auto","created_at":"2024-07-26 15:56:28","extension":"png","order_by":9,"title":"Figure 9","display":"","copyAsset":false,"role":"figure","size":324200,"visible":true,"origin":"","legend":"\u003cp\u003eA sample scene of the proposed XR-based catheter tracking system.\u003c/p\u003e","description":"","filename":"9.png","url":"https://assets-eu.researchsquare.com/files/rs-4645065/v1/7c1a0e68ef024afc1a08b201.png"},{"id":61178036,"identity":"98397856-d22b-49cf-9f3b-8a5268db23d4","added_by":"auto","created_at":"2024-07-26 15:56:28","extension":"png","order_by":10,"title":"Figure 10","display":"","copyAsset":false,"role":"figure","size":219680,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eLeft:\u003c/strong\u003e 2D-View, \u003cstrong\u003eMiddle:\u003c/strong\u003e 2D3D-View, \u003cstrong\u003eRight:\u003c/strong\u003e3D-View\u003c/p\u003e","description":"","filename":"10.png","url":"https://assets-eu.researchsquare.com/files/rs-4645065/v1/271717b20eee93ac90585c1d.png"},{"id":61176832,"identity":"57614dfc-735b-41e3-8dcf-64d02bb32f10","added_by":"auto","created_at":"2024-07-26 15:48:28","extension":"png","order_by":11,"title":"Figure 11","display":"","copyAsset":false,"role":"figure","size":18188,"visible":true,"origin":"","legend":"\u003cp\u003eXR study results. Plot of measured averages across all participants with standard deviations. t(s) signifies total time duration for each session and tT(s) signifies average time per target. * represents a statistically significant p-value \u0026lt; 0.05.\u003c/p\u003e","description":"","filename":"11.png","url":"https://assets-eu.researchsquare.com/files/rs-4645065/v1/2b03d1ff8ad403cb323a575f.png"},{"id":68749658,"identity":"42a9e105-ded4-4e36-9d8b-dab83ab2a399","added_by":"auto","created_at":"2024-11-11 16:00:35","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":2861929,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4645065/v1/cf669565-655c-46c1-b3d6-5249ab9c2e4d.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"High-Fidelity Pose Estimation for Real-Time Extended Reality (XR) Visualization for Cardiac Catheterization","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003ePercutaneous cardiac procedures demand precision in manipulating catheters through blood vessels and heart chambers for access in many cardiovascular conditions such as coronary artery disease, valvular heart disease, and structural heart defects \u003csup\u003e1\u0026ndash;6\u003c/sup\u003e. The current methods, relying on fluoroscopy and echocardiography imaging, fall short in providing a comprehensive view of the catheter's dynamic position and orientation within the patient, necessitating significant mental reconstruction of the anatomical context \u003csup\u003e7,8\u003c/sup\u003e. As the field of complex catheter-based structural heart interventions advances, there is a pressing need for systems that offer enhanced visualization and guidance. This paper delves into the technical challenges required to provide real-time catheter tracking to an extended reality (XR) system with high speed and accuracy. Such capabilities will have applications for various use in percutaneous transcatheter cardiac interventions training \u003csup\u003e9\u003c/sup\u003e, pre-procedural planning, intra-operative guidance \u003csup\u003e10\u003c/sup\u003e, and Telesurgery to enable physicians to remotely maneuver catheters \u003csup\u003e11\u003c/sup\u003e. Conventional training techniques, anchored in fluoroscopy-guided observations and hands-on practice, pose challenges such as high costs, radiation exposure, and limited spatial comprehension for trainees \u003csup\u003e12\u0026ndash;14\u003c/sup\u003e. The limitation of 2D visualization in representing the catheter's 3D location prompts the exploration of innovative solutions. XR, by combining 3D visualizations with real procedural tools, such as the actual catheter used in clinical practice, offers an unparalleled advantage in providing trainees with natural interactions and psychomotor skill development closer to live cases \u003csup\u003e15\u0026ndash;17\u003c/sup\u003e, albeit conventional system have yet to provide XR visualization.\u003c/p\u003e \u003cp\u003eOur work focuses on introducing a cutting-edge XR-based 3D visualization system centered on the Meta Quest 3 platform. By leveraging dual-cameras and video processing to track the catheter's 3D position and shape, our system addresses the limitations in depth perception of 2D fluoroscopic views. The real-time 3D catheter reconstruction, seamlessly integrates into an interactive environment, designed to enhance visualization, and offers a more immersive training experience compared to existing methods \u003csup\u003e18\u003c/sup\u003e. The use of XR, with its ability to replicate real procedural tools, ensures that operators can develop psychomotor skills in a manner closely resembling live cases, a feat not achievable with virtual reality (VR) alone. This study aligns with the broader trend of immersive technologies gaining prominence in medical training, particularly in the realm of cardiac interventions \u003csup\u003e19\u003c/sup\u003e. These advancements go beyond mere technical innovation, addressing critical concerns such as radiation exposure, cost-effectiveness, and the need for standardized training environments \u003csup\u003e20\u003c/sup\u003e. The integration of XR technologies into percutaneous transcatheter training not only presents a leap forward in skill acquisition but also offers a safer and more accessible approach for trainees.\u003c/p\u003e \u003cp\u003eThe outcomes of this work hold implications not only for the field of cardiac interventions, but also for medical training as a whole. The establishment of a comprehensive understanding of the benefits and challenges associated with 3D visualization in MR-based systems contributes to the ongoing discourse on optimizing simulation technologies. Our innovation introduces a novel approach to an XR-based cardiac catheterization system, incorporating the Meta Quest 3 and real-time 3D catheter reconstruction derived from a physical catheter manipulated by the user. By generating a precise 3D model of the catheter seamlessly within a patient-specific rendering of the heart within the Meta Quest 3, our system strives to offer enhanced visualization and immersive training experiences compared to existing methods. The unique multi-view perspectives provided by our system has the potential to lower learning curved for developing dexterity for new catheter systems and to prepare for specific cases with complex anatomies.\u003c/p\u003e"},{"header":"2. METHODS","content":"\u003cp\u003eThe proposed method involves designing and building a real-time catheter tracking system compatible with the commercial Meta Quest 3 XR headset. This system will be suitable for use with a variety of commercially available catheters (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e-a). The hardware component utilizes a 3D-printed frame to secure the catheter within a region of interest (boundary of heart), simultaneously recorded by two orthogonally positioned cameras (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e-b). A custom-designed computer vision (CV) algorithm (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e-c) infers the catheter's shape and orientation from the biplane views (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e-d, e). This data is used to reconstruct the catheter's 3D shape (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e-f) and transmitted in real-time to the XR headset. The headset leverages the Unity game engine to provide a stand-alone rendering environment (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e-i). This system will facilitate a specific set of tasks related to precise catheter positioning. The reconstructed catheter data is co-registered and visualized within a patient-specific anatomical heart model (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e-h) generated from a cardiac computed tomography (CT) scan acquired at end-diastole DICOM format (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e-g). This integration occurs within the Meta Quest 3 (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e-j). Users can physically manipulate a real commercial catheter (identical to those used in a cath lab) and maneuver it in real-time within the 3D patient heart model, observing the process via the Meta Quest 3 XR headset. The following sections provide a detailed technical description of each development stage.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003e2.1. Design and Fabrication of the 3D Printed Setup\u003c/h2\u003e \u003cp\u003eTo facilitate tracking of the catheter and establish a physical platform for controlled catheter manipulation, we designed a cubic 3D model using Dassault Systems SolidWorks 2022 and fabricated it using a 3D printer (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e). As illustrated in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e, several components were integrated into the 3D setup to enable accurate computer vision tracking of the catheter. A removable inlet with a 4 mm diameter, centered at (0,0,0) in our 3D Cartesian coordinate system, is included for catheter insertion and articulation into the central open space within the cube. This inlet can be easily exchanged to accommodate catheters of varying thickness. The setup contains mounts designed to securely hold two cameras at specific locations along two sides of the cube, forming an orthogonal biplane imaging system. From these fixed positions, the two cameras capture video of the catheter movement through the central tracking region of interest (ROI).\u003c/p\u003e \u003cp\u003eTo specify the ROI, eight fiducial markers (\u003cem\u003em\u003c/em\u003e\u003csub\u003e\u003cem\u003eTi\u003c/em\u003e\u003c/sub\u003e and \u003cem\u003em\u003c/em\u003e\u003csub\u003e\u003cem\u003eFi\u003c/em\u003e\u003c/sub\u003e for i\u0026thinsp;=\u0026thinsp;1,2,3, and 4, T and F denotes Top, and Front respectively) were 3D printed onto specific locations within the model on two cross-shaped pillars, ensuring that the ROI can accommodate the size of a human heart. The fiducial markers are in two orthogonal planes, allowing the transformation of each camera\u0026rsquo;s perspective to a global coordinate system. This coordinate transformation is essential in reconstructing the 3D catheter trajectory within the 3D model workspace, ensuring that heart is centered in the ROI and the inlet of both the cube and heart are aligned. The cubic planes are printed using a white color to provide a high contrast background against the catheter when viewed by the cameras. This design choice helps with more accurate isolation and segmentation of the catheter from the background by the computer vision tracking algorithm. The setup, including all sections and backgrounds, is 3D printed from rigid materials (VeroClear and VeroUltraWhite for backgrounds) on a Stratasys J826 PolyJet printer.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e2.2. Computer Vision-Based Catheter Segmentation and 3D Trajectory Extraction\u003c/h2\u003e \u003cp\u003eThe main component enabling real-time tracking of the catheter position is a computer vision segmentation algorithm that first identifies and segments the catheter and then reconstructs the full 3D trajectory from synchronized orthogonal camera footage captured from the workspace. The processing pipeline operates on a per-frame basis, taking in video streams from two cameras positioned orthogonally around the 3D cube model. The cameras are intrinsically calibrated and have known, fixed poses relative to the prespecified coordinate space. The proposed vision algorithm has nine steps, as illustrated in Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eEach frame of the real-time video is denoted by \u003cem\u003eI(r, c, t)\u003c/em\u003e, where \u003cem\u003er\u003c/em\u003e, \u003cem\u003ec\u003c/em\u003e, and \u003cem\u003et\u003c/em\u003e represent the pixel row, pixel column, and time dimensions, respectively. To start, every frame undergoes a perspective transformation using a perspective transformation matrix (Eq.\u0026nbsp;\u003cspan refid=\"Equ1\" class=\"InternalRef\"\u003e1\u003c/span\u003e) mapping the front and the top cameras to the same global coordinate space, based on the known fiducial points (denoted by \u003cem\u003em\u003c/em\u003e\u003csub\u003e\u003cem\u003eT\u003c/em\u003e\u003c/sub\u003e, and \u003cem\u003em\u003c/em\u003e\u003csub\u003e\u003cem\u003eF\u003c/em\u003e\u003c/sub\u003e). Vectors \u003cem\u003em\u003c/em\u003e\u003csub\u003e\u003cem\u003eT\u003c/em\u003e\u003c/sub\u003e and \u003cem\u003em\u003c/em\u003e\u003csub\u003e\u003cem\u003eF\u003c/em\u003e\u003c/sub\u003e, each consisting of eight coordinate values (\u003cem\u003em\u003c/em\u003e\u003csub\u003e\u003cem\u003eTi\u003c/em\u003e\u003c/sub\u003e and \u003cem\u003em\u003c/em\u003e\u003csub\u003e\u003cem\u003eFi\u003c/em\u003e\u003c/sub\u003e for i\u0026thinsp;=\u0026thinsp;1,2,3, and 4 as depicted in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e), are used to specify the four corners of the ROI for the top and front planes, respectively. These vectors are marked on the two printed crosses within the setup (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). The perspective transformation facilitates consistent image processing in the unified coordinate space and cancels out the distortion. As expressed in Eq.\u0026nbsp;\u003cspan refid=\"Equ1\" class=\"InternalRef\"\u003e1\u003c/span\u003e, the transformed coordinate points (x', y') are obtained from the local camera-based coordinates (x, y). The coordinate transformation matrix implements an affine transformation using rotation and scaling (a1 to a4), translation (b1 and b2), and projection (c1 and c2). The parameters of the transformation matrix are initially unknown. To obtain these parameters, a set of eight equations need to be solved. To do so, as an initial step, four fiducial points are selected within the input image. Subsequently, these predefined points are mapped to predetermined locations based on the known dimensions and positions of ROIs. This procedure yields a system of eight equations and eight unknowns, allowing for a solvable configuration. We can then compute the resulting perspective transformation matrix. After the acquisition of the transformation matrix, all the input frames undergo perspective transformation, yielding coordinate-normalized images. Matrix parameters are acquired using the initial frame of each video, enabling seamless application across subsequent frames in real-time scenarios.\u003cdiv id=\"Equ1\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ1\" name=\"EquationSource\"\u003e\n$$\\left[\\begin{array}{c}X{\\prime }\\\\ Y{\\prime }\\\\ 1\\end{array}\\right]=\\left[\\begin{array}{ccc}{a}_{1}\u0026amp; {a}_{2}\u0026amp; {b}_{1}\\\\ {a}_{3}\u0026amp; {a}_{4}\u0026amp; {b}_{2}\\\\ {c}_{1}\u0026amp; {c}_{2}\u0026amp; 1\\end{array}\\right]\\left[\\begin{array}{c}X\\\\ Y\\\\ 1\\end{array}\\right]$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e1\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eWith the frames aligned, preprocessing steps are applied including Gaussian smoothing to reduce background noise and enable robust detection; a larger Gaussian kernel size increases the smoothness but can also degrade localization precision. Next, contrast/brightness adjustments enhance the visibility of the catheter against the white background. To isolate the catheter, an adaptive thresholding operation is applied, which converts the grayscale frame into a binary image based on dynamic local thresholding. The adaptive thresholding algorithm computes individualized threshold value, denoted by \u003cem\u003eT(x, y)\u003c/em\u003e, for each pixel \u003cem\u003e(x, y)\u003c/em\u003e, by averaging the pixel values present in a localized neighborhood window centered on the target pixel. In other words, Adaptive thresholding is as follows:\u003c/p\u003e \u003cp\u003e \u003cem\u003eT(x,y)\u0026thinsp;=\u0026thinsp;mean(neighborhood) \u0026ndash; C\u003c/em\u003e (\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e)\u003c/p\u003e \u003cp\u003eWhere \u003cem\u003emean(neighborhood)\u003c/em\u003e is the mean of the pixel values within the neighborhood proximity window. To further normalize the pixel intensity threshold, the constant C, is subsequently subtracted from this mean value. C is a customizable constant that can be positive, negative, or zero (but is typically positive). A positive C raises the threshold leading to a darker image, a negative C lowers it leading to a brighter image, and zero uses the mean directly. The optimal value for C depends on the specific image and desired outcome. It's generally chosen through experimentation to achieve the best possible segmentation or object detection results. Each pixel's final value, represented by g(x, y), is then obtained by masking the input frames using the computed threshold as in:\u003cdiv id=\"Equ2\" class=\"Equation\"\u003e\u003cdiv format=\"TEX\" class=\"mathdisplay\" id=\"FileID_Equ2\" name=\"EquationSource\"\u003e\n$$g\\left(x,y\\right)=\\left\\{\\begin{array}{c}1, PV(x,y) \u0026gt; T(x,y) \\\\ 0, PV(x,y) \\le T(x,y)\\end{array}\\right.$$\u003c/div\u003e\u003cdiv class=\"EquationNumber\"\u003e3\u003c/div\u003e\u003c/div\u003e\u003c/p\u003e \u003cp\u003eWhere the variable \u003cem\u003ePV(x, y)\u003c/em\u003e denotes the pixel value within the input image. Consequently, the process involves binary classification of each pixel, wherein the adaptation to a locally determined threshold takes precedence over a globally assigned value for the entire image. The configurability of this method is manifested through the adjustable parameters of the neighborhood block size and the constant \u003cem\u003eC\u003c/em\u003e. While larger block sizes serve to mitigate the impact of noisy artifacts, it is imperative to acknowledge that they concurrently introduce a trade-off by diminishing the precision of localization.\u003c/p\u003e \u003cp\u003eIn the subsequent stage of the proposed vision algorithm, a morphological operation for skeletonization is introduced to attenuate the binary representation of the catheter object, shaping it into a central, pixel-wide arc that characterizes the medial axis trajectory. This transformation yields a concise portrayal, streamlining the subsequent tracking process. Distinct skeletonization methods are available, such as Zhang's method \u003csup\u003e21\u003c/sup\u003e and Lee\u0026rsquo;s method \u003csup\u003e22\u003c/sup\u003e. In this instance, Zhang\u0026rsquo;s method has been implemented, which involves a series of sequential passes across the image, systematically eliminating pixels situated on the periphery of the object. This iterative process persists until further removal of pixels becomes unattainable.\u003c/p\u003e \u003cp\u003eInitially, the tip location is determined by identifying the point with the maximum number of neighboring true values (white pixels) \u003csup\u003e23\u003c/sup\u003e. The coordinates of the inferred tip are then recorded as the first data point within an array, with the corresponding pixels being set to true values in the skeletonized binary catheter image. This procedure is repeated iteratively until all pixel\u0026rsquo;s transition to true values, thereby documenting the coordinates of the entire catheter trajectory from its tip to its entry point. This yields a two-dimensional directional array (vector) that succinctly encapsulates the spatial trajectory of the catheter for each frame\u0026mdash;from its tip to its entry point. These pixel coordinates are subsequently mapped onto a real-world 3D coordinate system in millimeters, utilizing known phantom dimensions and camera intrinsic parameters. The coordinates are further down sampled to generate a subset of K points, where the initial point denotes the catheter's tip, the concluding point designates the entry point, and the intervening K-2 points are evenly distributed to delineate the catheter's curvature. The computer vision algorithm is implemented in Python using the OpenCV library and operates in real-time on both planes. The resulting K points infer the X, Y coordinates from the top plane and Z coordinate from the front plane. Combining these points results in a real-time 3D K-point tracking system, providing a holistic representation of the catheter's spatial configuration and its curvature.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003e2.3. Patient-Specific 3D Heart Model Generation\u003c/h2\u003e \u003cp\u003eTo generate a patient-specific 3D model of the heart, we utilized Materialize Mimics Research software version 21.0 for 3D image processing. The initial step involved importing a cardiac computed tomography (CT) scan acquired at end-diastole in the DICOM format, as demonstrated in Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e-a. Subsequently, this data underwent image segmentation within Mimics Software to delineate the heart and spine, forming a unified 3D mask while preserving their respective spatial relationships. The resulting 3D segmentation was saved as an STL file. To further refine the model and eliminate extraneous components, such as vessels and ribs, we employed Geomagic Wrap software developed by 3D Systems Geomagic Corporation. This refinement process effectively removed artifacts and smoothed the mesh, as illustrated in Fig.\u0026nbsp;\u0026lt;link rid=\"fig4\"\u0026gt;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u0026lt;/link\u0026gt;\u003c/span\u003e-b and \u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e-c. The generated 3D heart is used in the proposed XR scene. The initial step involved exporting the model from Geomagic as an STL file, which was then imported into the CAD software SolidWorks. Within SolidWorks, meticulous attention was given to the positioning of six posts inside the Right Atrium, ensuring optimal accessibility from the inferior vena cava using a catheter (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e-d). Each post was constructed from an extrusion of a 2.9 mm diameter circle, resulting in a 4 mm long cylinder. The xyz coordinates were precisely measured at the center of each post (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e-e). Upon finalizing the model, four spherical markers, each with a diameter of 3 mm, were added parallel to the spine base (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e-f). These markers will serve as reference points for orientation and integration within the Unity scene system. Finally, the entire model was exported as a single GLTF file, with a bin file associated with each individual element (spine, heart, post 1, post 2, etc.), facilitating its seamless integration into the XR scene.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003e2.4. Mixed Reality Rendering\u003c/h2\u003e \u003cp\u003eEmploying the Unity game engine, we have integrated the inferred tracking data of a catheter and a patient-specific cardiac model into a mixed reality application tailored for the Meta Quest 3 headset. Based on the steps explained in section \u003cspan refid=\"Sec5\" class=\"InternalRef\"\u003e2.3\u003c/span\u003e, a 3D model of the patient's heart, reconstructed from a CT scan \u003csup\u003e9,10,24\u003c/sup\u003e, is rendered using the Unity game engine. Furthermore, the catheter's spline obtained from the inferred K points, acquired in real-time by our proposed computer vision algorithm, are also rendered in the scene. For this mapping to be accurate, alignment of fiducial points on the 3D printed model with corresponding anatomical landmarks is essential.\u003c/p\u003e \u003cp\u003eThe communication between the output of the Python-based computer vision algorithm and the 3D rendering in Unity, is implemented by using the web-based library Flask, which is a common choice for efficient real-time data transfer. More specifically, inferred tracking data comprised of the position of each of the K points on the catheter, as well as the rotation of the tip is saved using a JSON file and then transmitted to Unity through a Flask-based WebSocket connection. The Flask server can be run on any IP address and port that the user specifies, with the default being localhost at port 5000. To connect this data to the Unity application, the user is prompted to enter this IP address and port when they launch the app and connect to the server using the SocketIO Unity package. After receiving the transmitted tracking data, Unity renders dynamic updates to both the catheter location and curvature in the 3D scene. Using the fiducial markers, an affine transformation is used to correctly map the catheter placement and movement with respect to the 3D model of the heart. The catheter rendering is superimposed within the patient-specific heart rendering and surrounding scene within the Quest 3 headset, creating a cohesive XR environment. This integrated platform offers an interactive experience that goes beyond mere visualization, allowing users to observe and analyze the catheter's movements in real time with quantitative feedback based on tracking relative to pre-defined target positions. The visual representation is contextualized within the reconstructed cardiac anatomy, providing a valuable tool for training simulations. Moreover, this innovative approach holds considerable potential for real-life applications in clinical settings, where the interactive XR platform could contribute to improved catheterization procedures through enhanced training and procedural guidance.\u003c/p\u003e \u003c/div\u003e"},{"header":"3. RESULTS AND DISCUSSION","content":"\u003cp\u003eReal-time pose detection of a catheter is a critical need for a XR system to perform accurate three-dimensional tracking to provide guidance and navigation for clinical procedures. Integrating this capability into a XR system can enhance medical training, allowing practitioners to refine dexterity, spatial awareness, and procedural skills. Moreover, the detailed 3D representation of the catheter aids in preoperative planning, enabling clinicians to strategize and optimize their approach, thereby reducing errors during actual interventions. This approach, complemented by the proposed K-points tracking algorithm (i.e., arbitrary number of pre-defined points tracked along the length of a catheter), was evaluated through comprehensive experiments using both a commercially available catheter and 3D-printed catheter phantoms characterized by predetermined angles and curvatures.\u003c/p\u003e \u003cp\u003eTo evaluate the system's adaptability to diverse commercial catheter types, we conducted initial tests with two specific catheters: the OSCOR Sheath and the PHILIPS Intracardiac Echocardiography (ICE) catheter. Samples of each catheter are depicted in Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003e. For a more thorough demonstration of the system's functionality with these catheters, additional videos can be found in the Supplementary materials (\u003cb\u003eSupplementary Videos Sa, and Sb\u003c/b\u003e). As another evaluation, our assessment encompassed a dual-focused approach, including the analysis of tip tracking accuracy with the OSCOR Sheath and a scrutiny of the algorithm's capacity for 3D pose reconstruction across the entire catheter length using a mock 3D-printed catheter. The outcomes of these experiments will yield valuable insights into the algorithm's potential application and performance in practical clinical scenarios, such as cardiac interventions. This, in turn, aims to enhance procedural precision, efficiency, and overall patient care by minimizing complications in catheter-based procedures.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003e3.1 Tip Localization Accuracy\u003c/h2\u003e \u003cp\u003eAs an initial test, the accuracy of localizing the catheter tip position was quantified. Five fiducial points (P1-P5 in Fig.\u0026nbsp;\u003cspan refid=\"Fig6\" class=\"InternalRef\"\u003e6\u003c/span\u003e) with known 3D coordinates were 3D printed onto the boundary of interest. The commercial catheter was maneuvered such that its tip touched each fiducial point, with the contact captured by the biplane cameras. The absolute difference of 3D coordinates between the reconstructed tip position (measured) from the computer vision algorithm and the actual fiducial coordinates was computed as shown in the table in Fig.\u0026nbsp;\u003cspan refid=\"Fig6\" class=\"InternalRef\"\u003e6\u003c/span\u003e-d. As you can see in this table, the average tip localization error is less than 1 mm in 3D coordinates. This level of accuracy provides confidence in the ability to precisely reconstruct the catheter's position.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec9\" class=\"Section2\"\u003e \u003ch2\u003e3.2 3D Pose Reconstruction of different Catheter Configurations\u003c/h2\u003e \u003cp\u003eTo evaluate the reconstruction of full catheter trajectories, experiments were conducted with 3D printed catheter phantoms designed with three different fixed angles: 30\u0026deg;, 45\u0026deg;, and 60\u0026deg; (Fig.\u0026nbsp;\u003cspan refid=\"Fig7\" class=\"InternalRef\"\u003e7\u003c/span\u003e-a). Each of the 3D printed and fixed angles catheter phantoms was positioned at four different locations within the workspace, aligned to the inlet hole. At each position, the catheter phantom was rotated manually at various speeds while both cameras captured video for processing. The K-points tracking algorithm processed the biplane videos, reconstructing the full 3D trajectory over time. Qualitative results depicting the reconstructed 3D trajectories across different angles and positions are shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig7\" class=\"InternalRef\"\u003e7\u003c/span\u003e-b-d for 30\u0026deg;, 45\u0026deg;, and 60\u0026deg; fixed angle catheters, respectively. For quantitative assessment, the catheter's shape was isolated from specific frames and the angles were calculated from the coordinates of each of the K-points. Results showed an average reconstruction error of ~\u0026thinsp;0.6\u0026deg;, ~\u0026thinsp;0.37\u0026deg;, and ~\u0026thinsp;0.1\u0026deg; for the 30\u0026deg;, 45\u0026deg;, and 60\u0026deg; catheters, respectively, across all k-points. Representative 2D projections of the reconstructed shapes are provided in Fig.\u0026nbsp;\u003cspan refid=\"Fig7\" class=\"InternalRef\"\u003e7\u003c/span\u003e-e-g. The low reconstruction error highlights the capability to reliably reconstruct diverse catheter configurations. In \u003cb\u003eSupplementary Videos Sc to Se\u003c/b\u003e, you can see several sample videos for this test.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eTo subject the system to a more challenging test, we fabricated three distinct catheters with arbitrary curves using 3D printing, as illustrated in Fig.\u0026nbsp;\u003cspan refid=\"Fig8\" class=\"InternalRef\"\u003e8\u003c/span\u003e-a. Each 3D-printed catheter was securely positioned at a fixed location within the workspace, aligning with the inlet hole. Subsequently, the catheter phantom underwent manual rotation at various speeds while cameras recorded footage for subsequent processing. The K-points tracking algorithm analyzed the biplane videos, reconstructing the complete 3D trajectory over time. Qualitative results, showcasing the reconstructed 3D trajectories across different curves. To validate the fidelity of the reconstructed shape against the designed curved catheter form, we extracted the 2D view of the 3D reconstructed trajectory. Simultaneously, we extracted the 2D view of the 3D designed curved catheters from SolidWorks, which was used to design the curved catheters. We manually superimposed the designed catheters onto the 2D view of the reconstructed data. As depicted in Figs.\u0026nbsp;\u003cspan refid=\"Fig8\" class=\"InternalRef\"\u003e8\u003c/span\u003e-b-d, a high degree of shape matching is evident, confirming that the reconstructed catheter shape precisely mirrors the design of the test catheter. In \u003cb\u003eSupplementary Videos Sf to Sh\u003c/b\u003e you can see several sample videos for this test.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec10\" class=\"Section2\"\u003e \u003ch2\u003e3.3 Extended Reality (XR) Results\u003c/h2\u003e \u003cp\u003eAs shown in our previous work, the Unity engine's rendering capabilities enabled the presentation of a patient-specific heart, overlaid with real-time tracking of the catheter position and shape\u003csup\u003e9\u003c/sup\u003e. This work improved upon that XR scene with several improvements: i) No embedded wires are required in the catheter for EM sensor tracking, making it compatible with various commercially available catheters or sheaths. In contrast, previous methods necessitated embedding at least three wires into the catheter, which proved to be expensive, time-consuming, and unfeasible for a variety of catheters, such as ICE catheters. ii) The catheter is rendered using the new spline function that is composed of k-points (instead of just 3 that were tracked using the EM sensors) improving the fidelity of the rendered catheter shape, iii) The software was made compatible with the Meta Quest 3 headset, instead of just the Hololens 2, and the scene was programmed to allow for 6 pre-defined targets within the heart, instead of just 3 that were hardcoded to recapitulate a transseptal puncture. As shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig9\" class=\"InternalRef\"\u003e9\u003c/span\u003e, the visualization includes a central 3D view of the heart, with additional 2D view on the left, as well as endoscopic view and a close-up target view on the right side of the scene. This interactive XR platform allowed users to explore catheter navigation in relation to the intricate cardiac anatomy reconstructed from medical scans. The immersive experience allows the user to move their head or walk around while the headset maintains the rendered scene in the exact position it was set, preventing the scene from blocking the view of the user to see anything else in the room (i.e., patient, other imaging). Additionally, using the voice command \u0026ldquo;Reset View,\u0026rdquo; the user can move the scene directly in front of their current head position, allowing for additional flexibility This XR system serves as a versatile tool for visualizing catheterization procedures, providing users with real-time feedback on the catheter's motion relative to pre-defined targets.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cdiv id=\"Sec11\" class=\"Section3\"\u003e \u003ch2\u003e3.3.1 Functionality and User Experience Assessment\u003c/h2\u003e \u003cp\u003eTo evaluate the functionality and user experience of the proposed XR system for guiding a percutaneous transcatheter procedure, we conducted an experimental study involving six participants. This study was carried out in accordance with the guidelines and regulations of the Institutional Review Board (IRB) at Weill Cornell Medicine, which gave approval for this study to be conducted. Participants were employees at Weill Cornell Medicine, but were not authors of this paper, and all provided informed consent for participation in the study. It should be noted that participants were not cardiologists or interventionalists with prior experience with delivery catheters. The primary objective was to assess both the quantitative performance metrics and qualitative feedback regarding the system's efficacy in navigating a catheter through predetermined targets within the heart using different visualization modes, allowing us to assess the significance of 3D depth perception in localizing a catheter. The experimental setup began with a structured familiarization period, allowing participants a maximum of 10 minutes to acquaint themselves with the XR system and the catheter. This period was crucial to ensure that participants had a basic understanding of the system's operation and functionality before engaging in the experimental sessions. Following the familiarization period, participants engaged in three distinct sessions, each designed to evaluate a specific visualization mode:\u003c/p\u003e \u003cp\u003e \u003cstrong\u003e2D-View\u003c/strong\u003e \u003cp\u003eIn this session, participants were presented with two-dimensional representations (Fig.\u0026nbsp;\u003cspan refid=\"Fig10\" class=\"InternalRef\"\u003e10\u003c/span\u003e\u003cb\u003e-Left\u003c/b\u003e) as would be seen in standard 2D monitors. The objective was to navigate the catheter to predetermined targets without any depth perception, such that mental coordination between multiple views is needed to triangulate a 3D position in space.\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003e2D3D-View\u003c/strong\u003e \u003cp\u003eIn this session, participants were presented with the same two-dimensional views as in Session 1, alongside a 3D mixed reality representation of the heart in the center of the display (Fig.\u0026nbsp;\u003cspan refid=\"Fig10\" class=\"InternalRef\"\u003e10\u003c/span\u003e\u003cb\u003e-Middle\u003c/b\u003e). This hybrid visualization mode aimed to provide participants with additional spatial context to aid in target navigation utilizing the immersive nature of mixed reality headsets.\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003e3D-View\u003c/strong\u003e \u003cp\u003eIn this session, participants solely engaged with the 3D representation of the heart (Fig.\u0026nbsp;\u003cspan refid=\"Fig10\" class=\"InternalRef\"\u003e10\u003c/span\u003e\u003cb\u003e-Right\u003c/b\u003e). This visualization mode aimed to simplify the attention of the user to only the 3D rendering to determine if the additional views from the 2D modes are beneficial, distracting, or inconsequential.\u003c/p\u003e \u003c/p\u003e \u003cp\u003eThe order of sessions was randomized to mitigate potential order effects and ensure the validity of the experimental findings. Throughout each session, meticulous records were kept, capturing the following key metrics:\u003c/p\u003e \u003cp\u003e \u003cb\u003et (\u003c/b\u003eTime\u003cb\u003e)\u003c/b\u003e: This metric represents the time duration (in seconds) taken by participants to complete the task within each session. It serves as a quantitative measure of procedural performance.\u003c/p\u003e \u003cp\u003e \u003cb\u003enT (\u003c/b\u003enumber of Targets\u003cb\u003e)\u003c/b\u003e: This metric denotes the number of predetermined targets successfully reached by participants during each session. It serves as a quantitative measure of procedural accuracy and success rate.\u003c/p\u003e \u003cp\u003e \u003cb\u003etT (\u003c/b\u003etime per Target\u003cb\u003e)\u003c/b\u003e: This metric calculates the average time taken by participants to reach each individual target within a session.\u003c/p\u003e \u003cp\u003e \u003cb\u003e%tT/2D (\u003c/b\u003epercentage of tT relative to tT of 2D-View\u003cb\u003e)\u003c/b\u003e: This metric compares the time taken by participants to reach targets in alternative visualization modes (2D3D-View and 3D-View) relative to the baseline 2D-View mode. It quantifies the efficiency gains achieved with the alternative visualization modes and serves as a comparative measure of procedural speed.\u003c/p\u003e \u003cp\u003eThe quantitative results of this experiment, as shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig11\" class=\"InternalRef\"\u003e11\u003c/span\u003e (see \u003cb\u003eTable S1\u003c/b\u003e for all results\u003cb\u003e)\u003c/b\u003e, underscore the significant advantages of a 3D-View for guiding catheterization procedures over 2D-View and even the 2D3D-View. The 3D-View consistently yielded the fastest task completion times and the highest number of successfully reached targets by all participants, irrespective of age group, indicating its clear performance superiority. Participants unanimously reported a superior experience with the 3D-View compared to other visual representations, a preference echoed across age groups. Despite varying levels of difficulty encountered, the 3D-View markedly facilitated more efficient performance compared to 2D and 2D3D views, as evidenced by time per target (tT) and percentage of tT relative to 2D-View session (%tT/2D). The %tT/2D metric illustrates that 2D3D-View and 3D-View allowed participants to reach targets approximately 3.1 (%tT/2D = %32.5) and 7.3 (%tT/2D = %13.6) times faster, respectively, than with 2D-View alone, signifying substantial speed and efficiency improvements with important implications for real-world interventional procedures. While 2D3D-View provided all information available in 3D-View, the additional 2D views appeared to distract from user performance (although not statistically significant with this low sample size), suggesting that the 3D-View alone is optimal for this procedure.\u003c/p\u003e \u003cp\u003eTo conduct a qualitative assessment, we have developed a survey comprising eight questions that explore various aspects of the participants' experiences. When asked to rate their experience on a scale of 1 to 5 (with 5 being the highest), the subjects unanimously agreed that the 3D view mode was significantly more helpful than the 2D view mode for improving the accuracy and speed of catheter placement (almost all subjects rated 5). This suggests that the 3D visualization provided by the XR system greatly enhanced the subjects' spatial comprehension and procedural skills during the simulated cardiac intervention. Furthermore, the subjects found the digital interface of the XR image-guidance system to be generally user-friendly (average rating of 4.2). They also expressed a high degree of confidence in the system's utility for both pre-operative training/planning and intra-operative procedures (average ratings of 5 and 4.5, respectively). The overall experience with the system was rated very positively, with an average score of 4.8 out of 5. When asked about their experience with video games, the subjects reported a wide range of experience levels, with an average rating of 2.7 out of 5. However, their experience with XR headsets was relatively low, with an average rating of 1.5 out of 5. These survey results demonstrate the potential of XR technologies to transform medical training and real-world procedures through enhanced visualization and interactivity. The subjects' feedback suggests that the developed XR image-guidance system could significantly improve spatial comprehension and procedural skills for complex cardiac interventions.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003c/div\u003e"},{"header":"4. CONCLUSION","content":"\u003cp\u003eThis paper presents a novel XR platform for real-time 3D catheter tracking and visualization during simulated percutaneous cardiac procedures. The proposed computer vision algorithm leverages biplane imaging to accurately reconstruct varied catheter configurations within a 3D printed setup. Integration with the Meta Quest 3 headset enabled immersive visualization of the tracked catheter trajectory of a reconstructed patient-specific heart model.\u003c/p\u003e \u003cp\u003eThe comprehensive experiments showcase precise tip localization of the catheter within approximately 1 mm of error, coupled with a high fidelity of shape reconstruction along its length. Our experimental study, involving six participants, revealed that the 3D visualization provided by the proposed XR system significantly outperformed 2D visualization, both in terms of speed and user experience. These findings strongly suggest that the XR system holds promise for enhancing catheterization training by bolstering spatial comprehension and procedural skills. Such results underscore the system's potential to revolutionize transcatheter cardiac procedures training through enhanced spatial comprehension. The XR environment offers an interactive and intuitive platform for honing the intricate psychomotor skills essential for catheter manipulation within patient anatomy.\u003c/p\u003e \u003cp\u003eThis study, however, has two primary limitations that should be further explored in future work. These two limitations are: i) \u003cem\u003eSmall sample size\u003c/em\u003e. Although the results show statistically significant differences, the sample size is still small and therefore a larger study should be performed to show generalization of the impact of the differences seen between three viewing modes. ii) \u003cem\u003eSubjects are non-cardiologists.\u003c/em\u003e All participants in this study were non-cardiologists or even interventionalists and therefore the conclusions can\u0026rsquo;t necessarily be generalized to the performance of trained individuals. However, this study does imply great impact for the learning curve of untrained individuals.\u003c/p\u003e \u003cp\u003eOverall, this innovative approach shows promise in transforming training and future percutaneous transcatheter cardiac procedures through the combined benefits of enhanced 3D visualization, immersive interactions, and patient-specific reconstruction. While the current results validate the technical feasibility of the approach, further research can help translate this technology to practical clinical deployment. Through progressive research to enhance the visualization, expanding the clinical integration, and validating the training outcomes, this XR catheter tracking platform aims to effectively prepare next-generation cardiologists to master this vital and complex procedural skill.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e \u003ch2\u003eCompeting Interests\u003c/h2\u003e \u003cp\u003eThe authors declare no competing interests.\u003c/p\u003e \u003c/p\u003e\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eMethodology and Machine vision algorithm design, M.A.; Coding and implementation of the algorithm, M.A.; Design and implementation of the 3D setup, M.A.; Testing and investigation, M.A.; Experimental study , and Data analysis, M.A.; Front-end development and coding, S.S.; Unity implementation and mixed reality, S.S.; Testing and investigation, S.S.; Code repository handling, S.J.; Testing and investigation, S.J.; Heart model and target design, A.C.; Resources, S.C.W.; Front-end project supervision, A.S.; Conceptualization, B.M.; Project administration, B.M.; Supervision, B.M.; Funding acquisition, B.M. All authors have read and agreed to the published version of the manuscript.\u003c/p\u003e\u003ch2\u003eAcknowledgments\u003c/h2\u003e \u003cp\u003eWe thank Philips for providing the VeriSight ICE catheter supported by a research grant.\u003c/p\u003e\u003ch2\u003eData Availability\u003c/h2\u003e\u003cp\u003eData is provided within the manuscript or supplementary information files.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eBashore, T. M. \u003cem\u003eet al.\u003c/em\u003e 2012 American College of Cardiology Foundation/Society for Cardiovascular Angiography and Interventions expert consensus document on cardiac catheterization laboratory standards update: A report of the American College of Cardiology Foundation Task Force on Expert Consensus documents developed in collaboration with the Society of Thoracic Surgeons and Society for Vascular Medicine. \u003cem\u003eJ Am Coll Cardiol\u003c/em\u003e 59, 2221\u0026ndash;2305 (2012). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org:10.1016/j.jacc.2012.02.010\u003c/span\u003e\u003cspan address=\"https://doi.org:10.1016/j.jacc.2012.02.010\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMalik, P. \u003cem\u003eGrossman\u0026rsquo;s Cardiac Catheterization, Angiography, and Intervention\u003c/em\u003e. 7 edn, Vol. 23 (2007).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGaba, P. \u003cem\u003eet al.\u003c/em\u003e Percutaneous Coronary Intervention vs Coronary Artery Bypass Graft Surgery for Left Main Disease in Patients With and Without Acute Coronary Syndromes: A Pooled Analysis of 4 Randomized Clinical Trials. JAMA Cardiology 8, 631\u0026ndash;639 (2023). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org:10.1001/jamacardio.2023.1177\u003c/span\u003e\u003cspan address=\"https://doi.org:10.1001/jamacardio.2023.1177\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eOkumus, N., Abraham, S., Puri, R. \u0026amp; Tang, W. H. W. Aortic Valve Disease, Transcatheter Aortic Valve Replacement, and the Heart Failure Patient. JACC: Heart Failure 11, 1070\u0026ndash;1083 (2023). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org:doi:10.1016/j.jchf.2023.07.003\u003c/span\u003e\u003cspan address=\"https://doi.org:doi:10.1016/j.jchf.2023.07.003\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBangalore, S. \u003cem\u003eet al.\u003c/em\u003e Evidence-Based Practices in the Cardiac Catheterization Laboratory: A Scientific Statement From the American Heart Association. Circulation 144, e107-e119 (2021). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org:10.1161/cir.0000000000000996\u003c/span\u003e\u003cspan address=\"https://doi.org:10.1161/cir.0000000000000996\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYoon, S.-H. \u003cem\u003eet al.\u003c/em\u003e Outcomes After Transcatheter Edge-to-Edge Mitral Valve Repair According to Mitral Regurgitation Etiology and Cardiac Remodeling. JACC: Cardiovascular Interventions 15, 1711\u0026ndash;1722 (2022). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org:doi:10.1016/j.jcin.2022.07.004\u003c/span\u003e\u003cspan address=\"https://doi.org:doi:10.1016/j.jcin.2022.07.004\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHolmvang, L., L\u0026uuml;scher, M. S., Clemmensen, P., Thygesen, K. \u0026amp; Grande, P. Very Early Risk Stratification Using Combined ECG and Biochemical Assessment in Patients With Unstable Coronary Artery Disease (A Thrombin Inhibition in Myocardial Ischemia [TRIM] Substudy). Circulation 98, 2004\u0026ndash;2009 (1998). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org:doi:10.1161/01.CIR.98.19.2004\u003c/span\u003e\u003cspan address=\"https://doi.org:doi:10.1161/01.CIR.98.19.2004\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSorajja, P., Michael, J. L. \u0026amp; Morton, J. K. \u003cem\u003eCardiac Catheterization Handbook E-Book\u003c/em\u003e. (Elsevier Health Sciences, 2021).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJang, S.-J. \u003cem\u003eet al.\u003c/em\u003e Development of a Hybrid Training Simulator for Structural Heart Disease Interventions. 2, 2000109 (2020). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org:https://doi.org/10.1002/aisy.202000109\u003c/span\u003e\u003cspan address=\"https://doi.org:10.1002/aisy.202000109\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLiu, J. \u003cem\u003eet al.\u003c/em\u003e An augmented reality system for image guidance of transcatheter procedures for structural heart disease. PloS one 14, e0219174 (2019). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org:10.1371/journal.pone.0219174\u003c/span\u003e\u003cspan address=\"https://doi.org:10.1371/journal.pone.0219174\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWu, D. \u003cem\u003eet al.\u003c/em\u003e Comparative Analysis of Interactive Modalities for Intuitive Endovascular Interventions. IEEE Transactions on Visualization \u0026amp; Computer Graphics, 1\u0026ndash;18 (5555). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org:10.1109/tvcg.2024.3362628\u003c/span\u003e\u003cspan address=\"https://doi.org:10.1109/tvcg.2024.3362628\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJoshi, A. \u0026amp; Wragg, A. Simulator Training in Interventional Cardiology. Interv Cardiol 11, 70\u0026ndash;73 (2016). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org:10.15420/icr.2016.11.1.70\u003c/span\u003e\u003cspan address=\"https://doi.org:10.15420/icr.2016.11.1.70\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMercuri, M., Sheth, T. \u0026amp; Natarajan, M. K. Radiation exposure from medical imaging: a silent harm? \u003cem\u003eCmaj\u003c/em\u003e 183, 413\u0026ndash;414 (2011). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org:10.1503/cmaj.101885\u003c/span\u003e\u003cspan address=\"https://doi.org:10.1503/cmaj.101885\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eStahl, C. M., Meisinger, Q. C., Andre, M. P., Kinney, T. B. \u0026amp; Newton, I. G. Radiation Risk to the Fluoroscopy Operator and Staff. AJR Am J Roentgenol 207, 737\u0026ndash;744 (2016). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org:10.2214/ajr.16.16555\u003c/span\u003e\u003cspan address=\"https://doi.org:10.2214/ajr.16.16555\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDe Ponti, R. \u003cem\u003eet al.\u003c/em\u003e Superiority of simulator-based training compared with conventional training methodologies in the performance of transseptal catheterization. J Am Coll Cardiol 58, 359\u0026ndash;363 (2011). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org:10.1016/j.jacc.2011.02.063\u003c/span\u003e\u003cspan address=\"https://doi.org:10.1016/j.jacc.2011.02.063\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eViglialoro, R. M. \u003cem\u003eet al.\u003c/em\u003e Augmented Reality, Mixed Reality, and Hybrid Approach in Healthcare Simulation: A Systematic Review. Applied Sciences 11, 2338 (2021).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGallagher, A., McClure, N., McGuigan, J., Crothers, I. \u0026amp; Browning, J. Virtual reality training in laparoscopic surgery: a preliminary assessment of minimally invasive surgical trainer virtual reality (MIST VR). Endoscopy 31, 310\u0026ndash;313 (1999).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLinte, C. A., Moore, J., Wiles, A. D., Wedlake, C. \u0026amp; Peters, T. M. Virtual reality-enhanced ultrasound guidance: a novel technique for intracardiac interventions. Comput Aided Surg 13, 82\u0026ndash;94 (2008). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org:10.3109/10929080801951160\u003c/span\u003e\u003cspan address=\"https://doi.org:10.3109/10929080801951160\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBarsom, E. Z., Graafland, M. \u0026amp; Schijven, M. P. Systematic review on the effectiveness of augmented reality applications in medical training. Surgical endoscopy 30, 4174\u0026ndash;4183 (2016).\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKlein, L. W. \u003cem\u003eet al.\u003c/em\u003e Occupational health hazards in the interventional laboratory: time for a safer environment. Radiology 250, 538\u0026ndash;544 (2009). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org:10.1148/radiol.2502082558\u003c/span\u003e\u003cspan address=\"https://doi.org:10.1148/radiol.2502082558\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhang, T. Y. \u0026amp; Suen, C. Y. A fast parallel algorithm for thinning digital patterns. Commun. ACM 27, 236\u0026ndash;239 (1984). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org:10.1145/357994.358023\u003c/span\u003e\u003cspan address=\"https://doi.org:10.1145/357994.358023\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLee, T. C., Kashyap, R. L. \u0026amp; Chu, C. N. Building Skeleton Models via 3-D Medial Surface Axis Thinning Algorithms. CVGIP: Graphical Models and Image Processing 56, 462\u0026ndash;478 (1994). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org:https://doi.org/10.1006/cgip.1994.1042\u003c/span\u003e\u003cspan address=\"https://doi.org:10.1006/cgip.1994.1042\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAnnabestani, M. \u0026amp; Naghavi, N. Non-uniform deformation and curvature identification of ionic polymer metal composite actuators. Journal of Intelligent Material Systems and Structures 26, 582\u0026ndash;598 (2014). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org:10.1177/1045389X14538535\u003c/span\u003e\u003cspan address=\"https://doi.org:10.1177/1045389X14538535\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTorabinia, M. \u003cem\u003eet al.\u003c/em\u003e Deep learning-driven catheter tracking from bi-plane X-ray fluoroscopy of 3D printed heart phantoms. Mini-invasive Surgery 5, 32 (2021). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org:10.20517/2574-1225.2021.63\u003c/span\u003e\u003cspan address=\"https://doi.org:10.20517/2574-1225.2021.63\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Percutaneous cardiac intervention, Extended Reality, Mixed Reality, Computer Vision, 3D visualization","lastPublishedDoi":"10.21203/rs.3.rs-4645065/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4645065/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eExtended reality (XR) technologies are emerging as promising platforms for medical training and procedural guidance, particularly in complex cardiac interventions. This paper presents a high-fidelity methodology to perform real-time 3D catheter tracking and visualization during simulated cardiac interventions. A custom 3D-printed setup with mounted cameras enables biplane video capture of a catheter. A computer vision algorithm processes the biplane images in real-time to reconstruct the 3D catheter trajectory represented by any designated number of points along its length. This method accurately localizes the catheter tip within 1 mm and can reconstruct any arbitrary catheter configuration. The tracked catheter data is integrated into an interactive Unity-based scene rendered on the Meta Quest 3 headset. The visualization seamlessly combines a reconstructed 3D patient-specific heart model with the dynamically tracked catheter, creating an immersive extended reality training environment. Our experimental study, involving six participants, demonstrated that the 3D visualization provided by the proposed XR system significantly outperformed 2D visualization in terms of speed and user experience. This suggests that the XR system has the potential to enhance catheterization training by improving spatial comprehension and procedural skills. The proposed system demonstrates the potential of XR technologies to transform percutaneous cardiac interventions through improved visualization and interactivity.\u003c/p\u003e","manuscriptTitle":"High-Fidelity Pose Estimation for Real-Time Extended Reality (XR) Visualization for Cardiac Catheterization","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-07-26 15:48:23","doi":"10.21203/rs.3.rs-4645065/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2024-08-26T06:03:26+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-08-07T21:01:58+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-08-05T03:15:59+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"199428006904845320639093029191259437070","date":"2024-07-28T11:45:03+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"42428953042516320303122685149602604921","date":"2024-07-16T05:26:50+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-07-06T14:17:09+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-07-06T14:15:31+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2024-07-02T07:12:53+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-07-02T07:09:53+00:00","index":"","fulltext":""},{"type":"submitted","content":"Scientific Reports","date":"2024-06-26T23:00:51+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"4af562ed-5f1e-4088-8e91-05ca6ee2c92f","owner":[],"postedDate":"July 26th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":34827601,"name":"Physical sciences/Engineering/Biomedical engineering"},{"id":34827602,"name":"Health sciences/Cardiology/Interventional cardiology"}],"tags":[],"updatedAt":"2024-11-11T15:58:34+00:00","versionOfRecord":{"articleIdentity":"rs-4645065","link":"https://doi.org/10.1038/s41598-024-76384-z","journal":{"identity":"scientific-reports","isVorOnly":false,"title":"Scientific Reports"},"publishedOn":"2024-11-06 15:56:51","publishedOnDateReadable":"November 6th, 2024"},"versionCreatedAt":"2024-07-26 15:48:23","video":"","vorDoi":"10.1038/s41598-024-76384-z","vorDoiUrl":"https://doi.org/10.1038/s41598-024-76384-z","workflowStages":[]},"version":"v1","identity":"rs-4645065","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4645065","identity":"rs-4645065","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}