Geometric hand-eye calibration for an endoscopic neurosurgery system

Aiming at the problem of low accuracy of industrial robot hand-eye calibration, a robot hand-eye calibration method based on an intermediate measurement system is proposed. The checkerboard calibration board with the marker points pasted is photographed by the camera, and the coordinates of the marker points in the checkerboard calibration board coordinate system are calculated according to the principle of cross-ratio invariance, and the coordinate system matrix between the C-Track optical dynamic tracking system and the industrial robot is established. The transformation relationship finally obtains the closed-loop coordinate system matrix transformation equation with the marker point, the checkerboard calibration board and the C-Track optical dynamic tracking system as the intermediate measurement system, thereby avoiding the introduction of the mobile industrial robot manipulator in the traditional hand-eye calibration process Kinematic error. The experimental results show that the translation error in the hand-eye relationship matrix is 0.6 mm, and the rotation error is 0.3°. Compared with the traditional hand-eye calibration method, this method improves the accuracy of hand-eye calibration, simplifies the hand-eye calibration process, and meets the needs of actual processing and production.

This project aimed to create an immersive digital twin laboratory that incorporates advanced tracking and visualization capabilities. In collaboration with Fort Lewis College, NREL designed a state-of-the-art physical visualization laboratory, developed a software platform to enable interaction with tracked physical objects in the laboratory, and provided proof-of-concept curricula that included manipulating these tracked objects. The project was initiated to address the growing need for innovative educational tools in engineering education. As renewable energy systems, particularly solar installations, become more complex, there is a pressing need to bridge the gap between theoretical knowledge and practical application. Traditional methods of teaching solar engineering concepts often fall short of providing students with a comprehensive, hands-on understanding. This immersive digital twin laboratory was conceived to fill that gap by creating a safe, non-energized setting where students can interact with augmented solar installation objects, gaining valuable insights into system performance, design, and maintenance. The project utilized extended reality (XR) technologies, including head-mounted displays (HMDs) and a whole-room optical motion tracking system, to connect physical objects with their digital twins in real time. The laboratory was equipped with MagicLeap 2 HMDs, supported by a Vicon Vero 2.2 Optical Tracking System, which provided precise 6-degrees-of-freedom (6-DOF) tracking. We developed a software platform to manage the interaction between the tracked physical objects and their virtual counterparts, enabling real-time data synchronization, object recognition, and virtual overlays. We designed the system to be flexible and extendable, allowing for future integration of additional objects and curriculum. This research advances the field of engineering education by demonstrating the potential of immersive digital twin environments. The laboratory provides a dynamic learning space where students can experiment, collaborate, and learn without the risks associated with live experimentation. The ability to simulate and manipulate solar installation objects under various conditions has broad implications for workforce development, particularly in renewable energy. The project also highlights the economic feasibility of using XR technologies in educational settings, offering a cost-effective solution for institutions looking to enhance their curriculum. By fostering a deeper understanding of solar energy systems, this work contributes to the broader goal of supporting the global energy transition and preparing the next generation of engineers and technicians.

Purpose: The goal of this work is development of an endoscopic system for three-dimensional localization of tissue surfaces during image-guided surgery. The system has the potential to track tissue movement in real time and to measure the shape of organ surfaces within the endoscopes' fields of view via localization of thousands of surface points. The system is intended to provide real-time localization of surface points as a means to guide coregistration with preoperative anatomic image sets and to provide the constraints required by deformable tissue models. Method and Materials: A benchtop prototype of the system consisting of a pair of conventional endoscopes, a computer-controlled laser scanner, a high-speed CCD camera, and a commercial optical tracking system has been constructed in Creare's laboratory. Calibration of the prototype has been performed by verifying the accuracy of the optical tracking system used to measure the position of the endoscopes, measuring the repeatability of the laser scanner when projecting the laser through an endoscope, characterization of the distortion present in the endoscope optics, and the measurement of the maximum usable field-of-view. The accuracy of the system has been determined based on the difference between three-dimensional coordinates measured by our system with those measured by the commercial optical tracking system. Results: The repeatability of the system to consistently localize a fixed point was determined by measuring the location of 25 points while maintaining fixed positions of the endoscopes and laser mirrors. The mean three-dimensional repeatability of the localizations was 0.08 mm with a standard deviation of 0.05 mm. The accuracy of the system was ascertained relative to measurements made with the optical tracking system and a standard six LED probe, known to have an accuracy on the order of 0.1 mm. Using a planar test phantom with machined semicircular holes, the projecting endoscope laser was steered to the center of each hole, and the position of the illuminated spot determined using the system. The position of each hole was also measured with the six LED probe, and the resulting three-dimensional error found to be 1.9 mm with a standard deviation of 0.3 mm. The system can localize approximately 400 points per second. Conclusion: The use of a scanning laser endoscope has the potential to provide cost-effective, real-time localization of any point within the surgical field that can be simultaneously observed by two endoscopes. The initial results of our work show that points can be localized accurately and rapidly. Educational Objectives: 1. Calibration techniques for endoscopic laser projection. 2. Discussion of endoscopic localization as a means of assisting image- guided surgery.

Purpose: To evaluate the immobilization accuracy of a novel vacuum bite block re-locatable head frame (RHF) system in patients receiving intracranial stereotactic radiation therapy (SRT) by means of an infrared optical tracking system (OTS). Methods and Materials: The RHF was designed for multi-session treatments on the Leksell GammaKnife Perfexion™. In the present REB approved study, the RHF was used for patients undergoing linac-based SRT. A passive optical marker was placed on the bite block as a surrogate for patient motion. The OTS monitored patients continuously throughout each fraction. For each fraction, cone-beam computed tomography (CBCT) was used to verify the patient position at three time points: after the initial setup, pre-delivery following the automatic couch adjustments and post-delivery. To investigate the reliability of the optical measurements, the OTS measurements were compared against the implemented couch shifts for patient setup. Intra-fractional anatomical motion, as measured from the second and third CBCT scans, was compared with the difference in OTS marker positions. Real-time OTS data throughout each fraction was assessed to determine if patients moved by more than 1 mm during treatments. Results: Currently, data from three patients have been analyzed. The couch motion measured from the OTS strongly correlated with the actual reported couch shift (Pearson correlation coefficient of 0.99). Mean differences between OTS and true couch motion were 0.01, 0.0, and 0.04 mm in the lateral, anterior-posterior, and superior-inferior directions, respectively. Mean differences between OTS and CBCT reported motion were 0.37, 0.07, and 0.01 mm respectively with a mean vector magnitude of 0.41 mm. The real-time positional data indicated that there were few instances (3) of patient motion greater than 1mm. Conclusion: The OTS is a reliable tool for evaluation of intra-fraction patient motion. The results of this study indicate that the RHF provides sufficient immobilization accuracy for SRT.

Augmented reality (AR) allows the surgeon to represent holographic patient-specific anatomical information and surgical instruments in the physical world. To correctly superimpose virtual and physical objects, a hand-eye (HE) calibration method for mapping the virtual and physical spaces was proposed. Mathematical relationships between the virtual camera and the physical space were derived. Finally, the accuracy and robustness of the proposed HE calibration method were qualitatively and quantitatively evaluated. The proposed calibration method allows us to determine an optimal invariant spatiotemporal mapping between the virtual camera and the physical space. Qualitatively and quantitatively reliable and accurate estimates for the physical-virtual mapping transformation were verified. Consequently, imaging data and surgical instruments holograms can be precisely represented in the physical space.

Purpose: The feasibility of total marrow irradiation (TMI) using highly conformal image‐guided Tomotherapy is recently being explored as a radiation dose escalation pre‐conditioning TMI regimen for high risk hematological malignancies. Though, initial trial shows potential of this technology, treatment procedure is at present an open loop. This is a concern for safety as well as accuracy of the treatment delivery. The primary objective of this study is to develop a real time optical tracking system (OTS) which will offer better understanding of intra‐fraction whole body motion and develop a closed loop feedback approach to treatment that will enhance the safety of this new treatment.Method and Materials: A stereoscopic camerasystem is utilized to track near‐infrared reflective markers placed on the Rando phantom. The cameras were calibrated to determine the 3D location of each detected marker. The entire stereoscopic marker detection system returns a set of 3D marker locations at a rate of 60 Hz. Five criteria for evaluating motion capture technology are considered for selection of the optical detection system — accuracy, bandwidth, frame rate, capture volume, and real time performance. The translational motion detection uncertainties in OTS are measured and compared with onboard three dimensional megavoltage CT (MVCT) scanning system for pre‐treatment setup verification. Results: The preliminary work has shown : (i) the translational motion with better than 4mm accuracy, (ii) the vision‐based acquisition can be performed very quickly (in second) compared to the MVCT scan (5–10 minutes) method currently used for initial patient positioning; (iii) bony portions of the body can be accurately tracked in three dimensions. Conclusion: The proposed method entices the effectiveness of vision‐based whole body patient motion tracking for use with the helical tomotherapy treatment.

Body sensor calibration and construction of 3D maps for robot navigation using the framework of conformal geometric algebra

Optical motion tracking systems are widely employed in virtual- and mixed reality applications enabling users to directly interact with 3D space. Important efforts have been made to make such technologies available to a larger group of users by significantly reducing their deployment costs. In this poster we introduce the result of a German-Brazilian cooperation: The first commercial low-cost marker-based optical tracking system developed in South-America. Despite using cheaper technologies and being low-cost which usually imply worse performance and accuracy, we developed a system with sufficient accuracy for professional use. There have been significant research efforts to develop cheaper tracking devices to provide interaction capabilities in virutal- and mixed reality. The Augmented Reality Tool Kit (ARToolkit) [1] is a good example of a very popular low-cost academic optical tracker working with any type of webcam. As a non-academic example of affordable tracking systems, Nintendo's latest video game console (Nintendo Wii) comes with the Wiimote interaction device containing a built-in motion tracker that allows for immersive interaction metaphors in home environments. However, most such available low-cost solutions lack in suitability for professional applications due to their usually insufficient accuracy, tracking range or latency time. The starting point of this project was PTrack [2], an academic marker-based single-camera tracker developed at Fraunhofer IGD. PTrack development has shown that an accurate optical tracking system can be built from components that are affordable for a much wider group of users, from the academic AR/VR community to companies’ developers. The performance and accuracy of PTrack were assessed through a comprehensive set of experiments which have proved the suitability of the system for professional applications. One of the main topics raised by PTrack as further research was an extension of the system into a multiple stereo camera configuration, which led to the present work.

Online viewpoint optimization method for high-accuracy intraoperative optical measurement in orthopedic surgeries: An active navigation approach

This paper presents a novel graphical user interface developed for a navigation system for ultrasound-guided computer-assisted shoulder arthroscopic surgery. The envisioned purpose of the interface is to assist the surgeon in determining the position and orientation of the arthroscopic camera and other surgical tools within the anatomy of the patient. The user interface features real time position tracking of the arthroscopic instruments with an optical tracking system, and visualization of their graphical representations relative to a three-dimensional shoulder surface model of the patient, created from computed tomography images. In addition, the developed graphical interface facilitates fast and user-friendly intra-operative calibration of the arthroscope and the arthroscopic burr, capture and segmentation of ultrasound images, and intra-operative registration. A pilot study simulating the computer-aided shoulder arthroscopic procedure on a shoulder phantom demonstrated the speed, efficiency and ease-of-use of the system.

With image-guided tomotherapy, highly targeted total marrow irradiation (TMI) has become a feasible alternative to conventional total body irradiation. The uncertainties in patient localization and intrafraction motion of the whole body during hour-long TMI treatment may pose a risk to the safety and accuracy of targeted radiation treatment. The feasibility of near-infrared markers and optical tracking system (OTS) is accessed along with a megavoltage scanning system of tomotherapy. Three near-infrared markers placed on the face of a rando phantom are used to evaluate the capability of OTS in measuring changes in the markers' positions as the rando is moved in the translational direction. The OTS is also employed to determine breathing motion related changes in the position of 16 markers placed on the chest surface of human volunteers. The maximum uncertainty in locating marker position with the OTS is 1.5 mm. In the case of normal and deep breathing motion, the maximum marker position change is observed in anterior-posterior direction with the respective values of 4 and 12 mm. The OTS is able to measure surface changes due to breathing motion. The OTS may be optimized to monitor whole body motion during TMI to increase the accuracy of treatment delivery and reduce the radiation dose to the lungs.

BackgroundThe pendulum test is a quantitative method used to assess knee extensor spasticity in humans with spinal cord injury (SCI). Yet, the clinical implementation of this method remains limited. The goal of our study was to develop an objective and portable system to assess knee extensor spasticity during the pendulum test using inertial measurement units (IMU).MethodsSpasticity was quantified by measuring the first swing angle (FSA) using a 3-dimensional optical tracking system (with external markers over the iliotibial band, lateral knee epicondyle, and lateral malleolus) and two wireless IMUs (positioned over the iliotibial band and mid-part of the lower leg) as well as a clinical exam (Modified Ashworth Scale, MAS).ResultsMeasurements were taken on separate days to assess test–retest reliability and device agreement in humans with and without SCI. We found no differences between FSA values obtained with the optical tracking system and the IMU-based system in control subjects and individuals with SCI. FSA values from the IMU-based system showed excellent agreement with the optical tracking system in individuals with SCI (ICC > 0.98) and good agreement in controls (ICC > 0.82), excellent test–retest reliability across days in SCI (ICC = 0.93) and good in controls (ICC = 0.87). Notably, FSA values measured by both systems showed a strong association with MAS scores (ρ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\uprho$$\\end{document} ~ −0.8) being decreased in individuals with SCI with higher MAS scores, reflecting the presence of spasticity.ConclusionsThese findings suggest that our new portable IMU-based system provides a robust and flexible alternative to a camera-based optical tracking system to quantify knee extensor spasticity following SCI.

Mobile applications tend to ask for users’ location in order to improve the service they provide. However, aside from increasing their service utility, they may also store these data, analyze them or share them with external parties. These privacy threats for users are a hot topic of research, leading to the development of so called Location Privacy Protection Mechanisms. LPPMs often are configurable algorithms that enable the tuning of the privacy protection they provide and thus the leveraging of the service utility. However, they usually do not provide ways to measure the achieved privacy in practice for all users of mobile devices, and even less clues on how a given configuration will impact privacy of the data given the specificities of everyone’s mobility. Moreover, as most Location Based Services require the user position in real time, these measures and predictions should be achieved in real time. In this paper we present a metric to evaluate privacy of obfuscated data based on users’ points of interest as well as a predictive model of the impact of a LPPM on these measure; both working in a real time fashion. The evaluation of the paper’s contributions is done using the state of the art LPPM Geo-I on synthetic mobility data generated to be representative of real-life users’ movements. Results highlight the relevance of the metric to capture privacy, the fitting of the model to experimental data, and the feasibility of the on-line mechanisms due to their low computing complexity.

Computer-aided surgery systems provide visual guidance to surgeons by showing the real-time pose of surgical instruments overlaid on preoperative medical images of the patient. Surgical instrument poses are localized in space using mainly optical tracking systems (OTS) and electromagnetic tracking systems (EMTS). OTS systems require clear line-of-sight, which is difficult to ensure in current overcrowded operating rooms. On the other hand, EMTSs provide less accuracy and suffer from magnetic field distortion in the presence of metal objects. In this paper, we propose a sensor fusion algorithm to compensate for the drawbacks of OTS and EMTS, and achieve robust tracking of surgical instruments. Spatial alignment of OTS and EMTS data will be achieved through a calibration procedure. The proposed sensor fusion algorithm uses an unscented Kalman filter (UKF), an extension of the standard Kalman filter based on a deterministic sampling of nonlinear functions. Quaternion representation for rotations is used to avoid singularities of other parameterizations (e.g., Euler angles). In cases of optical marker occlusion, our algorithm will take advantage of the EMTS to estimate the position of the hidden marker(s) and feed it in the UKF. The fusion algorithm will also keep track of an error matrix between OTS and EMTS measured poses, providing an up-to-date estimate of the electromagnetic distortion. This will be used to correct EMTS measurement before using it to compensate for possible optical line-of-sight occlusions. The proposed sensor fusion (SF) method is compared to a predictive UKF based on optical data (NSF). Our results show that the SF effectively compensates for short marker occlusion (nine samples) providing a continuous estimate of the instrument pose with an error significantly lower than the NSF method, and reaching a clinically acceptable accuracy. Furthermore, the proposed algorithm increases the accuracy of EMTS in the presence of magnetic field distortion.

Near infrared optical tracking system (NIOTS) is widely used in biomedical application, especially in surgical navigation. It largely determines the accuracy of tracking surgical tools. So we design an NIOTS and propose the corresponding algorithms to make it work well. Two near infrared cameras (NICs) are constructed in this NIOTS and a near infrared calibration board is designed to calibrate them. The NIOTS is then calibrated using the calibration results of NICs. After the image points of markers on surgical tools are obtained, a two-step stereo matching algorithm is proposed to match the subpixel points in the left and right images. Then the three dimensional coordinates of markers are computed and the surgical tools are tracked in real time. The experimental results show the accuracy of system calibration, stereo matching, and tracking. The average tracking errors of three markers in 150 positions are 0.0090 mm, 0.0071 mm, and 0.0060 mm, which demonstrate the designed NIOTS can accurately track surgical tools.