Learning to Anchor Visual Odometry: KAN-Based Pose Regression for Planetary Landing

<abstract><p>For the autonomous and intelligent operation of robots in unknown environments, simultaneous localization and mapping (SLAM) is essential. Since the proposal of visual odometry, the use of visual odometry in the mapping process has greatly advanced the development of pure visual SLAM techniques. However, the main challenges in current monocular odometry algorithms are the poor generalization of traditional methods and the low interpretability of deep learning-based methods. This paper presented a hybrid self-supervised visual monocular odometry framework that combined geometric principles and multi-frame temporal information. Moreover, a post-odometry optimization module was proposed. By using image synthesis techniques to insert synthetic views between the two frames undergoing pose estimation, more accurate inter-frame pose estimation was achieved. Compared to other public monocular algorithms, the proposed approach showed reduced average errors in various scene sequences, with a translation error of $ 2.211\% $ and a rotation error of $ 0.418\; ^{\circ}/100m $. With the help of the proposed optimizer, the precision of the odometry algorithm was further improved, with a relative decrease of approximately 10$ \% $ intranslation error and 15$ \% $ in rotation error.</p></abstract>

Visual odometry (VO) plays a vital role in Simultaneous Localization and Mapping (SLAM). Most of the current learning-based VO methods utilize Convolutional Neural Network (CNN) as framework. However, CNN is weak in integrating global context information. In the design of loss function, the majority of these methods ignore the restraint relationship between translation and rotation prediction. In this work we propose an end-to-end global-context-aware visual odometry scheme with Epipolar-Geometry-constrained loss function to estimate the relative 6-DoF poses of monocular camera. The proposed scheme designs an augmented-attention-enhanced global context block on top of contextual CNN to learn the long-range dependencies and internal correlation. To overcome the problem of mutual restraint between the translation and rotation errors, an Epipolar-Geometry-constrained loss function is developed to simultaneously improve the prediction accuracy of both translation and rotation. The evaluation results on public datasets and self-collected dataset show that the proposed system outperforms state-of-the-art learning-based VO methods with a large margin.

We introduce a novel approach for bronchoscopic navigation that leverages neural radiance fields (NeRF) to passively locate the endoscope solely from bronchoscopic images. This approach aims to overcome the limitations and challenges of current bronchoscopic navigation tools that rely on external infrastructures or require active adjustment of the bronchoscope. To address the challenges, we leverage NeRF for bronchoscopic navigation, enabling passive endoscope localization from bronchoscopic images. We develop a two-stage pipeline: offline training using preoperative data and online passive pose estimation during surgery. To enhance performance, we employ Anderson acceleration and incorporate semantic appearance transfer to deal with the sim-to-real gap between training and inference stages. We assessed the viability of our approach by conducting tests on virtual bronchscopic images and a physical phantom against the SLAM-based methods. The average rotation error in our virtual dataset is about 3.18 and the translation error is around 4.95 mm. On the physical phantom test, the average rotation and translation error are approximately 5.14 and 13.12 mm. Our NeRF-based bronchoscopic navigation method eliminates reliance on external infrastructures and active adjustments, offering promising advancements in bronchoscopic navigation. Experimental validation on simulation and real-world phantom models demonstrates its efficacy in addressing challenges like low texture and challenging lighting conditions.

In surgical navigation-assisted pedicle screw fixation, cross-source pre- and intra-operative point clouds registration faces challenges like significant initial pose differences and low overlapping ratio. Classical algorithms based on feature descriptor have high computational complexity and are less robust to noise, leading to a decrease in accuracy and navigation performance. To address these problems, this paper proposes a coarse-to-fine registration framework. In the coarse registration stage, a Point Matching algorithm based on Curvature Feature Learning (CFL-PM) is proposed. Through CFL-PM and Farthest Point Sampling (FPS), the coarse registration of overlapping regions between the two point clouds is achieved. In the fine registration stage, the Iterative Closest Point (ICP) is used for further optimization. The proposed method effectively addresses the challenges of noise, initial pose and low overlapping ratio. In noise-free point cloud registration experiments, the average rotation and translation errors reached 0.34° and 0.27 mm. Under noisy conditions, the average rotation error of the coarse registration is 7.28°, and the average translation error is 9.08 mm. Experiments on pre- and intra-operative point cloud datasets demonstrate the proposed algorithm outperforms the compared algorithms in registration accuracy, speed, and robustness. Therefore, the proposed method can achieve the precise alignment of the surgical navigation-assisted pedicle screw fixation.

Objective To evaluate the effect of setup errors on the 2D image projection and image registration, and then propose an improved registration method based on mutual information. Methods An anthropomorphic head phantom was used to simulate the rotational and translational setup errors. The geometric disparities were reflected by the changes of mutual information. Known setup errors were intentionally introduced to twenty cases divided into two groups demarcated by 3 mm translation error and 3° rotation error: ten cases with larger errors and ten with smaller errors. Then the anterior-posterior and lateral portal images were captured by the electronic portal imaging device (EPID), based on which the setup errors were calculated using two mutual information registration method respectively: the vender provided one, and the improved method as proposed. The calculated errors were compared with the actual setup errors to evaluate robustness of the method. Results For the ten cases with smaller setup errors, the average translational registration disparities using the conventional method were 0.3, 0.4, and 0.3 mm in x, y and z directions respectively. The rotational disagreements were 0.4° in both x and z directions. The average time consumption was 28.7 s. The corresponding discrepancies analyzed using the improved method were 0.3, 0.4, 0.3 mm, 0.5° and 0.4°, respectively. On average, 31.1 s was needed for registration. For the ten cases with larger setup errors, the mean disparities of the conventional method were 0.9, 0.7, 0.8 mm, 0.9° and 0.8°, 29.9 s taken on average. The corresponding result of the improved method was 0.5, 0.4, 0.5 mm, 0.6° and 0.5°, 33.2 s taken on average. Conclusions Regarding smaller setup errors, the two methods showed little difference and both had good performance in image registration accuracy. For larger setup errors, however, the improved mutual information registration method exhibited significantly higher accuracy than the conventional method, at cost of clinically acceptable registration time. Key words: Setup errors; Electronic portal imaging device; Portal image; Image registration

The aim of this study is to calculate setup margin based on six-dimensional (6D) corrected residual positional errors from kV cone beam computed tomography (CBCT) and from intrafraction projection kV imaging in coplanar and in noncoplanar couch positions in stereotactic radiotherapy. Six dimensional positional corrections were carried out before patient treatments, using a robotic couch and CBCT matching. A CBCT and stereoscopic ExacTrac image were acquired post-table position correction. Further, a series of intrafraction ExacTrac images were obtained for the variable couch position. Translational and rotational errors were identified as lateral (X), longitudinal (Y), vertical (Z); roll (Ɵ°), pitch (Φ°) and yaw (Ψ°). A total of 699 intrafraction image sets (361 coplanar and 338 noncoplanar) for 51 SRS/SRT patients were analysed. Rotational errors were corrected in terms of translational coordinates. Residual set-up margins were calculated from CBCT shifts. ExacTrac shifts give residual+intrafraction setup margins as a function of coplanar and noncoplanar couchpositions. The average residual positional error obtained from CBCT in X, Y, Z, Ɵ, Φ, Ψ were 0.1±0.4mm, 0.0±0.6mm,0.0±0.5mm, 0.2±0.8°, 0.1±0.6° and -0.1±0.7° respectively. For ExacTrac, the shits were -0.5±0.9mm, -0.0±1mm, -0.6±1.0mm, 0.4±0.9°, -0.2±0.6°, and -0.0±0.8°. CBCT calculated linear setup margins in X, Y, Z direction were 0.5, 1.2, and 1mm respectively. ExacTrac yielded coplanar and noncoplanar linear setup margins were 1.2, 1.3, 1.5, 1.4, 1.5, and 2.1mm respectively. CBCT-based gross residual set-up margin is equal to 1mm. ExacTrac calculated residual plus intrafraction setup margin falls within a 2mm range; attributed to intrafraction patient movement, table position inaccuracies, and poor image fusion in noncoplanar geometry. There could be variations in the required additional margin between centers and between machines, which require further studies.

BackgroundDry eye syndrome has been recently reported in patients who underwent whole brain radiotherapy (WBRT). WBRT based on a couch with three‐degrees of freedom (3D) can occasionally be performed in which the rotational head motion is not corrected. This study assessed the dependency of the rotational errors on the mask and the dose variation of the lens and lacrimal gland in WBRT patients.MethodsTranslational and rotational setup errors were obtained at the first treatment with cone‐beam CT (CBCT) for patients under WBRT and frameless stereotactic radiosurgery (SRS) (n = 20 each) immobilized using a conventional WB mask and an SRS mask with a bite block, respectively. For the CT sets of SRS cases, WBRT plans were generated for the study. To simulate the rotational error, rotated CT images were created with each rotational error, on which initial WBRT plans were copied and doses were recalculated. The lens and lacrimal gland doses with and without rotation errors were compared.ResultsDespite similar translational setup errors for the two masks, the SRS mask showed a dramatic reduction in rotational errors compared to those of the WB mask. The errors varied within −2.9° to 2.9° and −1.2° to 0.7° for the WB and SRS masks, respectively. Accordingly, the SRS mask confined the change in the maximum lens dose, mean dose of the lacrimal gland, and lacrimal volume receiving 15 Gy to one‐third of those using the WB mask.ConclusionWhen the six‐degrees of freedom (6D) couch is not available, the frameless SRS mask is beneficial to WBRT for the faithful treatment as it was planned.

In recent years, SLAM, visual odometry, and structure-from-motion approaches have widely addressed the problems of 3D reconstruction and ego-motion estimation. Of the many input modalities that can be used to solve these ill-posed problems, the pure visual alternative using a single monocular RGB camera has attracted the attention of multiple researchers due to its low cost and widespread availability in handheld devices. One of the best proposals currently available is the Direct Sparse Odometry (DSO) system, which has demonstrated the ability to accurately recover trajectories and depth maps using monocular sequences as the only source of information. Given the impressive advances in single-image depth estimation using neural networks, this work proposes an extension of the DSO system, named DeepDSO. DeepDSO effectively integrates the state-of-the-art NeW CRF neural network as a depth estimation module, providing depth prior information for each candidate point. This reduces the point search interval over the epipolar line. This integration improves the DSO algorithm’s depth point initialization and allows each proposed point to converge faster to its true depth. Experimentation carried out in the TUM-Mono dataset demonstrated that adding the neural network depth estimation module to the DSO pipeline significantly reduced rotation, translation, scale, start-segment alignment, end-segment alignment, and RMSE errors.

In this paper, a method to achieve visual odometry by two RGB-D cameras is proposed. The two cameras are placed to face in opposite directions. We adopt a novel method to obtain the position and attitude relationship between the front and rear camera coordinate systems, which uses a motion capture system and 2-dimensional bar code. Additionally, for the data observed by different cameras, a tightly coupled scheme is adopted for data fusion, and the feasibility of the algorithm is verified. At last, we achieve visual odometry by two non-overlapping RGB-D cameras. The average error is near 2.5 mm for the entire recorded image dataset while the average displacement is 40.61 mm. The result shows that the designed visual odometry is more robust and accurate than the visual odometry of a single camera.

If 3D rigid motion is estimated with some error a distorted version of the scene structure will in turn be computed. Of computational interest are these regions in space where the distortions are such that the depths become negative, because in order to be visible the scene has to lie in front of the image. The stability analysis for the structure-from-motion problem presented in this paper investigates the optimal relationship between the errors in the estimated translational and rotational parameters of a rigid motion, that results in the estimation of a minimum number of negative depth values. The input used is the value of the flow along some direction, which is more general than optic flow or correspondence. For a planar retina it is shown that the optimal configuration is achieved when the projections of the translational and rotational errors on the image plane are perpendicular. Furthermore, the projection of the actual and the estimated translation lie on a line passing through the image center. For a spherical retina given a rotational error, the optimal translation is the correct one, while given a translational error. The optimal rotational error is normal to the translational one at an equal distance from the real and estimated translations. The proofs, besides illuminating the confounding of translation and rotation in structure from motion, have an important application to ecological optics, explaining differences of planar and spherical eye or camera designs in motion and shape estimation.

If 3D rigid motion can be correctly estimated from image sequences, the structure of the scene can be correctly derived using the equations for image formation. However, an error in the estimation of 3D motion will result in the computation of a distorted version of the scene structure. Of computational interest are these regions in space where the distortions are such that the depths become negative, because in order for the scene to be visible it has to lie in front of the image, and thus the corresponding depth estimates have to be positive. The stability analysis for the structure from motion problem presented in this paper investigates the optimal relationship between the errors in the estimated translational and rotational parameters of a rigid motion that results in the estimation of a minimum number of negative depth values. The input used is the value of the flow along some direction, which is more general than optic flow or correspondence. For a planar retina it is shown that the optimal configuration is achieved when the projections of the translational and rotational errors on the image plane are perpendicular. Furthermore, the projection of the actual and the estimated translation lie on a line through the center. For a spherical retina, given a rotational error, the optimal translation is the correct one; given a translational error, the optimal rotational negative deptherror depends both in direction and value on the actual and estimated translation as well as the scene in view. The proofs, besides illuminating the confounding of translation and rotation in structure from motion, have an important application to ecological optics. The same analysis provides a computational explanation of why it is easier to estimate self-motion in the case of a spherical retina and why shape can be estimated easily in the case of a planar retina, thus suggesting that nature‘s design of compound eyes (or panoramic vision) for flying systems and camera-type eyes for primates (and other systems that perform manipulation) is optimal.

Real-time 3D optical tracking of free-hand imaging devices or surgical tools has been studied and employed for object localization in many minimally invasive interventions. However, the surgical workspace for many interventional procedures is often sub-dermal with tool access through ports from surgical incisions or anatomical orifices. To maintain the optical line-of-sight criterion, external extensions of inserted imaging devices and rigid surgical tools must be tracked to localize the internal tool tips. Unfortunately, tracking by this form of correspondence is very susceptible to noise as orientation errors on the external tracked end compound into both rotational and translational errors on the internal, workspace position. These translational errors are proportional to the length of the probe and the sine of the angulation error, so small angulation errors can quickly compromise the accuracy of the tool tip localization. We propose a real-time tracking correction technique that uses the rotational fulcrum created by the device entry port to minimize the effect of translational and rotational noise errors for tool tip localization. Our technique could apply to many types of interventions, but we focus on the application to the prostate biopsy procedure for tracking a transrectal ultrasound (TRUS) probe commonly used for prostate biopsies. In vitro studies were performed using the Claron Technology MicronTracker 2 to track a TRUS probe in a fixed rotational device. Our experimental results showed an order of magnitude improvement in RMS localization of the internal TRUS probe tip using fulcrum correction over the raw tracking information.

We present a proof of principle, experimental validation of the potential of proton ‘Range Probes’ (RP) for patient positioning verification in proton therapy. In this work, we have evaluated experimentally the accuracy of RP by using tissue-like samples and an in-house developed multilayer ionization chamber (MLIC). In addition we build on our previous, simulation based work to present first experimental measurements of RP through anthropomorphic phantoms to detect either rotational or translational positioning errors. For this, a technique has been proposed to characterize the residual integral depth dose curve (RIDDC) after range mixing. This parametrization has been used to evaluate the similarity between Monte Carlo calculated error scenarios of the database and the measured RIDDC, while considering the intrinsic uncertainties of both modalities in order to deduce the positioning error. Finally, the additional dose applied to the patient when using clinical RP with known fluence has been estimated by measuring the local dose of a single RP. In tissue phantoms, the prediction accuracy of the water equivalent path length was 0.70%, with the highest deviations being found in low density samples (up to 5.67%). In addition, the results of the patient positioning verification measurements demonstrated that using carefully selected RPs, 1D translational or rotational errors could be detected with an accuracy of 1 mm and 2°, respectively, and that these would be associated with a low additional dose burden to the patient. In summary, these promising results suggest that the RP method could be a simple, fast and low-dose tool for verifying patient set-up during proton therapy treatment.

Accurate self-localization is a critical problem in the autonomous driving system. In this paper, we proposed a localization system which integrates stereo camera and three-dimensional city model containing building and road mark information. The stereo camera generates visual odometry, reconstructs building scene and detects road mark. We aligned building scene with Normal Distribution Transform (NDT) map to get the absolute localization. Road mark detection result helps to rectify the inner lane positioning with road map. The experiment conducted in Hitotsubashi, Tokyo indicates that the lateral and heading error of visual odometry can be corrected and sub-meter accuracy localization is achieved.

Evaluation of the dosimetric influence of interfractional 6D setup error in hypofractionated prostate cancer treated with IMRT and VMAT using daily kV-CBCT