Enhanced calibration of camera setups for high-performance visual odometry

In this paper, we propose a new scheme based on the Sampson distance (SD) to describe visual feature residuals for visual-inertial odometry (VIO). Unlike the epipolar-constraint-based SD for visual odometry (VO), the proposed SD is formulated based on the perspective projection constraint. We proved in theory that the proposed SD retains the good properties of those earlier SD criteria in the literature of VO and it represents a visual feature residual more accurately than the prevailing transfer distance (TD) in existing VIO methods. We formulate three distance criteria, including TD, reprojection error (RE), and SD, and compared their performances by simulation. The results show that the SD is much more accurate than the TD and it is a very accurate estimate of the gold standard criteria—RE. Based on the SD, we modified VINS-Mono by replacing its TD-based visual residuals with the SD-based residuals and study the SD's efficacy in pose estimation by experiments with several public datasets. The results reveal that the SD-based VINS-Mono has a substantial improvement over the original VINS-Mono in pose estimation accuracy. This indicates that the SD is a better distance criterion than the TD for representing visual feature residuals. The proposed SD may find its applications to broader areas in computer vision and robotics.

The YTU dataset and recurrent neural network based visual-inertial odometry

A novel RGB-D visual odometry method for dynamic environment is proposed. Majority of visual odometry systems can only work in static environments, which limits their applications in real world. In order to improve the accuracy and robustness of visual odometry in dynamic environment, a Feature Regions Segmentation algorithm is proposed to resist the disturbance caused by the moving objects. The matched features are divided into different regions to separate the moving objects from the static background. The features in the largest region which belong to the static background are used to estimate the camera pose finally. The effectiveness of our visual odometry method is verified in a dynamic environment of our lab. Furthermore, an exhaustive experimental evaluation is conducted on benchmark datasets including static environments and dynamic environments compared with the state-of-art visual odometry systems. The accuracy comparison results show that the proposed algorithm outperforms those systems in large scale dynamic environments. Our method tracks the camera movement correctly while others failed. In addition, our method can give the same good performances in static environment. Experiments demonstrate that the proposed RGB-D visual odometry can obtain accurate and robust estimation results in dynamic environments.

Most previous learning-based visual odometry (VO) methods take VO as a pure tracking problem. In contrast, we present a VO framework by incorporating two additional components called Memory and Refining. The Memory component preserves global information by employing an adaptive and efficient selection strategy. The Refining component ameliorates previous results with the contexts stored in the Memory by adopting a spatial-temporal attention mechanism for feature distilling. Experiments on the KITTI and TUM-RGBD benchmark datasets demonstrate that our method outperforms state-of-the-art learning-based methods by a large margin and produces competitive results against classic monocular VO approaches. Especially, our model achieves outstanding performance in challenging scenarios such as texture-less regions and abrupt motions, where classic VO algorithms tend to fail.

Depth estimation and camera calibration of a focused plenoptic camera for visual odometry

With the rapid development of autonomous vehicle technologies, how to perform high-precision localization in unknown complex outdoor environment has become an important issue. Visual odometry is one of the low-cost and the most widely utilized localization methods. Traditional methods predict relative pose based on the principle of multi-view geometry, which is sensitive to camera parameters and environmental changes. This paper studies deep learning-based methods which can be more robust. A novel end-to-end unsupervised visual odometry framework based on confidence evaluation is proposed. Its process can be divided into two stages. The first is predicting the initial relative pose transformation with the help of confidence mask which is generated by measuring the relative similarity of geometric corresponding regions in associated images. The second is evaluating the confidence of the output pose estimate based on the trajectory geometric consistency and then refining it. Quantitative and qualitative evaluation of the proposed approach on KITTI dataset are presented to demonstrate its effectiveness in improving pose estimation accuracy and robustness.

A local bundle adjustment is an important proce- dure to improve the accuracy of a visual odometry solution. However, it is computationally very expensive as it jointly optimize all the poses of cameras and locations of map points. To reduce the computational complexity of a local bundle adjustment, the state-of-the-art algorithms [1]–[3] were proposed to manipulate the map point variables, using extra matrix operations, from their linearized optimization solutions. Instead of relying on complex matrix manipulations, this paper proposes a novel way of addressing this complexity issue-we represent a map point as a function of two camera poses, and uses the triangulated location of the map point when needed. Our method is more efficient than ones in the full-SLAM formulation in solving the visual odometry problem in that 1) the complexity of our solution is lower than those of the state-of-the-art methods, 2) no extra matrix operations required to eliminate map points, 3) no need guesses on map points' initial locations. Experiemental results, through simulated experiments and experiments with the KITTI dataset, demonstrated that our results are more accurate than those of a full-SLAM approach with lower runtime complexities.

Recent direct visual odometry and SLAM algorithms have demonstrated impressive levels of precision. However, they require a photometric camera calibration in order to achieve competitive results. Hence, the respective algorithm cannot be directly applied to an off-the-shelf-camera or to a video sequence acquired with an unknown camera. In this work we propose a method for online photometric calibration which enables to process auto exposure videos with visual odometry precisions that are on par with those of photometrically calibrated videos. Our algorithm recovers the exposure times of consecutive frames, the camera response function, and the attenuation factors of the sensor irradiance due to vignetting. Gain robust KLT feature tracks are used to obtain scene point correspondences as input to a nonlinear optimization framework. We show that our approach can reliably calibrate arbitrary video sequences by evaluating it on datasets for which full photometric ground truth is available. We further show that our calibration can improve the performance of a state-of-the-art direct visual odometry method that works solely on pixel intensities, calibrating for photometric parameters in an online fashion in realtime.

Estimating relative camera pose is the key problem of visual odometry (VO). To achieve better efficiency, sparse key-points are usually relied on for the estimation. Yet, feature extraction and matching are still computationally demanding, hindering the overall VO from real-time processing. Exploiting the superiorities of an FPGA in terms of high efficiency, low power consumption, and low cost, this paper proposes a multiple master-slave FPGA architecture for an SIFT-based stereo VO. The master-slave design enables high reconfiguration for the data throughputs among various modules. These modules include SIFT, matching, pose estimation, and their corresponding controllers. In the SIFT module, hardware implemented image pyramid is proposed, where scales are determined off-line via a minimization approach. Local linear exhausted search (LES) matching is considered for both the stereo and the frame matching. In the pose estimation module, a novel hardware design of deriving closest orthogonal matrix for 3D-3D correspondences of relative pose estimation is proposed. Experimental results show that 33.2 fps can be achieved using KITTI dataset without the need of a large number of hardware resources. The proposed reconfigurable design also facilitates its expansions of adopting CCD cameras as well as developing SLAM and other applications.

Recent visual odometry (VO) methods incorporating geometric algorithm into deep-learning architecture have shown outstanding performance on the challenging monocular VO task. Despite encouraging results are shown, previous methods ignore the requirement of generalization capability under noisy environment and various scenes. To address this challenging issue, this work first proposes a novel optical flow network (PANet). Compared with previous methods that predict optical flow as a direct regression task, our PANet computes optical flow by predicting it into the discrete position space with optical flow probability volume, and then converting it to optical flow. Next, we improve the bundle adjustment module to fit the self-supervised training pipeline by introducing multiple sampling, ego-motion initialization, dynamic damping factor adjustment, and Jacobi matrix weighting. In addition, a novel normalized photometric loss function is advanced to improve the depth estimation accuracy. The experiments show that the proposed system not only achieves comparable performance with other state-of-the-art self-supervised learning-based methods on the KITTI dataset, but also significantly improves the generalization capability compared with geometry-based, learning-based and hybrid VO systems on the noisy KITTI and the challenging outdoor (KAIST) scenes.

This paper addresses the problem of simultaneous estimation of a vehicle's ego motion and motions of multiple moving objects in the scene–called eoru motions –through a monocular vehicle-mounted camera. Localization of multiple moving objects and estimation of their motions is crucial for autonomous vehicles. Conventional localization and mapping techniques (e.g., visual odometry and simultaneous localization and mapping) can only estimate the ego motion of the vehicle. The capability of a robot localization pipeline to deal with multiple motions has not been widely investigated in the literature. We present a theoretical framework for robust estimation of multiple relative motions in addition to the camera ego motion. First, the framework for general unconstrained motion is introduced and then it is adapted to exploit the vehicle kinematic constraints to increase efficiency. The method is based on projective factorization of the multiple-trajectory matrix . First, the ego motion is segmented and then several hypotheses are generated for the eoru motions . All the hypotheses are evaluated and the one with the smallest reprojection error is selected. The proposed framework does not need any a priori knowledge of the number of motions and is robust to noisy image measurements. The method with a constrained motion model is evaluated on a popular street-level image dataset collected in urban environments (the KITTI dataset), including several relative ego-motion and eoru-motion scenarios. A benchmark dataset ( Hopkins 155 ) is used to evaluate this method with a general motion model. The results are compared with those of the state-of-the-art methods considering a similar problem, referred to as multibody structure from motion in the computer vision community.

Visual Odometry (VO) is a proven technology for planetary exploration rovers, facilitating their localization with a small error over medium-sized trajectories. However, due to VO's incremental mode of operation, its estimation error accumulates over time, resulting in considerable drift for long trajectories. This paper proposes a global localization method that counters VO drift by matching boulders extracted from overhead and ground images and using them periodically to relocalize the rover and refine VO estimates. The performance of the proposed method is evaluated with the aid of overhead imagery of different resolutions. Experimental results demonstrate that a very terse representation, consisting of approximate boulder locations only, suffices for significantly improving the accuracy of VO over long traverses.

Self-supervised monocular depth and visual odometry (VO) are often cast as coupled tasks. Accurate depth contributes to precise pose estimation and vice versa. Existing architectures typically exploit stacking convolution layers and long short-term memory (LSTM) units to capture long-range dependencies. However, their intrinsic locality hinders the model from getting the expected performance gain. In this article, we propose a Transformer-based architecture, named Transformer-based self-supervised monocular depth and VO (TSSM-VO), to tackle these problems. It comprises two main components: 1) a depth generator that leverages the powerful capability of multihead self-attention (MHSA) on modeling long-range spatial dependencies and 2) a pose estimator built upon a Transformer to learn long-range temporal correlations of image sequences. Moreover, a new data augmentation loss based on structural similarity (SSIM) is introduced to constrain further the structural similarity between the augmented depth and the augmented predicted depth. Rigorous ablation studies and exhaustive performance comparison on the KITTI and Make3D datasets demonstrate the superiority of TSSM-VO over other self-supervised methods. We expect that TSSM-VO would enhance the ability of intelligent agents to understand the surrounding environments.

In this paper, an approach for reducing the drift in monocular visual odometry algorithms is proposed based on a feedforward neural network. A visual odometry algorithm computes the incremental motion of the vehicle between the successive camera frames, then integrates these increments to determine the pose of the vehicle. The proposed neural network reduces the errors in the pose estimation of the vehicle which results from the inaccuracies in features detection and matching, camera intrinsic parameters, and so on. These inaccuracies are propagated to the motion estimation of the vehicle causing larger amounts of estimation errors. The drift reducing neural network identifies such errors based on the motion of features in the successive camera frames leading to more accurate incremental motion estimates. The proposed drift reducing neural network is trained and validated using the KITTI dataset and the results show the efficacy of the proposed approach in reducing the errors in the incremental orientation estimation, thus reducing the overall error in the pose estimation.

In classical computer vision, rectification is an integral part of multi-view depth estimation. It typically includes epipolar rectification and lens distortion correction. This process simplifies the depth estimation significantly, and thus it has been adopted in CNN approaches. However, rectification has several side effects, including a reduced field of view (FOV), resampling distortion, and sensitivity to calibration errors. The effects are particularly pronounced in case of significant distortion (e.g., wide-angle fisheye cameras). In this paper, we propose a generic scale-aware self-supervised pipeline for estimating depth, euclidean distance, and visual odometry from unrectified monocular videos. We demonstrate a similar level of precision on the unrectified KITTI dataset with barrel distortion comparable to the rectified KITTI dataset. The intuition being that the rectification step can be implicitly absorbed within the CNN model, which learns the distortion model without increasing complexity. Our approach does not suffer from a reduced field of view and avoids computational costs for rectification at inference time. To further illustrate the general applicability of the proposed framework, we apply it to wide-angle fisheye cameras with 190° horizontal field of view. The training framework UnRectDepthNet takes in the camera distortion model as an argument and adapts projection and unprojection functions accordingly. The proposed algorithm is evaluated further on the KITTI rectified dataset, and we achieve state-of-the-art results that improve upon our previous work FisheyeDistanceNet [1]. Qualitative results on a distorted test scene video sequence indicate excellent performance1.