Direct Sparse Odometry with Rolling Shutter

Nowadays, state-of-the-art direct visual odometry (VO) methods essentially rely on points to estimate the pose of the camera and reconstruct the environment. Direct Sparse Odometry (DSO) became the standard technique and many approaches have been developed from it. However, only recently, two monocular plane-based DSOs have been presented. The first one uses a learning-based plane estimator to generate coarse planes as input for optimization. When these coarse estimates are too far from the minimum, the optimization may fail. Thus, the entire system result is dependent on the quality of the plane predictions and restricted to the training data domain. The second one only detects planes in vertical and horizontal orientation as being more adequate to structured environments. To the best of our knowledge, we propose the first Stereo Plane-based VO inspired by the DSO framework. Differing from the above-mentioned methods, our approach purely uses planes as features in the sliding window optimization and uses a dual quaternion as pose parameterization. The conducted experiments showed that our method presents a similar performance to Stereo DSO, a point-based approach.

We propose Stereo Direct Sparse Odometry (Stereo DSO) as a novel method for highly accurate real-time visual odometry estimation of large-scale environments from stereo cameras. It jointly optimizes for all the model parameters within the active window, including the intrinsic/extrinsic camera parameters of all keyframes and the depth values of all selected pixels. In particular, we propose a novel approach to integrate constraints from static stereo into the bundle adjustment pipeline of temporal multi-view stereo. Real-time optimization is realized by sampling pixels uniformly from image regions with sufficient intensity gradient. Fixed-baseline stereo resolves scale drift. It also reduces the sensitivities to large optical flow and to rolling shutter effect which are known shortcomings of direct image alignment methods. Quantitative evaluation demonstrates that the proposed Stereo DSO outperforms existing state-of-the-art visual odometry methods both in terms of tracking accuracy and robustness. Moreover, our method delivers a more precise metric 3D reconstruction than previous dense/semi-dense direct approaches while providing a higher reconstruction density than feature-based methods.

In recent years, SLAM, visual odometry, and structure-from-motion approaches have widely addressed the problems of 3D reconstruction and ego-motion estimation. Of the many input modalities that can be used to solve these ill-posed problems, the pure visual alternative using a single monocular RGB camera has attracted the attention of multiple researchers due to its low cost and widespread availability in handheld devices. One of the best proposals currently available is the Direct Sparse Odometry (DSO) system, which has demonstrated the ability to accurately recover trajectories and depth maps using monocular sequences as the only source of information. Given the impressive advances in single-image depth estimation using neural networks, this work proposes an extension of the DSO system, named DeepDSO. DeepDSO effectively integrates the state-of-the-art NeW CRF neural network as a depth estimation module, providing depth prior information for each candidate point. This reduces the point search interval over the epipolar line. This integration improves the DSO algorithm’s depth point initialization and allows each proposed point to converge faster to its true depth. Experimentation carried out in the TUM-Mono dataset demonstrated that adding the neural network depth estimation module to the DSO pipeline significantly reduced rotation, translation, scale, start-segment alignment, end-segment alignment, and RMSE errors.

Unsupervised monocular visual odometry with decoupled camera pose estimation

A novel RGB-D visual odometry method for dynamic environment is proposed. Majority of visual odometry systems can only work in static environments, which limits their applications in real world. In order to improve the accuracy and robustness of visual odometry in dynamic environment, a Feature Regions Segmentation algorithm is proposed to resist the disturbance caused by the moving objects. The matched features are divided into different regions to separate the moving objects from the static background. The features in the largest region which belong to the static background are used to estimate the camera pose finally. The effectiveness of our visual odometry method is verified in a dynamic environment of our lab. Furthermore, an exhaustive experimental evaluation is conducted on benchmark datasets including static environments and dynamic environments compared with the state-of-art visual odometry systems. The accuracy comparison results show that the proposed algorithm outperforms those systems in large scale dynamic environments. Our method tracks the camera movement correctly while others failed. In addition, our method can give the same good performances in static environment. Experiments demonstrate that the proposed RGB-D visual odometry can obtain accurate and robust estimation results in dynamic environments.

A method that optimizes visual odometry, especially using visual odometry in the scene with absence of features is proposed in this paper. First, in order to estimate the pose of camera when the effect of using feature-based method is not good enough, direct method is implemented as the solution. Second, comparing with traditional visual odometry method, this method takes the environment restriction into consideration. On the one hand, if enough feature points are detected in a frame, feature-based method is to be used; on the other hand, direct method is used in the scene of missing features so that the accuracy of pose estimation could be guaranteed and improved. Finally, the feasibility of this method is proven by a number of experiments conducted on the experiment platform.

Most previous learning-based visual odometry (VO) methods take VO as a pure tracking problem. In contrast, we present a VO framework by incorporating two additional components called Memory and Refining. The Memory component preserves global information by employing an adaptive and efficient selection strategy. The Refining component ameliorates previous results with the contexts stored in the Memory by adopting a spatial-temporal attention mechanism for feature distilling. Experiments on the KITTI and TUM-RGBD benchmark datasets demonstrate that our method outperforms state-of-the-art learning-based methods by a large margin and produces competitive results against classic monocular VO approaches. Especially, our model achieves outstanding performance in challenging scenarios such as texture-less regions and abrupt motions, where classic VO algorithms tend to fail.

Decentralized visual simultaneous localization and mapping (DSLAM) can share locations and environmental information between robots, which is an essential task for many multi-robot applications. The visual odometry (VO) is a basic component to estimate the 6-DoF absolute pose for robot applications. Decentralized place recognition (DPR) is a fundamental element to produce candidate place matches for sharing information among different robots. The goal of this paper is to build a CNN-based real-time DSLAM system on embedded FPGA platforms. Because of the high precision requirement of VO, the existing quantization methods can not be directly applied. We improve the fixed-point fine-tune method for the CNN-based monocular VO, which enables VO can be deployed on the fixed-point FPGA accelerator. We also explore the influence of the DPR frequency on the DSLAM results, and find out a proper DPR frequency to balance the accuracy and speed. A cross-component pipeline scheduling method is proposed to improve DPR frequency and further improve the final accuracy of DSLAM under the same hardware resource constraints.

Recent visual odometry (VO) methods incorporating geometric algorithm into deep-learning architecture have shown outstanding performance on the challenging monocular VO task. Despite encouraging results are shown, previous methods ignore the requirement of generalization capability under noisy environment and various scenes. To address this challenging issue, this work first proposes a novel optical flow network (PANet). Compared with previous methods that predict optical flow as a direct regression task, our PANet computes optical flow by predicting it into the discrete position space with optical flow probability volume, and then converting it to optical flow. Next, we improve the bundle adjustment module to fit the self-supervised training pipeline by introducing multiple sampling, ego-motion initialization, dynamic damping factor adjustment, and Jacobi matrix weighting. In addition, a novel normalized photometric loss function is advanced to improve the depth estimation accuracy. The experiments show that the proposed system not only achieves comparable performance with other state-of-the-art self-supervised learning-based methods on the KITTI dataset, but also significantly improves the generalization capability compared with geometry-based, learning-based and hybrid VO systems on the noisy KITTI and the challenging outdoor (KAIST) scenes.

Since the feature-based monocular simultaneous localization and mapping (SLAM) system consumes a lot of computing resources on extracting features and associating descriptors, it still discards potentially valuable photometric information. To solve this problem, a direct monocular visual odometry (DMVO) approach is investigated to model the SLAM system by photometric error optimization, where a photometric error model is established, residual pattern defined in direct sparse odometry (DSO) is modified, and the Gauss-Newton algorithm is adopted to implement the DMVO. The DMVO approach in this paper uses more image information and totally avoids consuming computing resources on features and descriptors. The comparative tests on the TUM monocular visual odometry dataset are presented and the results showed that the alignment error of DMVO approach is smaller than that of DSO and ORB-SLAM with loop-closure and re-localization disabled.

Visual odometry can be divided into direct and indirect methods. Indirect methods which have more computation complexities are more robust, while direct methods are faster but less robust. To achieve the goal in terms of efficiency and robustness, we propose a novel semi-direct visual odometry approach with depth prior, which can be applied to most monocular direct odometry approaches. The proposed method is implemented based Direct Sparse Odometry. Experimental results on TUM datasets show that the proposed approach, which combines the advantages of both the direct and indirect methods, are more efficient and robust than the state-of-the-art method.

Visual odometry can be augmented by depth information such as provided by RGB-D cameras, or from lidars associated with cameras. However, such depth information can be limited by the sensors, leaving large areas in the visual images where depth is unavailable. Here, we propose a method to utilize the depth, even if sparsely available, in recovery of camera motion. In addition, the method utilizes depth by triangulation from the previously estimated motion, and salient visual features for which depth is unavailable. The core of our method is a bundle adjustment that refines the motion estimates in parallel by processing a sequence of images, in a batch optimization. We have evaluated our method in three sensor setups, one using an RGB-D camera, and two using combinations of a camera and a 3D lidar. Our method is rated #2 on the KITTI odometry benchmark irrespective of sensing modality, and is rated #1 among visual odometry methods.

Camera–inertial measurement unit (IMU) sensor fusion has been extensively studied in recent decades. Numerous observability analysis and fusion schemes for motion estimation with self-calibration have been presented. However, it has been uncertain whether the intrinsic parameters of both the camera and the IMU are observable under general motion. To answer this question, by using the Lie derivatives, we first prove that for a rolling shutter (RS) camera–IMU system, all the intrinsic and extrinsic parameters, camera time offset, and readout time of the RS camera are observable with an unknown landmark. To our knowledge, we are the first to present such a proof. Next, to validate this analysis and to solve the drift issue of a structureless filter during standstills, we develop a keyframe-based sliding window filter (KSWF) for odometry and self-calibration, which works with a monocular RS camera or stereo RS cameras. Though the keyframe concept is widely used in vision-based sensor fusion, to our knowledge, the KSWF is the first of its kind to support self-calibration. Our simulation and real data tests have validated that it is possible to fully calibrate the camera–IMU system using the observations of opportunistic landmarks under diverse motion. Real data tests confirmed previous allusions that keeping landmarks in the state vector can remedy the drift in standstill and showed that the keyframe-based scheme is an alternative solution.

The most significant part of any autonomous intelligent robot is the localization module that gives the robot knowledge about its position and orientation. This knowledge assists the robot to move to the location of its desired goal and complete its task. Visual Odometry (VO) measures the displacement of the robots’ camera in consecutive frames which results in the estimation of the robot position and orientation. Deep Learning, nowadays, helps to learn rich and informative features for the problem of VO to estimate frame-by-frame camera movement. Recent Deep Learning-based VO methods train an end-by-end network to solve VO as a regression problem directly without visualizing and sensing the label of training data in the training procedure. In this paper, a new approach to train Convolutional Neural Networks (CNNs) for the regression problems, such as VO, is proposed. The proposed method first changes the problem to a classification problem to learn different subspaces with similar observations. After solving the classification problem, the problem converts to the original regression problem to solve using the knowledge achieved by solving the classification problem. This approach helps CNN to solve regression problem globally in a local domain learned in the classification step, and improves the performance of the regression module for approximately 10%.

Visual odometry systems are usually categorized as feature-based visual odometry systems and direct visual odometry systems. The feature-based visual odometry systems spend time on calculating descriptors while direct visual odometry systems directly calculate residuals. The optimization of direct visual odometry systems is sensitive with initial states while feature-based visual odometry systems did not suffer from this problem. This paper presents a features assisted stereo direct visual odometry system with virtual wide field-of-view tracking. It combines the advantages of feature-based methods and direct methods. It is achieved by incorporating a feature assisted predicting module and a virtual wide field-of-view tracking module into stereo direct sparse odometry (SDSO) framework. The feature-based matching is combined in direct tracking module for efficient and robust data association. Furthermore, before data association, a virtual wide field-of-view inverse depth frame is created in case of tracking lost in aggressive rotating conditions. Evaluation results on the EuRoC datasets and the KITTI dataset show that, with the aid of the new tracking module, the stereo visual odometry becomes more robust and more accurate.