Global-Context-Aware Visual Odometry System With Epipolar-Geometry-Constrained Loss Function

<abstract><p>For the autonomous and intelligent operation of robots in unknown environments, simultaneous localization and mapping (SLAM) is essential. Since the proposal of visual odometry, the use of visual odometry in the mapping process has greatly advanced the development of pure visual SLAM techniques. However, the main challenges in current monocular odometry algorithms are the poor generalization of traditional methods and the low interpretability of deep learning-based methods. This paper presented a hybrid self-supervised visual monocular odometry framework that combined geometric principles and multi-frame temporal information. Moreover, a post-odometry optimization module was proposed. By using image synthesis techniques to insert synthetic views between the two frames undergoing pose estimation, more accurate inter-frame pose estimation was achieved. Compared to other public monocular algorithms, the proposed approach showed reduced average errors in various scene sequences, with a translation error of $ 2.211\% $ and a rotation error of $ 0.418\; ^{\circ}/100m $. With the help of the proposed optimizer, the precision of the odometry algorithm was further improved, with a relative decrease of approximately 10$ \% $ intranslation error and 15$ \% $ in rotation error.</p></abstract>

The current visual simultaneous localization and mapping (SLAM) systems require the use of matched feature point pairs to estimate camera pose and construct environmental maps. Therefore, they suffer from poor performance of the visual feature matchers. To address this problem, a visual SLAM using deep feature matcher is proposed, which is mainly composed of three parallel threads: Visual Odometry, Backend Optimizer and LoopClosing. In the Visual Odometry, the deep feature extractor with convolutional neural networks is utilized for extracting feature points in each image frame. Then, the deep feature matcher is used for obtaining the corresponding feature landmark pairs. Afterwards, a fusion method based on the last and the reference frame is proposed for camera pose estimation. The Backend Optimizer is designed to execute local bundle adjustment for a part of camera poses and landmarks (map points). While LoopClosing, consisting of a lightweight deep loop closure detector and the same matcher as the one used in Visual Odometry is utilized for loop correction based on pose graph. The proposed system has been tested extensively on most of benchmark KITTI odometry dataset. The experimental results show that our system yields better performance than the existing visual SLAM systems. It can not only run in real-time at a speed of 0.08 s per frame, but also reduce estimation error by at least 0.1 m.

In this paper we consider a Simultaneous Localization and Mapping (SLAM) problem for a moving agent using Visual Odometry (VO) while measuring the range from a set of Ultra Wide Band (UWB) antennas, deployed in unknown position in the environment. The solution approach is based on a switching observer which, under standard working conditions, for each observed UWB, uses a two dimensional Extended Kalman Filter (EKF) providing an estimate of the range and bearing of the observed UWB with respect to the agent. This information is then used in a Robust EKF algorithm which solves the SLAM problem with performances that, even before closing the loop, are comparable to the ones that a VO algorithm (namely ORB-SLAM2) would obtain only after closing the loop. Moreover, a resilient module is added to the algorithm to evaluate the reliability of the position estimate of each observed UWB. When the Visual Odometry is not available, the switching observer uses an auxiliary EKF to provide an estimate of the agent position. This makes the proposed approach robust with respect to several kinds of unmodeled disturbances, like multipath effects, and automatically adapts to sensor failures with resilience (e.g. when Visual Odometry or UWB measurements are not available).

Most simultaneous localization and mapping (SLAM) systems assume that SLAM is conducted in a static environment. When SLAM is used in dynamic environments, the accuracy of each part of the SLAM system is adversely affected. We term this problem as dynamic SLAM. In this study, we propose solutions for three main problems in dynamic SLAM: camera tracking, three-dimensional map reconstruction, and loop closure detection. We propose to employ geometry-based method, deep learning-based method, and the combination of them for object segmentation. Using the information from segmentation to generate the mask, we filter the keypoints that lead to errors in visual odometry and features extracted by the CNN from dynamic areas to improve the performance of loop closure detection. Then, we validate our proposed loop closure detection method using the precision-recall curve and also confirm the framework’s performance using multiple datasets. The absolute trajectory error and relative pose error are used as metrics to evaluate the accuracy of the proposed SLAM framework in comparison with state-of-the-art methods. The findings of this study can potentially improve the robustness of SLAM technology in situations where mobile robots work together with humans, while the object-based point cloud byproduct has potential for other robotics tasks.

To solve the accurate positioning problem of mobile robots, simultaneous localization and mapping (SLAM) or visual odometry (VO) based on visual information are widely used. However, most visual SLAM or VO cannot meet the accuracy requirements in dynamic indoor environments. This paper proposes a robust visual odometry based on deep learning to eliminate feature points matching error. However, when a camera and dynamic objects are in relative motion, the frames of camera will produce ghosting, especially in high-dynamic environments, which bring additional positioning error; in view of this problem, a novel method based on the average optical flow value of the dynamic region is proposed to identify feature points of the ghosting, and then the feature points of the ghosting and dynamic region are removed. After the remaining feature points are matched, we use a non-linear optimization method to calculate the pose. The proposed algorithm is tested on TUM RGB-D dataset, and the results show that our VO improves the positioning accuracy than other robust SLAM or VO and is strongly robust especially in high-dynamic environments.

[abstFig src='/00290002/10.jpg' width='300' text='Visual odometry + trajectory correction' ] This paper proposes the ORB-SHOT SLAM or OS-SLAM, which is a novel method of 3D loop closing for trajectory correction of RGB-D visual SLAM. We obtain point clouds from RGB-D sensors such as Kinect or Xtion, and we use 3D SHOT descriptors to describe the ORB corners. Then, we train an offline 3D vocabulary that contains more than 600,000 words by using two million 3D descriptors based on a large number of images from a public dataset provided by TUM. We convert new images to bag-of-visual-words (BoVW) vectors and push these vectors into an incremental database. We query the database for new images to detect the corresponding 3D loop candidates, and compute similarity scores between the new image and each corresponding 3D loop candidate. After detecting 2D loop closures using ORB-SLAM2 system, we accept those loop closures that are also included in the 3D loop candidates, and we assign them corresponding weights according to the scores stored previously. In the final graph-based optimization, we create edges with different weights for loop closures and correct the trajectory by solving a nonlinear least-squares optimization problem. We compare our results with several state-of-the-art systems such as ORB-SLAM2 and RGB-D SLAM by using the TUM public RGB-D dataset. We find that accurate loop closures and suitable weights reduce the error on trajectory estimation more effectively than other systems. The performance of ORB-SHOT SLAM is demonstrated by 3D reconstruction application.

Visual odometry systems are usually categorized as feature-based visual odometry systems and direct visual odometry systems. The feature-based visual odometry systems spend time on calculating descriptors while direct visual odometry systems directly calculate residuals. The optimization of direct visual odometry systems is sensitive with initial states while feature-based visual odometry systems did not suffer from this problem. This paper presents a features assisted stereo direct visual odometry system with virtual wide field-of-view tracking. It combines the advantages of feature-based methods and direct methods. It is achieved by incorporating a feature assisted predicting module and a virtual wide field-of-view tracking module into stereo direct sparse odometry (SDSO) framework. The feature-based matching is combined in direct tracking module for efficient and robust data association. Furthermore, before data association, a virtual wide field-of-view inverse depth frame is created in case of tracking lost in aggressive rotating conditions. Evaluation results on the EuRoC datasets and the KITTI dataset show that, with the aid of the new tracking module, the stereo visual odometry becomes more robust and more accurate.

In this paper we describe VOPT (Visual Odometry by Patch Tracking), a robust algorithm for visual odometry, which is able to operate with sparse or dense maps computed by simultaneous localization and mapping (SLAM) algorithms. By using an iterative multi-scale procedure, VOPT is able to estimate the individual motion, photometric correction and reliability tracking confidence of a set of planar patches. In order to overcome the high computational cost of the patch adjustment, we use a GPU-based least-square solver, achieving real-time performance. The algorithm can also be used as a building block to other procedures for automatic initialization and recovery of 3D scene. Our tests show that VOPT outperforms the well-known PTAMM and the state-of-art ORB-SLAM algorithm in challenging videos using the same input maps.

The most significant part of any autonomous intelligent robot is the localization module that gives the robot knowledge about its position and orientation. This knowledge assists the robot to move to the location of its desired goal and complete its task. Visual Odometry (VO) measures the displacement of the robots’ camera in consecutive frames which results in the estimation of the robot position and orientation. Deep Learning, nowadays, helps to learn rich and informative features for the problem of VO to estimate frame-by-frame camera movement. Recent Deep Learning-based VO methods train an end-by-end network to solve VO as a regression problem directly without visualizing and sensing the label of training data in the training procedure. In this paper, a new approach to train Convolutional Neural Networks (CNNs) for the regression problems, such as VO, is proposed. The proposed method first changes the problem to a classification problem to learn different subspaces with similar observations. After solving the classification problem, the problem converts to the original regression problem to solve using the knowledge achieved by solving the classification problem. This approach helps CNN to solve regression problem globally in a local domain learned in the classification step, and improves the performance of the regression module for approximately 10%.

Visual odometry (VO) is the process of estimating the ego-motion of an agent sequentially (e.g. vehicle, human, and robot) using only the input of single or multiple cameras attached to it. VO plays an important role in Simultaneous Localization and Mapping (SLAM), which has been widely applied in many fields. The development of VO would greatly promote the transition of visual mapping and navigation to the industry. This paper focuses on direct methods and aims to summarize recent direct algorithms from technical views. Image representation and frame matching strategy are the two key components of VO. This paper is organized based on these two aspects. A qualitative analysis of existing work is then presented. This paper simultaneously serves as a position and tutorial paper.

Simultaneous Localization And Mapping (SLAM) has become a crucial aspect in the fields of autonomous driving and robotics. One crucial component of visual SLAM is the Field-of-View (FoV) of the camera, as a larger FoV allows for a wider range of surrounding elements and features to be perceived. However, when the FoV of the camera reaches the negative half-plane, traditional methods for representing image feature points using <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$[u,v,1]^T$</tex-math> </inline-formula> become ineffective. While the panoramic FoV is advantageous for loop closure, its benefits are not easily realized under large-attitude-angle differences where loop-closure frames cannot be easily matched by existing methods. As loop closure on wide-FoV panoramic data further comes with a large number of outliers, traditional outlier rejection methods are not directly applicable. To address these issues, we propose LF-VISLAM, a <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">V</b> isual <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">I</b> nertial <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SLAM</b> framework for cameras with extremely <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L</b> arge <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</b> oV with loop closure. A three-dimensional vector with unit length is introduced to effectively represent feature points even on the negative half-plane. The attitude information of the SLAM system is leveraged to guide the feature point detection of the loop closure. Additionally, a new outlier rejection method based on the unit length representation is integrated into the loop closure module. We collect the PALVIO dataset using a <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">P</b> anoramic <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A</b> nnular <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">L</b> ens (PAL) system with an entire FoV of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$360^\circ{\times}(40^\circ{\sim}120^\circ)$</tex-math> </inline-formula> and an Inertial Measurement Unit (IMU) for <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">V</b> isual <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">I</b> nertial <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">O</b> dometry (VIO) to address the lack of panoramic SLAM datasets. Experiments on the established PALVIO and public datasets show that the proposed LF-VISLAM outperforms state-of-the-art SLAM methods. Our code will be open-sourced at https://github.com/flysoaryun/LF-VISLAM. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —Motivated by the challenges of handling large-FoV cameras in SLAM applications, this paper proposes LF-VISLAM, a novel SLAM framework that uses a large-FoV camera and IMU sensors. Our framework is equipped with a loop closure thread that can use attitude information to eliminate accumulated errors. We have made algorithmic adjustments and optimizations to the negative half-plane features to better adapt to cameras with large FoV. Experimental evaluations demonstrate that LF-VISLAM significantly outperforms traditional SLAM methods. Additionally, the code will be open-sourced, providing easy access for research and implementation. Overall, LF-VISLAM is a promising solution to improve the performance of SLAM in challenging environments of autonomous driving and robotics.

Most previous learning-based visual odometry (VO) methods take VO as a pure tracking problem. In contrast, we present a VO framework by incorporating two additional components called Memory and Refining. The Memory component preserves global information by employing an adaptive and efficient selection strategy. The Refining component ameliorates previous results with the contexts stored in the Memory by adopting a spatial-temporal attention mechanism for feature distilling. Experiments on the KITTI and TUM-RGBD benchmark datasets demonstrate that our method outperforms state-of-the-art learning-based methods by a large margin and produces competitive results against classic monocular VO approaches. Especially, our model achieves outstanding performance in challenging scenarios such as texture-less regions and abrupt motions, where classic VO algorithms tend to fail.

Convolutional neural networks (CNNs) have shown great advantages in computer vision fields, and loss functions are of great significance to their gradient descent algorithms. Softmax loss, a combination of cross-entropy loss and Softmax function, is the most commonly used one for CNNs. Hence, it can continuously increase the discernibility of sample features in classification tasks. Intuitively, to promote the discrimination of CNNs, the learned features are desirable when the inter-class separability and intra-class compactness are maximized simultaneously. Since Softmax loss hardly motivates this inter-class separability and intra-class compactness simultaneously and explicitly, we propose a new method to achieve this simultaneous maximization. This method minimizes the distance between features of homogeneous samples along with Softmax loss and thus improves CNNs' performance on vision-related tasks. Experiments on both visual classification and face verification datasets validate the effectiveness and advantages of our method.

Several works have focused on Simultaneous Localization and Mapping (SLAM), which is a topic that has been studied for more than a decade to meet the needs of robots to navigate in an unknown environment. SLAM is an essential perception functionality in several applications, especially in robotics and autonomous vehicles. RGB-D cameras are among the sensors commonly used with recent SLAM algorithms. They provide an RGB image and the associated depth map, making it possible to solve scale drift with less complexity and create a dense 3D environment representation. Many RGB-D SLAM algorithms have been studied and evaluated on publicly available datasets without considering sensor specifications or image acquisition modes that could improve or decrease localization accuracy. In this work, we deal with indoor localization, taking into account the sensor specifications. In this context, our contribution is a deep experimental study to highlight the impact of the sensor acquisition modes on the localization accuracy, and a parametric optimization protocol for a precise localization in a given environment. Furthermore, we apply the proposed protocol to optimize a depth-related parameter of the SLAM algorithm. The study is based on a publicly available dataset in an indoor environment with a depth sensor. The reconstruction results’ analysis is founded on the study of different metrics involving translational and rotational errors. These metrics errors are compared with those obtained with a state-of-the-art stereo vision-based SLAM algorithm.

Visual Simultaneous Localization and Mapping (SLAM) plays an important role in computer vision and robotic field. With the development of Convolutional Neural Network (CNN), most scholars currently fuse CNN with visual SLAM to reduce the impact of dynamic objects on visual SLAM. To address the impact of semantic segmentation networks with lower precision and quad-tree algorithm with over-uniform distribution of feature points on the location accuracy of SLAM, we proposed an STDC-SLAM: Short-Term Dense Concatenate Network SLAM, which was based on ORB-SLAM3. In the proposed system, a real-time STDC network was used for semantic thread to segment dynamic objects. On the one hand, we designed a segmentation refinement module to optimize the semantic segmentation maps using images depth information. On the other hand, we improved the Qtree-ORB algorithm by reducing the iterations of low-quality feature points in the rejection thread. We have evaluated our SLAM in public data sheets and compared it with ORB-SLAM3, DynaSLAM, PSPNet-SLAM. Experiments showed that our SLAM improved in localization accuracy compared to DynaSLAM and in processing speed compared to DynaSLAM and PSPNet-SLAM.