Robust Stereo Visual-Inertial Odometry Using Nonlinear Optimization

Stereo-vision devices have rigorous requirements for extrinsic parameter calibration. In Stereo Visual Inertial Odometry (VIO), inaccuracy in or changes to camera extrinsic parameters may lead to serious degradation in estimation performance. In this manuscript, we propose an online calibration method for stereo VIO extrinsic parameters correction. In particular, we focus on Multi-State Constraint Kalman Filter (MSCKF [1]) framework to implement our method. The key component is to formulate stereo extrinsic parameters as part of the state variables and model the Jacobian of feature reprojection error with respect to stereo extrinsic parameters as sub-block of update Jacobian. Therefore we can estimate stereo extrinsic parameters simultaneously with inertial measurement unit (IMU) states and camera poses. Experiments on EuRoC dataset and real-world outdoor dataset demonstrate that the proposed algorithm produce higher positioning accuracy than the original S-MSCKF [2], and the noise of camera extrinsic parameters are self-corrected within the system.

Reliable extrinsic calibration is critical to fuse camera and inertial measurement unit (IMU), which are usually used in stereo visual-inertial odometry (VIO), however, it is difficult to obtain extrinsic parameters in practice. This paper proposes a stereo VIO with the capability of calibrating the unknown extrinsic parameters online. The initial values of IMU-camera and camera-camera transformations are estimated during the initial process, which fully leverages commonly observed features between stereo cameras. The pose and velocity are estimated and the extrinsic parameters are jointly refined. Additionally, the accelerometer and gyroscope biases, and gravity direction are taken into account. The proposed VIO scheme have been demonstrated to be effective without prior knowledge of extrinsic parameters.

In the γ radioactive environment, high-energy photons induce degradation of the image sensor, which effects feature detection and tracking in the Visual Inertial Odometry (VIO) algorithm and deteriorates its localization performance. To address this issue, in this work, we propose a monocular VIO method using edge-based point features. To mitigate the effects of radiation noise, firstly, in the image preprocessing module, the median filter is used for real-time image denoising. Secondly, in the data association module, both Shi-Tomasi and edge-based point features are detected. The edge-based point feature is the endpoint or corner point in the salient edge map, which is more robust to radiation noise. Then, the bi-directional motion parallaxes and the RANdom SAmple Consensus (RANSAC) method are exploited to reject outliers. Finally, the point features measurements and Inertial Measurement Unit (IMU) pre-integration measurements are added into a tightly-coupled sliding window optimization VIO framework for localization estimation. The proposed method is verified by synthetic and real γ radioactive environment datasets. The experimental results show that the proposed method achieves more accurate and robust localization than the state-of-the-art VIO approaches in the γ radioactive environments.

In this study, we built a monocular visual inertial odometry (VIO) dataset with a proposed self-calibrating platform for an inertial measurement unit (IMU). For a learning-based method, lack of database is the biggest limitation. One goal of this study is to generate a high-quality VIO database. Meanwhile, a properly calibrated inertial measurement unit (IMU) is critical for improving VIO system accuracy and precision. However, a professional IMU calibration tool is usually expensive and large. The proposed self-calibrating platform shows its benefits in eliminating the manual work in calibration and reducing the cost of the expensive platform, which brings more opportunities to students and researchers. This self-calibrating system was useful for collecting monocular visual inertial odometry datasets while driving. . The calibrated IMU generates 14 numbers at 100Hz: time, 3-axis accelerometer measurements, 3-axis gyroscope measurements, 3 Euler angles, and 4 quaternions. This VIO system is composed of a monocular camera, GoPro Hero 3, an IMU, BNO055, and a GPS, ANT-555. The sensors are mounted on a John Deere Gator™ Full-Size Crossover Utility Vehicle and are operated on an Intel NUC, a mini-size computer. The final dataset contains monocular images, IMU measurements, and GPS outputs.

Accurate and continuous positioning in global navigation satellite system (GNSS) deprived urban areas is crucial for autonomous vehicles and mobile mapping systems. To achieve this goal, we propose a stereo visual-inertial odometry approach using a multistate constraint Kalman filter (MSCKF). In contrast with the conventional MSCKF, in which an inertial navigation system (INS) propagates the vehicle motion, and then the propagation is corrected by measurements of salient features extracted from images of a single camera, we update the propagation with observations extracted from images of a stereo pair of cameras. This way, additional constraints across the stereo pairs of images are exploited to improve the pose estimation. Experimental results on several KITTI datasets show that the stereo MSCKF outperforms the mono one achieving an average positioning error of 0.9% of the trajectory length compared to 1.3% for the mono approach and 2.5% for INS-only integration. These results show that visual-inertial odometry has a promising potential for vehicle positioning in short periods of GNSS signal outage.

Recovering rigid registration between successive camera poses lies at the heart of 3D reconstruction, SLAM and visual odometry. Registration relies on the ability to compute discriminative 2D features in successive camera images for determining feature correspondences, which is very challenging in feature-poor environments, i.e. low-texture and/or low-light environments. In this paper, we aim to address the challenge of recovering rigid registration between successive camera poses in feature-poor environments in a Visual Inertial Odometry (VIO) setting. In addition to inertial sensing, we instrument a small aerial robot with an RGBD camera and propose a framework that unifies the incorporation of 3D geometric entities: points, lines, and planes. The tracked 3D geometric entities provide constraints in an Extended Kalman Filtering framework. We show that by directly exploiting 3D geometric entities, we can achieve improved registration. We demonstrate our approach on different texture-poor environments, with some containing only flat texture-less surfaces providing essentially no 2D features for tracking. In addition, we evaluate how the addition of different 3D geometric entities contributes to improved pose estimation by comparing an estimated pose trajectory to a ground truth pose trajectory obtained from a motion capture system. We consider computationally efficient methods for detecting 3D points, lines and planes, since our goal is to implement our approach on small mobile robots, such as drones.

The YTU dataset and recurrent neural network based visual-inertial odometry

This dissertation presents a comprehensive approach to addressing the challenges associated with the control and navigation of Unmanned Aerial Vehicles (UAVs), specifically focusing on robustness under GPS-denied, turbulent, and dark indoor conditions with various disturbances. Initial efforts involved designing and theoretically validating Sliding Mode Controllers (SMC) with Disturbance Observers using MATLAB/SIMULINK, optimized using offline Particle Swarm Optimization (PSO). In order to evaluate their performance, a lightweight simulator was developed to simulate both quadcopter and hexacopter configurations under the PX4 architecture. This simulator served to validate and compare the optimized SMC against PSO-optimized PID controllers. The results of these comparisons demonstrated the superior adaptability of the SMC, especially in environments characterized by significant sensor noise and disturbances. Subsequent practical indoor tests were conducted with real quadcopters, which provided additional verification of the robustness and reliability of the enhanced control theory with the actual UAVs under dynamic disturbances. To further evaluate the compatibility and adaptability of the SMC, this dissertation establishes a framework for interfacing with high-performance STMH7 chips, serving as the UAV microcontroller unit (MCU). This framework facilitates the connection of various sensors and provides guidance for constructing different quadcopter frames. Moreover, it enables the comparison of different flight controllers on distinct air-frames. Concurrently, the optimized SMC allows for remote dynamic parameter tuning during flight. In pursuit of improved localization capacity, the dissertation also explores the integration of an Intel Realsense D435i camera with an Inertial Measurement Unit (IMU) for Visual-Inertial Odometry (VIO) fusion within the Robot Operating System (ROS) environment. This integration employs an onboard companion computer for real-time VIO estimation and flight command. The effectiveness of the VIO algorithm is verified through both simulations and real-world experiments, highlighting its ability to enhance UAV localization and navigation. Additionally, a visual inertial odometry method for UAV Simultaneous Localization and Mapping (vSLAM) is tested and validated in both simulated and real drone environments. Furthermore, the dissertation introduces an innovative hybrid filtered multi-directional radar inertial odometry solution denoted as Hybrid-MRIO. This Error State Extended Kalman Filter (ES-EKF) based Radar Inertial Odometry (RIO) approach is implemented within the open-source PX4 autopilot system, enhancing UAV navigation accuracy and performance, and serves as a robust navigation alternative, particularly in challenging environments where traditional navigation systems prove ineffective, such as dark or smoky conditions. This solution integrates data from multiple software synchronized high-resolution Frequency-Modulated Continuous-Wave (FMCW) mmWave radars with an IMU. A subsequent 4D (x, y, z, doppler) mmWave radar SLAM (rSLAM) system has been developed, including three modules: front-end, loop detection module, and back-end. In the front-end, radar ego-velocity is utilized for states estimation and dynamic object removal, and a point cloud registration-based approach known as APDGICP (Adaptive Probability Distribution-GICP) is employed for keyframe detection. The loop detection module utilizes the intensity scan context to identify potential loop closure candidates. In the back-end, a pose graph is constructed, integrating RIO estimated odometry and identified loop closures for better localization. To develop fully-autonomous UAVs with their decision making modules, this dissertation details the development of a custom AI-powered fully-autonomous quadcopter equipped with companion computers and stereo IR cameras. Additionally, this dissertation discusses a swarm of small Unmanned Aerial Systems (SUAS) interconnected through a software-defined communications network. These drones demonstrated their capacity to independently/collaboratively execute a wide range of tasks, including target search, detection, identification, classification, tracking, and following, both in simulated and real-world scenarios. The system exhibits advanced collision avoidance capabilities and the ability to strategically respond to dynamic scenarios, such as changes in target behavior, highlighting its effectiveness in managing dynamic and challenging situations without the need for continuous human intervention.--Author's abstract

Vision-based localization is an essential problem for autonomous systems while the performance of vision-based odometry degrades in the challenging scenario. This letter presents S-VIO, an RGB-D visual inertial odometry (VIO) which fully uses multi-sensor measurements (i.e. depth, RGB and IMU), heterogeneous landmarks (i.e. points, lines and planes) and structural regularity of the environment to obtain a robust and accurate localization result. In order to detect the underlying structural regularity of the environment, a two-step Atlanta world inference method is proposed. Leveraging the gravity direction estimated by the VIO system, the proposed algorithm first generates horizontal Atlanta axis hypotheses from a set of recently optimized plane landmarks. Then the following plane landmarks and line clusters are used to filter out the occasionally observed axes based on the persistence of the observation. The remaining axes will survive and be saved in the Atlanta map for future re-observations. Particularly, an efficient mined-and-stabbed (MnS) method is applied to classify the structural line and extract the vanishing point from each line cluster. In addition, a closed-form initialization method for the structural line feature is proposed, which leverages the known direction to obtain a better initial estimation. Integrated with the above novelties, S-VIO is tested on two public real-world RGB-D inertial datasets. Experiments demonstrate that S-VIO has better accuracy and robustness compared to state-of-the-art VIO and RGB-D VIO algorithms. The code and the supplementary material will be available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/oranfire/SVIO</uri> .

The estimation error accumulation in the conventional visual inertial odometry (VIO) generally forbids accurate long-term operations. Some advanced techniques such as global pose graph optimization and loop closure demand relatively high computation and processing time to execute the optimization procedure for the entire trajectory and may not be feasible to be implemented in a low-cost robotic platform. In an attempt to allow the VIO to operate for a longer duration without either using or generating a map, this paper develops iterated cubature Kalman filter for VIO application that performs multiple corrections on a single measurement to optimize the current filter state and covariance during the measurement update. The optimization process is terminated using the maximum likelihood estimate based criteria. For comparison, this paper also develops a second solution to integrate VIO estimation with ranging measurements. The wireless communications between the vehicle and multiple beacons produce the ranging measurements and help to bound the accumulative errors. Experiments utilize publicly available dataset for validation, and a rigorous comparison between the two solutions is presented to determine the application scenario of each solution.

Recently, application of Unmanned Aerial Vehicle (UAV) attract much attention of not only academic but also industrial community. To control and make the most of UAV, state estimation is mandatory and it should work accurately on Real-time on computational-limited onboard PC. Especially, for the cases that GNSS-disabled environments or indoor situation, Visual Inertial Odometry using camera and IMU sensor has been actively studied. Generally, to research and develop the indoor navigation algorithm, high-performanced and super-expensive facilities like motion capture cameras are needed to measure the ground-truth. In this paper, we apply state-of-art Visual Inertial Odometry algorithms for state estimation of UAVs to FlightGoggles simulator which is photorealistic and which reflects UAV's real dynamics. Moreover, we compare the performance and resource utilization of those algorithms.

Nowadays, multi-sensor fusion technology is a fundamental prerequisite to achieve highly autonomous robots and robustness. Many studies have been conducted, such as visual–inertial odometry (VIO), integrated navigation, and LiDAR–inertial odometry. Typically, for VIO, gratifying results have been achieved, ascribing to the complementary sensing capabilities of inertial measurement units (IMUs) and cameras. However, this work mainly focuses on the fusion of visual and inertial data, while the IMU error is less considered, especially for low-cost or poorly calibrated microelectromechanical system (MEMS) IMU. Such errors may have a significant effect on the VIO performance. In this study, we compensated for the IMU using camera assistance. The key characteristic of the method is that we optimize the compensation parameters (scale factor) from coarse to fine by combining the time domain with the frequency domain. The proposed method is to use the time-domain and frequency-domain optimization to suppress large noise in the dynamic calibration process of the extremely low-cost sensor platform. The effectiveness of this method is validated through experiments and simulations. The minimal calibration error (0.46%) is commensurate with the advanced work. By feeding the compensated IMU into the VIO algorithm, the localization accuracy is improved by 9% to 15%. This method improves the performance in the VIO algorithm, which is equipped with the low-cost or poorly calibrated MEMS IMU and reduces the hardware and deployment costs of the system.

In this paper, we present a partial sparsification scheme for the marginalization of visual inertial odometry (VIO) systems. Sliding window optimization is widely used in VIO systems to guarantee constant complexity by optimizing over a set of recent states and marginalizing out past ones. The marginalization step introduces fill-in between variables incident to the marginalized ones, and most VIO systems discard measurements targeted at active landmark points to maintain sparsity of the marginalized information matrix, at the expense of potential information loss. The scheme is to first retain the dense prior from the marginalization excluding visual measurements, followed by a dense marginalization step that connects landmarks. The dense marginalization prior is then partially sparsified to extract pseudo factors that maintain the overall sparsity while minimizing the information loss. The proposed scheme is tested on public datasets and achieves appreciable results compared with several state-of-the-art approaches. The test also demonstrates that our scheme is applicable to real-time operations.

In this paper, we show how beacon-based indoor localization and additional environmental fingerprints like magnetic field data can be used to both accelerate and increase the robustness of Augmented Reality (AR) relocalization. We show how the combination of Visual Inertial Odometry (VIO) and beacons can be used to construct a dense indoor magnetic field map that can act as a fine-grained calibration for compasses to quickly determine a mobile device’s orientation. Unique to our approach is that we leverage accurate VIO trajectories to provide full vector magnetic field mapping that can be collected and used with devices placed in any orientation. We demonstrate a system running on an iPhone that can acquire location with 80th percentile 3D accuracy of 27cm in LOS and 46cm in NLOS, and our magnetic field mapping approach can instantly estimate orientation with 80th percentile accuracy of 11.7 degrees. We demonstrate an end-to-end system for generating a magnetic field map and subsequently localizing and tracking mobile users using beacons and VIO. This has the side effect of enabling multi-user (even cross-platform) AR applications, as all users can now be localized with respect to a common global reference without any sharing of visual feature maps.

Event-based visual odometry is a specific branch of visual simultaneous localization and mapping (SLAM) techniques, which aims at solving tracking and mapping subproblems (typically in parallel), by exploiting the special working principles of neuromorphic (i.e., event-based) cameras. Due to the motion-dependent nature of event data, explicit data association (i.e., feature matching) under large-baseline viewpoint changes is difficult to establish, making direct methods a more rational choice. However, state-of-the-art direct methods are limited by the high computational complexity of the mapping subproblem and the degeneracy of camera pose tracking in certain degrees of freedom (DoF) in rotation. In this article, we tackle these issues by building an event-based stereo visual-inertial odometry system, which is built upon a direct pipeline known as event-based stereo visual odometry (ESVO). Specifically, to speed up the mapping operation, we propose an efficient strategy for sampling contour points according to the local dynamics of events. The mapping performance is also improved in terms of structure completeness and local smoothness by merging the temporal stereo and static stereo results. To circumvent the degeneracy of camera pose tracking in recovering the pitch and yaw components of general 6-DoF motion, we introduce IMU measurements as motion priors via preintegration. To this end, a compact back-end is proposed for continuously updating the IMU bias and predicting the linear velocity, enabling an accurate motion prediction for camera pose tracking. The resulting system scales well with modern high-resolution event cameras and leads to better global positioning accuracy in large-scale outdoor environments. Extensive evaluations on five publicly available datasets featuring different resolutions and scenarios justify the superior performance of the proposed system against five state-of-the-art methods. Compared to ESVO, our new pipeline significantly reduces the camera pose tracking error by 40%–80% and 20%–80% in terms of absolute trajectory error and relative pose error, respectively; at the same time, the mapping efficiency is improved by a factor of five. We release our pipeline as an open-source software for future research in this field.