Improvement on Visual-Inertial Odometry via Epipolar Constraints

Nowadays, multi-sensor fusion technology is a fundamental prerequisite to achieve highly autonomous robots and robustness. Many studies have been conducted, such as visual–inertial odometry (VIO), integrated navigation, and LiDAR–inertial odometry. Typically, for VIO, gratifying results have been achieved, ascribing to the complementary sensing capabilities of inertial measurement units (IMUs) and cameras. However, this work mainly focuses on the fusion of visual and inertial data, while the IMU error is less considered, especially for low-cost or poorly calibrated microelectromechanical system (MEMS) IMU. Such errors may have a significant effect on the VIO performance. In this study, we compensated for the IMU using camera assistance. The key characteristic of the method is that we optimize the compensation parameters (scale factor) from coarse to fine by combining the time domain with the frequency domain. The proposed method is to use the time-domain and frequency-domain optimization to suppress large noise in the dynamic calibration process of the extremely low-cost sensor platform. The effectiveness of this method is validated through experiments and simulations. The minimal calibration error (0.46%) is commensurate with the advanced work. By feeding the compensated IMU into the VIO algorithm, the localization accuracy is improved by 9% to 15%. This method improves the performance in the VIO algorithm, which is equipped with the low-cost or poorly calibrated MEMS IMU and reduces the hardware and deployment costs of the system.

In this study, we built a monocular visual inertial odometry (VIO) dataset with a proposed self-calibrating platform for an inertial measurement unit (IMU). For a learning-based method, lack of database is the biggest limitation. One goal of this study is to generate a high-quality VIO database. Meanwhile, a properly calibrated inertial measurement unit (IMU) is critical for improving VIO system accuracy and precision. However, a professional IMU calibration tool is usually expensive and large. The proposed self-calibrating platform shows its benefits in eliminating the manual work in calibration and reducing the cost of the expensive platform, which brings more opportunities to students and researchers. This self-calibrating system was useful for collecting monocular visual inertial odometry datasets while driving. . The calibrated IMU generates 14 numbers at 100Hz: time, 3-axis accelerometer measurements, 3-axis gyroscope measurements, 3 Euler angles, and 4 quaternions. This VIO system is composed of a monocular camera, GoPro Hero 3, an IMU, BNO055, and a GPS, ANT-555. The sensors are mounted on a John Deere Gator™ Full-Size Crossover Utility Vehicle and are operated on an Intel NUC, a mini-size computer. The final dataset contains monocular images, IMU measurements, and GPS outputs.

The recent introduction of powerful embedded graphics processing units (GPUs) has allowed for unforeseen improvements in real-time computer vision applications. It has enabled algorithms to run onboard, well above the standard video rates, yielding not only higher information processing capability, but also reduced latency. This work focuses on the applicability of efficient low-level, GPU hardware-specific instructions to improve on existing computer vision algorithms in the field of visual-inertial odometry (VIO). While most steps of a VIO pipeline work on visual features, they rely on image data for detection and tracking, of which both steps are well suited for parallelization. Especially non-maxima suppression and the subsequent feature selection are prominent contributors to the overall image processing latency. Our work first revisits the problem of non-maxima suppression for feature detection specifically on GPUs, and proposes a solution that selects local response maxima, imposes spatial feature distribution, and extracts features simultaneously. Our second contribution introduces an enhanced FAST feature detector that applies the aforementioned non-maxima suppression method. Finally, we compare our method to other state-of-the-art CPU and GPU implementations, where we always outperform all of them in feature tracking and detection, resulting in over 1000fps throughput on an embedded Jetson TX2 platform. Additionally, we demonstrate our work integrated into a VIO pipeline achieving a metric state estimation at ∼200fps.Code available at: https://github.com/uzh-rpg/vilib

Recently, application of Unmanned Aerial Vehicle (UAV) attract much attention of not only academic but also industrial community. To control and make the most of UAV, state estimation is mandatory and it should work accurately on Real-time on computational-limited onboard PC. Especially, for the cases that GNSS-disabled environments or indoor situation, Visual Inertial Odometry using camera and IMU sensor has been actively studied. Generally, to research and develop the indoor navigation algorithm, high-performanced and super-expensive facilities like motion capture cameras are needed to measure the ground-truth. In this paper, we apply state-of-art Visual Inertial Odometry algorithms for state estimation of UAVs to FlightGoggles simulator which is photorealistic and which reflects UAV's real dynamics. Moreover, we compare the performance and resource utilization of those algorithms.

In the γ radioactive environment, high-energy photons induce degradation of the image sensor, which effects feature detection and tracking in the Visual Inertial Odometry (VIO) algorithm and deteriorates its localization performance. To address this issue, in this work, we propose a monocular VIO method using edge-based point features. To mitigate the effects of radiation noise, firstly, in the image preprocessing module, the median filter is used for real-time image denoising. Secondly, in the data association module, both Shi-Tomasi and edge-based point features are detected. The edge-based point feature is the endpoint or corner point in the salient edge map, which is more robust to radiation noise. Then, the bi-directional motion parallaxes and the RANdom SAmple Consensus (RANSAC) method are exploited to reject outliers. Finally, the point features measurements and Inertial Measurement Unit (IMU) pre-integration measurements are added into a tightly-coupled sliding window optimization VIO framework for localization estimation. The proposed method is verified by synthetic and real γ radioactive environment datasets. The experimental results show that the proposed method achieves more accurate and robust localization than the state-of-the-art VIO approaches in the γ radioactive environments.

In this paper we present a novel multi-stereo visual-inertial odometry (VIO) framework which aims to improve the robustness of a robot’s state estimate during aggressive motion and in visually challenging environments. Our system uses a fixed-lag smoother which jointly optimizes for poses and landmarks across all stereo pairs. We propose a 1-point RANdom SAmple Consensus (RANSAC) algorithm which is able to perform outlier rejection across features from all stereo pairs. To handle the problem of noisy extrinsics, we account for uncertainty in the calibration of each stereo pair and model it in both our front-end and back-end. The result is a VIO system which is able to maintain an accurate state estimate under conditions that have typically proven to be challenging for traditional state-of-the-art VIO systems. We demonstrate the benefits of our proposed multi-stereo algorithm by evaluating it with both simulated and real world data. We show that our proposed algorithm is able to maintain a state estimate in scenarios where traditional VIO algorithms fail.

Vision-based localization is an essential problem for autonomous systems while the performance of vision-based odometry degrades in the challenging scenario. This letter presents S-VIO, an RGB-D visual inertial odometry (VIO) which fully uses multi-sensor measurements (i.e. depth, RGB and IMU), heterogeneous landmarks (i.e. points, lines and planes) and structural regularity of the environment to obtain a robust and accurate localization result. In order to detect the underlying structural regularity of the environment, a two-step Atlanta world inference method is proposed. Leveraging the gravity direction estimated by the VIO system, the proposed algorithm first generates horizontal Atlanta axis hypotheses from a set of recently optimized plane landmarks. Then the following plane landmarks and line clusters are used to filter out the occasionally observed axes based on the persistence of the observation. The remaining axes will survive and be saved in the Atlanta map for future re-observations. Particularly, an efficient mined-and-stabbed (MnS) method is applied to classify the structural line and extract the vanishing point from each line cluster. In addition, a closed-form initialization method for the structural line feature is proposed, which leverages the known direction to obtain a better initial estimation. Integrated with the above novelties, S-VIO is tested on two public real-world RGB-D inertial datasets. Experiments demonstrate that S-VIO has better accuracy and robustness compared to state-of-the-art VIO and RGB-D VIO algorithms. The code and the supplementary material will be available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/oranfire/SVIO</uri> .

Event cameras are bio-inspired vision sensors that output pixel-level brightness changes instead of standard intensity frames. They offer significant advantages over standard cameras, namely a very high dynamic range, no motion blur, and a latency in the order of microseconds. We propose a novel, accurate tightly-coupled visual-inertial odom- etry pipeline for such cameras that leverages their outstanding properties to estimate the camera ego-motion in challenging conditions, such as high-speed motion or high dynamic range scenes. The method tracks a set of features (extracted on the image plane) through time. To achieve that, we consider events in overlapping spatio-temporal windows and align them using the current camera motion and scene structure, yielding motion-compensated event frames. We then combine these feature tracks in a keyframe- based, visual-inertial odometry algorithm based on nonlinear optimization to estimate the camera’s 6-DOF pose, velocity, and IMU biases. The proposed method is evaluated quantitatively on the public Event Camera Dataset [19] and significantly outperforms the state-of-the-art [28], while being computationally much more efficient: our pipeline can run much faster than real-time on a laptop and even on a smartphone processor. Fur- thermore, we demonstrate qualitatively the accuracy and robustness of our pipeline on a large-scale dataset, and an extremely high-speed dataset recorded by spinning an event camera on a leash at 850 deg/s.

We present a multi-camera visual-inertial odometry system based on factor graph optimization which estimates motion by using all cameras simultaneously while retaining a fixed overall feature budget. We focus on motion tracking in challenging environments, such as narrow corridors, dark spaces with aggressive motions, and abrupt lighting changes. These scenarios cause traditional monocular or stereo odometry to fail. While tracking motion with extra cameras should theoretically prevent failures, it leads to additional complexity and computational burden. To overcome these challenges, we introduce two novel methods to improve multi-camera feature tracking. First, instead of tracking features separately in each camera, we track features continuously as they move from one camera to another. This increases accuracy and achieves a more compact factor graph representation. Second, we select a fixed budget of tracked features across the cameras to reduce back-end optimization time. We have found that using a smaller set of informative features can maintain the same tracking accuracy. Our proposed method was extensively tested using a hardware-synchronized device consisting of an IMU and four cameras (a front stereo pair and two lateral) in scenarios including: an underground mine, large open spaces, and building interiors with narrow stairs and corridors. Compared to stereo-only state-of-the-art visual-inertial odometry methods, our approach reduces the drift rate, relative pose error, by up to 80% in translation and 39% in rotation.

In visual odometry applications, tracking of features in a video sequence greatly impacts the accuracy of ego-motion estimation. A robust visual tracking has to take into account either geometric or photometric conditions to exclude outliers in the stage of motion estimation. In this work we propose a novel method to adaptively constrain the matching of features in time sequence. The control of matching is based on two major Euclidean properties, namely scale and angle. The scale constraint refers to the disparity image, and the angular criterion is derived from epipolar geometry. The proposed method is tested with two selected sequences from KITTI visual odometry benchmark datasets. Experimental results indicate promising improvement is attainable over a conventional feature tracker for visual odometry.

PurposeThis paper aims to enable the unmanned aerial vehicles to inspect the surface condition of wind turbine in close range when the global positioning system signal is not reliable, and further improve its intelligence. So a visual-inertial odometry with point and line features is developed.Design/methodology/approachVisual front-end combining point and line features, as well as its purification strategies, are first presented to improve the robustness of feature tracking in low-textured scene and rapidity of segment detector. Additionally, the inertial measurement is integrated between keyframes as constrain to reduce tracking error existed in visual-only system. Second, the graph-based visual-inertial back-end is constructed. To parameterize line features effectively, the infinite line representation not sensitive to outdoor light is employed, in which Plücker and Cayley are selected for line re-projection and nonlinear optimization. Furthermore, Jacobians of the line re-projection errors are analytically derived for better accuracy.FindingsExperiments are performed in various scenes of the wind farm. The results demonstrate that the tight-coupled visual-inertial odometry with point and line features is more precise on all the samples than conventional algorithms in complex wind farm environments. Additionally, the constructed line feature map can be used in the following research for autonomous navigation.Originality/valueThe proposed visual-inertial odometry works robustly in strong electromagnetic interference, low-textured and illumination-change wind farm.

The fusion of visual and inertial odometry has matured greatly due to the complementarity of the two sensors. However, the use of high-quality sensors and powerful processors in some applications is difficult due to size and cost limitations, and there are also many challenges in terms of robustness of the algorithm and computational efficiency. In this work, we present VIO-Stereo, a stereo visual-inertial odometry (VIO), which jointly combines the measurements of the stereo cameras and an inexpensive inertial measurement unit (IMU). We use nonlinear optimization to integrate visual measurements with IMU readings in VIO tightly. To decrease the cost of computation, we use the FAST feature detector to improve its efficiency and track features by the KLT sparse optical flow algorithm. We also incorporate accelerometer bias into the measurement model and optimize it together with other variables. Additionally, we perform circular matching between the previous and current stereo image pairs in order to remove outliers in the stereo matching and feature tracking steps, thus reducing the mismatch of feature points and improving the robustness and accuracy of the system. Finally, this work contributes to the experimental comparison of monocular visual-inertial odometry and stereo visual-inertial odometry by evaluating our method using the public EuRoC dataset. Experimental results demonstrate that our method exhibits competitive performance with the most advanced techniques.

State of the art visual-inertial odometry approaches suffer from the requirement of high gradients and sufficient visual texture. Even direct photometric approaches select a subset of the image with high-gradient areas and ignore smooth gradients or generally low-textured areas. In this work, we show that taking all image information (i.e. every single pixel) enables visual-inertial odometry even on areas with very low texture and smooth gradients, inherently interpolating and estimating the scene with no texture based on its informative surrounding. This information propagation is only possible as we estimate all states and their uncertainties (robot pose, extrinsic sensor calibration, and scene depth) jointly in a fully dense filter framework. Our complexity reduction approach enables real-time execution despite the large size of the state vector. Compared to our previous basic feasibility study on this topic, this work includes higher order covariance propagation and improved state handling for a significant performance gain, thorough comparisons to state-of-the-art algorithms, larger mapping components with uncertainty, self-calibration capability, and real-data tests.

Tracking the pose of a robot has been gaining importance in the field of Robotics, e.g., paving the way for robot navigation. In recent years, monocular visual–inertial odometry (VIO) is widely used to do the pose estimation due to its good performance and low cost. However, VIO cannot estimate the scale or orientation accurately when robots move along straight lines or circular arcs on the ground. To address the problem, in this paper we take the wheel encoder into account, which can provide us with stable translation information as well as small accumulated errors and momentary slippage errors. By jointly considering the kinematic constraints and the planar moving features, an odometry algorithm tightly coupled with monocular camera, IMU, and wheel encoder is proposed to get robust and accurate pose sensing for mobile robots, which mainly contains three steps. First, we present the wheel encoder preintegration theory and noise propagation formula based on the kinematic mobile robot model, which is the basis of accurate estimation in backend optimization. Second, we adopt a robust initialization method to obtain good initial values of gyroscope bias and visual scale in reality, by making full use of the camera, IMU and wheel encoder measurements. Third, we bound the high computation complexity with a marginalization strategy that conditionally eliminates unnecessary measurements in the sliding window. We implement a prototype and several extensive experiments showing that our system can achieve robust and accurate pose estimation, in terms of the scale, orientation and location, compared with the state-of-the-art.

Simultaneous localization and mapping (SLAM) is a key technique for autonomous navigation of a robot in unknown environments. Recently, many researchers focus on the algorithm optimization of visual-inertial odometry (VIO) as a real-time SLAM method. In this paper, different from pure algorithm research, we propose a novel low-cost hardware solution based on the system on a chip (SoC) to achieve real-time performance and low power consumption of VIO. We design a novel hardware architecture to accelerate a popular VIO algorithm, with the time-costly image feature processing computed on FPGA and floating-point numerical computation been done on ARM. On the algorithm side, we fuse odometry information from the two wheels of the ground robot in a unified optimization method, which results in accurate metric scale estimation of the VIO. The experimental results show that our design and implementation have superior performance for ground robot applications, allowing real-time localization of a robot with a very low-cost hardware for computation in complex environments.