360-VIO: A Robust Visual–Inertial Odometry Using a 360° Camera

Man-made buildings exhibit structural regularity, which can provide strongly geometrical constraints for Visual-Inertial Odometry (VIO) systems. To make full use of the structural information, we propose a new structural regularity aided VIO with novel coordinate alignment and line triangulation under Manhattan world assumption. The proposed VIO system is built upon OpenVINS [1] and partly based on our previous work [2]. The proposed coordinate alignment method makes the Jacobians and reprojection errors become concise but also makes the required computation number for transformations decrease. In addition, a novel structural line triangulation method is provided, in which the global orientation of a structural line is used to refine its 3D position. All the novelties result in a more accurate and fast VIO system. The system is tested on EuRoC MAV dataset and a self-collected dataset. Experimental results demonstrate that the proposed method obtains better accuracy compared with state-of-the-art (SOTA) point-based systems (VINS-Mono [3] and OpenVINS [1]), point-line-based systems (PL-VINS [4] and Wei et al. [2]), and structural line-based system (StructVIO [5]). Notably, the self-collected dataset is recorded on Manhattan world scenes, and is also full of challenging weak texture and motion blur situations. On the dataset, the accuracy of our method is increased by 40.7% compared with the SOTA point-line-based systems.

Tracking the pose of a robot has been gaining importance in the field of Robotics, e.g., paving the way for robot navigation. In recent years, monocular visual–inertial odometry (VIO) is widely used to do the pose estimation due to its good performance and low cost. However, VIO cannot estimate the scale or orientation accurately when robots move along straight lines or circular arcs on the ground. To address the problem, in this paper we take the wheel encoder into account, which can provide us with stable translation information as well as small accumulated errors and momentary slippage errors. By jointly considering the kinematic constraints and the planar moving features, an odometry algorithm tightly coupled with monocular camera, IMU, and wheel encoder is proposed to get robust and accurate pose sensing for mobile robots, which mainly contains three steps. First, we present the wheel encoder preintegration theory and noise propagation formula based on the kinematic mobile robot model, which is the basis of accurate estimation in backend optimization. Second, we adopt a robust initialization method to obtain good initial values of gyroscope bias and visual scale in reality, by making full use of the camera, IMU and wheel encoder measurements. Third, we bound the high computation complexity with a marginalization strategy that conditionally eliminates unnecessary measurements in the sliding window. We implement a prototype and several extensive experiments showing that our system can achieve robust and accurate pose estimation, in terms of the scale, orientation and location, compared with the state-of-the-art.

In this study, we built a monocular visual inertial odometry (VIO) dataset with a proposed self-calibrating platform for an inertial measurement unit (IMU). For a learning-based method, lack of database is the biggest limitation. One goal of this study is to generate a high-quality VIO database. Meanwhile, a properly calibrated inertial measurement unit (IMU) is critical for improving VIO system accuracy and precision. However, a professional IMU calibration tool is usually expensive and large. The proposed self-calibrating platform shows its benefits in eliminating the manual work in calibration and reducing the cost of the expensive platform, which brings more opportunities to students and researchers. This self-calibrating system was useful for collecting monocular visual inertial odometry datasets while driving. . The calibrated IMU generates 14 numbers at 100Hz: time, 3-axis accelerometer measurements, 3-axis gyroscope measurements, 3 Euler angles, and 4 quaternions. This VIO system is composed of a monocular camera, GoPro Hero 3, an IMU, BNO055, and a GPS, ANT-555. The sensors are mounted on a John Deere Gator™ Full-Size Crossover Utility Vehicle and are operated on an Intel NUC, a mini-size computer. The final dataset contains monocular images, IMU measurements, and GPS outputs.

We present a multi-camera visual-inertial odometry system based on factor graph optimization which estimates motion by using all cameras simultaneously while retaining a fixed overall feature budget. We focus on motion tracking in challenging environments, such as narrow corridors, dark spaces with aggressive motions, and abrupt lighting changes. These scenarios cause traditional monocular or stereo odometry to fail. While tracking motion with extra cameras should theoretically prevent failures, it leads to additional complexity and computational burden. To overcome these challenges, we introduce two novel methods to improve multi-camera feature tracking. First, instead of tracking features separately in each camera, we track features continuously as they move from one camera to another. This increases accuracy and achieves a more compact factor graph representation. Second, we select a fixed budget of tracked features across the cameras to reduce back-end optimization time. We have found that using a smaller set of informative features can maintain the same tracking accuracy. Our proposed method was extensively tested using a hardware-synchronized device consisting of an IMU and four cameras (a front stereo pair and two lateral) in scenarios including: an underground mine, large open spaces, and building interiors with narrow stairs and corridors. Compared to stereo-only state-of-the-art visual-inertial odometry methods, our approach reduces the drift rate, relative pose error, by up to 80% in translation and 39% in rotation.

In the γ radioactive environment, high-energy photons induce degradation of the image sensor, which effects feature detection and tracking in the Visual Inertial Odometry (VIO) algorithm and deteriorates its localization performance. To address this issue, in this work, we propose a monocular VIO method using edge-based point features. To mitigate the effects of radiation noise, firstly, in the image preprocessing module, the median filter is used for real-time image denoising. Secondly, in the data association module, both Shi-Tomasi and edge-based point features are detected. The edge-based point feature is the endpoint or corner point in the salient edge map, which is more robust to radiation noise. Then, the bi-directional motion parallaxes and the RANdom SAmple Consensus (RANSAC) method are exploited to reject outliers. Finally, the point features measurements and Inertial Measurement Unit (IMU) pre-integration measurements are added into a tightly-coupled sliding window optimization VIO framework for localization estimation. The proposed method is verified by synthetic and real γ radioactive environment datasets. The experimental results show that the proposed method achieves more accurate and robust localization than the state-of-the-art VIO approaches in the γ radioactive environments.

Visual-inertial odometry (VIO) is an important technology for autonomous\nrobots with power and payload constraints. In this paper, we propose a novel\napproach for VIO with stereo cameras which integrates and calibrates the\nvelocity-control based kinematic motion model of wheeled mobile robots online.\nIncluding such a motion model can help to improve the accuracy of VIO. Compared\nto several previous approaches proposed to integrate wheel odometer\nmeasurements for this purpose, our method does not require wheel encoders and\ncan be applied when the robot motion can be modeled with velocity-control based\nkinematic motion model. We use radial basis function (RBF) kernels to\ncompensate for the time delay and deviations between control commands and\nactual robot motion. The motion model is calibrated online by the VIO system\nand can be used as a forward model for motion control and planning. We evaluate\nour approach with data obtained in variously sized indoor environments,\ndemonstrate improvements over a pure VIO method, and evaluate the prediction\naccuracy of the online calibrated model.\n

Nowadays, multi-sensor fusion technology is a fundamental prerequisite to achieve highly autonomous robots and robustness. Many studies have been conducted, such as visual–inertial odometry (VIO), integrated navigation, and LiDAR–inertial odometry. Typically, for VIO, gratifying results have been achieved, ascribing to the complementary sensing capabilities of inertial measurement units (IMUs) and cameras. However, this work mainly focuses on the fusion of visual and inertial data, while the IMU error is less considered, especially for low-cost or poorly calibrated microelectromechanical system (MEMS) IMU. Such errors may have a significant effect on the VIO performance. In this study, we compensated for the IMU using camera assistance. The key characteristic of the method is that we optimize the compensation parameters (scale factor) from coarse to fine by combining the time domain with the frequency domain. The proposed method is to use the time-domain and frequency-domain optimization to suppress large noise in the dynamic calibration process of the extremely low-cost sensor platform. The effectiveness of this method is validated through experiments and simulations. The minimal calibration error (0.46%) is commensurate with the advanced work. By feeding the compensated IMU into the VIO algorithm, the localization accuracy is improved by 9% to 15%. This method improves the performance in the VIO algorithm, which is equipped with the low-cost or poorly calibrated MEMS IMU and reduces the hardware and deployment costs of the system.

The traditional visual inertial simultaneous localisation and mapping system does not fully consider the dynamic objects in the scene, which can reduce the quality of visual feature point matching. In addition, dynamic objects in the scene can cause illumination changes which reduce the performance of the visual front end and loop closure detection of the system. To address this problem, this study combines 3D light detection and ranging (LiDAR), camera, and inertial measurement units in a tightly coupled manner to estimate the pose of mobile robots, thereby proposing a robust LiDAR visual inertial odometry that can effectively filter out dynamic feature points. In addition, a dynamic feature point detection algorithm with attention mechanism is introduced for target detection and optical flow tracking. In experimental analyses on public datasets and real indoor scenes, the proposed method improved the accuracy and robustness of pose estimation in scenes with dynamic objects and varying illumination compared with traditional methods.

Vision-based localization is an essential problem for autonomous systems while the performance of vision-based odometry degrades in the challenging scenario. This letter presents S-VIO, an RGB-D visual inertial odometry (VIO) which fully uses multi-sensor measurements (i.e. depth, RGB and IMU), heterogeneous landmarks (i.e. points, lines and planes) and structural regularity of the environment to obtain a robust and accurate localization result. In order to detect the underlying structural regularity of the environment, a two-step Atlanta world inference method is proposed. Leveraging the gravity direction estimated by the VIO system, the proposed algorithm first generates horizontal Atlanta axis hypotheses from a set of recently optimized plane landmarks. Then the following plane landmarks and line clusters are used to filter out the occasionally observed axes based on the persistence of the observation. The remaining axes will survive and be saved in the Atlanta map for future re-observations. Particularly, an efficient mined-and-stabbed (MnS) method is applied to classify the structural line and extract the vanishing point from each line cluster. In addition, a closed-form initialization method for the structural line feature is proposed, which leverages the known direction to obtain a better initial estimation. Integrated with the above novelties, S-VIO is tested on two public real-world RGB-D inertial datasets. Experiments demonstrate that S-VIO has better accuracy and robustness compared to state-of-the-art VIO and RGB-D VIO algorithms. The code and the supplementary material will be available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/oranfire/SVIO</uri> .

This dissertation presents a comprehensive approach to addressing the challenges associated with the control and navigation of Unmanned Aerial Vehicles (UAVs), specifically focusing on robustness under GPS-denied, turbulent, and dark indoor conditions with various disturbances. Initial efforts involved designing and theoretically validating Sliding Mode Controllers (SMC) with Disturbance Observers using MATLAB/SIMULINK, optimized using offline Particle Swarm Optimization (PSO). In order to evaluate their performance, a lightweight simulator was developed to simulate both quadcopter and hexacopter configurations under the PX4 architecture. This simulator served to validate and compare the optimized SMC against PSO-optimized PID controllers. The results of these comparisons demonstrated the superior adaptability of the SMC, especially in environments characterized by significant sensor noise and disturbances. Subsequent practical indoor tests were conducted with real quadcopters, which provided additional verification of the robustness and reliability of the enhanced control theory with the actual UAVs under dynamic disturbances. To further evaluate the compatibility and adaptability of the SMC, this dissertation establishes a framework for interfacing with high-performance STMH7 chips, serving as the UAV microcontroller unit (MCU). This framework facilitates the connection of various sensors and provides guidance for constructing different quadcopter frames. Moreover, it enables the comparison of different flight controllers on distinct air-frames. Concurrently, the optimized SMC allows for remote dynamic parameter tuning during flight. In pursuit of improved localization capacity, the dissertation also explores the integration of an Intel Realsense D435i camera with an Inertial Measurement Unit (IMU) for Visual-Inertial Odometry (VIO) fusion within the Robot Operating System (ROS) environment. This integration employs an onboard companion computer for real-time VIO estimation and flight command. The effectiveness of the VIO algorithm is verified through both simulations and real-world experiments, highlighting its ability to enhance UAV localization and navigation. Additionally, a visual inertial odometry method for UAV Simultaneous Localization and Mapping (vSLAM) is tested and validated in both simulated and real drone environments. Furthermore, the dissertation introduces an innovative hybrid filtered multi-directional radar inertial odometry solution denoted as Hybrid-MRIO. This Error State Extended Kalman Filter (ES-EKF) based Radar Inertial Odometry (RIO) approach is implemented within the open-source PX4 autopilot system, enhancing UAV navigation accuracy and performance, and serves as a robust navigation alternative, particularly in challenging environments where traditional navigation systems prove ineffective, such as dark or smoky conditions. This solution integrates data from multiple software synchronized high-resolution Frequency-Modulated Continuous-Wave (FMCW) mmWave radars with an IMU. A subsequent 4D (x, y, z, doppler) mmWave radar SLAM (rSLAM) system has been developed, including three modules: front-end, loop detection module, and back-end. In the front-end, radar ego-velocity is utilized for states estimation and dynamic object removal, and a point cloud registration-based approach known as APDGICP (Adaptive Probability Distribution-GICP) is employed for keyframe detection. The loop detection module utilizes the intensity scan context to identify potential loop closure candidates. In the back-end, a pose graph is constructed, integrating RIO estimated odometry and identified loop closures for better localization. To develop fully-autonomous UAVs with their decision making modules, this dissertation details the development of a custom AI-powered fully-autonomous quadcopter equipped with companion computers and stereo IR cameras. Additionally, this dissertation discusses a swarm of small Unmanned Aerial Systems (SUAS) interconnected through a software-defined communications network. These drones demonstrated their capacity to independently/collaboratively execute a wide range of tasks, including target search, detection, identification, classification, tracking, and following, both in simulated and real-world scenarios. The system exhibits advanced collision avoidance capabilities and the ability to strategically respond to dynamic scenarios, such as changes in target behavior, highlighting its effectiveness in managing dynamic and challenging situations without the need for continuous human intervention.--Author's abstract

This paper proposes a framework for tightly coupled monocular visual-inertial odometry (VIO) via epipolar constraints, which achieves accurate and effective 6 degrees-of-freedom (DOF) state estimation. After extracting and tracking of features, initial values of system state are then estimated for the VIO. The algorithm proposed establishes visual-inertial odometry atop a factor graph, which estimates the pose, velocity and bias of the IMU and the external between camera and IMU, allowing multiple measurements to be captured by different sensors as factors. The performance of the algorithm proposed is testified through the EuRoC datasets, and compared against other state-of-art VIO algorithms. The experimental results indicate that the introduction of epipolar constraints improves the state estimation accuracy effectively.

VIO-DualProNet: Visual-inertial odometry with learning based process noise covariance

Visual-inertial odometry with low-cost visual and inertial sensors can provide accurate navigation results in GNSS-denied environment. However, the mainstream methods currently suffer from the inevitable accumulation of position and heading errors over long-term operation. Inspired by the compound eyes of insects such as ants, polarization skylight compass can obtain the accurate orientation from the sky polarization pattern without cumulative error. This letter proposes a complete polarization-based visual-inertial odometry (POL-VIO) for challenge environment. This system offers a visual-inertial odometry enhanced by a skylight compass, which reduces heading drift and provides high-accuracy state estimation under complex environment. Additionally, it achieves online spatio-temporal calibration of the polarization light compass, incorporating ambiguity correction and horizontal angle compensation in polarized light orientation. Besides, we utilize the degree of skylight polarization to determine whether compass measurements are reliable. The polarization skylight compass enhances the reliability of heading constraints in complex environments by utilizing the Angle of Polarization information. Results from real-world experiments substantiate that our system can achieve a notable improvement in positioning and orientation accuracy under significant obstructed sky observations after long time of operation. On three datasets with a total length exceeding 7 km, the average positioning error of POL-VIO is less than 2 m/km.

We propose Super Odometry, a high-precision multi-modal sensor fusion framework, providing a simple but effective way to fuse multiple sensors such as LiDAR, camera, and IMU sensors and achieve robust state estimation in perceptually-degraded environments. Different from traditional sensor-fusion methods, Super Odometry employs an IMU-centric data processing pipeline, which combines the advantages of loosely coupled methods with tightly coupled methods and recovers motion in a coarse-to-fine manner. The proposed framework is composed of three parts: IMU odometry, Visual-inertial odometry, and LiDAR-inertial odometry. The Visual-inertial odometry and LiDAR-inertial odometry provide the pose prior to constrain the IMU bias and receive the motion prediction from IMU odometry. To ensure high performance in real-time, we apply a dynamic octree that only consumes 10% of the running time compared with a static KD-tree. The proposed system was deployed on drones and ground robots, as part of Team Explorer's effort to the DARPA Subterranean Challenge where the team won 1 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">st</sup> and 2 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">nd</sup> place in the Tunnel and Urban Circuits <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> , respectively.

Recently, application of Unmanned Aerial Vehicle (UAV) attract much attention of not only academic but also industrial community. To control and make the most of UAV, state estimation is mandatory and it should work accurately on Real-time on computational-limited onboard PC. Especially, for the cases that GNSS-disabled environments or indoor situation, Visual Inertial Odometry using camera and IMU sensor has been actively studied. Generally, to research and develop the indoor navigation algorithm, high-performanced and super-expensive facilities like motion capture cameras are needed to measure the ground-truth. In this paper, we apply state-of-art Visual Inertial Odometry algorithms for state estimation of UAVs to FlightGoggles simulator which is photorealistic and which reflects UAV's real dynamics. Moreover, we compare the performance and resource utilization of those algorithms.