Accurate and robust odometry by fusing monocular visual, inertial, and wheel encoder

In this study, we built a monocular visual inertial odometry (VIO) dataset with a proposed self-calibrating platform for an inertial measurement unit (IMU). For a learning-based method, lack of database is the biggest limitation. One goal of this study is to generate a high-quality VIO database. Meanwhile, a properly calibrated inertial measurement unit (IMU) is critical for improving VIO system accuracy and precision. However, a professional IMU calibration tool is usually expensive and large. The proposed self-calibrating platform shows its benefits in eliminating the manual work in calibration and reducing the cost of the expensive platform, which brings more opportunities to students and researchers. This self-calibrating system was useful for collecting monocular visual inertial odometry datasets while driving. . The calibrated IMU generates 14 numbers at 100Hz: time, 3-axis accelerometer measurements, 3-axis gyroscope measurements, 3 Euler angles, and 4 quaternions. This VIO system is composed of a monocular camera, GoPro Hero 3, an IMU, BNO055, and a GPS, ANT-555. The sensors are mounted on a John Deere Gator™ Full-Size Crossover Utility Vehicle and are operated on an Intel NUC, a mini-size computer. The final dataset contains monocular images, IMU measurements, and GPS outputs.

The precise measuring of vehicle location has been a critical task in enhancing the autonomous driving in terms of intelligent decision making and safe transportation. Internet of Vehicles ( IoV ) is an important infrastructure in support of autonomous driving, allowing real-time road information exchanging and sharing for localizing vehicles. Global positioning System ( GPS ) is widely used in the traditional IoV system. GPS is unable to meet the key application requirements of autonomous driving due to meter level error and signal deterioration. In this article, we propose a novel solution, named Semi-Direct Monocular Visual-Inertial Odometry using Point and Line Features ( SDMPL-VIO ) for precise vehicle localization. Our SDMPL-VIO model takes advantage of a low-cost Inertial Measurement Unit ( IMU ) and monocular camera, using them as the sensor to acquire the surrounding environmental information. Visual-Inertial Odometry ( VIO ), taking into account both point and line features, is proposed, which is able to deal with both weak texture and dynamic environment. We use a semi-direct method to deal with keyframes and non-keyframes, respectively. Dual sliding window mechanisms can effectively fuse point-line and IMU information. To evaluate our SDMPL-VIO system model, we conduct extensive experiments on both an indoor dataset (i.e., EuRoC) and an outdoor dataset (i.e., KITTI) from the real-world applications, respectively. The experimental results show that the accuracy of SDMPL-VIO proposed by us is better than the mainstream VIO system at present. Especially in the weak texture of the datasets, fast-moving datasets, and other challenging datasets, SDMPL-VIO has a relatively high robustness.

We propose and demonstrate an efficient pose estimation method, specifically small tilt angle variation, for visual servo control system design using a monocular camera, eliminating the need for iterative computations. This paper addresses the challenges associated with accurately estimating the relative pose between a chaser, equipped with a monocular camera, and a target that provides visual cues, particularly in scenarios where the captured image shows only minimal variations in the target's tilt angle. Traditional approaches often rely on direct feedback methods from image data or designs for pose estimation and control separately, necessitating computationally expensive iterations for pose estimation or complex implementation of visual servoing. To address these challenges, we adopt an extended Kalman filter, known as a robust nonlinear observer, and propose a non‐iterative approach that integrates motion dynamics models and control inputs into pose estimation from image data. This approach enables efficient integration of pose estimation and control solely relying on a monocular camera, reducing computational costs and improving robustness. Experimental validation using the ball‐and‐plate system demonstrates the effectiveness of our proposed method. In our setup, the camera is deliberately positioned directly above the pose estimation target, creating a scenario with minimal image variation in the tilt angle. Despite this condition, our approach achieves efficient and accurate pose estimation, demonstrating its feasibility for high‐speed pose estimation and control without iterative computations.

For robot-assisted assembly of complex optical systems, the alignment is facilitated by an accurate pose estimation of its components. However, wavefront-based pose estimation is typically ill-conditioned due to the inherent geometry of conventional industrially manufactured optical components. Therefore, we propose a novel approach in this paper to increase wavefront-based pose estimation accuracy via the design of freeform optics. For this purpose, an optimization problem is derived that parameterizes the component's surfaces by a predetermined freeform surface model. To show the efficacy of our approach, we provide simulation results to compare the pose estimation accuracy for a variety of optical designs. As an application example for the resulting improved pose estimation, a hand-eye calibration of a wavefront sensor is performed. This calibration originates from the field of robotics and represents the identification of a sensor coordinate system with respect to a global reference frame. For quantitative evaluation, the calibrating results are first presented with the aid of simulation data. Finally, the practical feasibility is demonstrated using a conventional industrial robot and additively manufactured freeform lenses.

State of the art visual-inertial odometry approaches suffer from the requirement of high gradients and sufficient visual texture. Even direct photometric approaches select a subset of the image with high-gradient areas and ignore smooth gradients or generally low-textured areas. In this work, we show that taking all image information (i.e. every single pixel) enables visual-inertial odometry even on areas with very low texture and smooth gradients, inherently interpolating and estimating the scene with no texture based on its informative surrounding. This information propagation is only possible as we estimate all states and their uncertainties (robot pose, extrinsic sensor calibration, and scene depth) jointly in a fully dense filter framework. Our complexity reduction approach enables real-time execution despite the large size of the state vector. Compared to our previous basic feasibility study on this topic, this work includes higher order covariance propagation and improved state handling for a significant performance gain, thorough comparisons to state-of-the-art algorithms, larger mapping components with uncertainty, self-calibration capability, and real-data tests.

Due to occlusion, appearance and disappearance of points, and noise distribution in image data, no matter what algorithms are used to register a 3D model and its projective image, they are always likely to introduce false matches. In this paper, we propose a novel algorithm to evaluate the established correspondences for more accurate camera pose estimation. The novel algorithm is based on two novel rigid motion constraints that represent necessary conditions for any pair of a 3D and a 2D point to represent a real 3D-2D correspondence. We then prove that as long as 3D-2D point matches satisfy the two novel rigid motion constraints, they are theoretically guaranteed to represent real correspondences. A large number of experiments based on both computer simulation and real images have indeed demonstrated that the proposed algorithm is accurate and robust for the evaluation of established correspondences and thus, leading to more accurate and robust camera pose estimation.

Combining camera, IMU and wheel encoder is a wise choice for car positioning because of the low cost and complementarity of the sensors. We propose a novel extended visual-inertial odometry algorithm tightly fusing data from the above three sensors. Firstly we propose an IMU-odometer pre-integration approach utilizing complete IMU measurements and wheel encoder readings, to make scale estimation more accurate in subsequent 4-degrees of freedom (DoF) optimization. Secondly we develop an original initialization module where encoder readings are fully utilized to refine gravity direction and provide an initial value for camera pose in real scale. Thirdly, we design a computationally efficient online extrinsic calibration method by fixing the linearization point for the rotational component of IMU-odometer extrinsic parameters, which is deployed depending on the convergence of accelerometer bias. Experimental results prove the robustness of our initialization module and the accuracy of the whole trajectory, as well as the improvement brought about by online extrinsic calibration. Our program can also run on an Nvidia Jetson TX2 module in real time.

In this paper, we propose a new scheme based on the Sampson distance (SD) to describe visual feature residuals for visual-inertial odometry (VIO). Unlike the epipolar-constraint-based SD for visual odometry (VO), the proposed SD is formulated based on the perspective projection constraint. We proved in theory that the proposed SD retains the good properties of those earlier SD criteria in the literature of VO and it represents a visual feature residual more accurately than the prevailing transfer distance (TD) in existing VIO methods. We formulate three distance criteria, including TD, reprojection error (RE), and SD, and compared their performances by simulation. The results show that the SD is much more accurate than the TD and it is a very accurate estimate of the gold standard criteria—RE. Based on the SD, we modified VINS-Mono by replacing its TD-based visual residuals with the SD-based residuals and study the SD's efficacy in pose estimation by experiments with several public datasets. The results reveal that the SD-based VINS-Mono has a substantial improvement over the original VINS-Mono in pose estimation accuracy. This indicates that the SD is a better distance criterion than the TD for representing visual feature residuals. The proposed SD may find its applications to broader areas in computer vision and robotics.

The practical application of machine perception to support physical manipulation in unstructured environments remains a barrier to the development of intelligent robotic systems. Recently, great progress has been made by the large-scale machine perception community, but these methods have made few contributions to the applied robotic perception. This is in part because such large-scale systems are designed to recognize category labels of large numbers of objects from a single image, rather than highly accurate, efficient, and robust pose estimation in environments for which a robot has reliable prior knowledge. In this paper, we illustrate the potential for synergistic integration of modern computer vision methods into robotics by augmenting a RANSAC-based registration method with a state-of-the art semantic segmentation algorithm. We detail a convolutional architecture for semantic labeling of the scene, modified to operate efficiently using integral images. We combine this labeling with two novel scene parsing variants of RANSAC, and show, on a new RGB-D dataset that contains complex configurations of textureless and highly specular objects, that our method demonstrates improved performance of pose estimation over the unaugmented algorithms.

In this paper, a robust pose (i.e., position and orientation) estimation algorithm using two-views captured by a calibrated monocular camera is presented. A collection of pose hypotheses is obtained when more than the minimum number of feature points required to uniquely identify a pose are available in both the images. The pose hypotheses - unit quaternion and unit translation vectors - lie on the S3 and S2 manifolds in the Euclidean 4-space and 3-space, respectively. Probability density function (pdf) of the rotation and translation pose hypotheses is evaluated by gridding the unit spheres where a robust, coarse pose estimate is identified at the mode of the pdf. Further, a "refining" pdf of the geodesic distance from the coarse pose estimate is constructed for the hypotheses within a grid containing the coarse estimate. A refined pose estimate is obtained by averaging the low-noise hypotheses in the neighbourhood of the mode of refining pdf. Pose estimation results of the proposed method are compared with RANSAC and nonlinear mean-shift (NMS) algorithms using the Oxford Corridor sequence and the robustness to feature outliers, image noise rejection, and scalability to number of features is analyzed using the synthetic data experiments. Processing time comparison with the RANSAC and NMS algorithms indicate that the deterministic time requirement of the proposed and NMS algorithms is amenable to a variety of visual servo control applications.

Visual–inertial odometry (VIO) is an important component for robots working in industrial environments to obtain accurate and robust pose estimation. In this article, we introduce a novel VIO algorithm designed specifically for 360° cameras, exploiting their wide field of view. We build our algorithm based on an EKF-based filtering framework and propose a novel measurement model for 360° camera leveraging the reprojection error on the tangent plane of the spherical surface. By this, our 360-VIO effectively mitigates errors caused by the image distortion and fully exploit the omnidirectional observation, resulting in superior accuracy and robustness toward illumination changes and fast camera movements. To the best of our knowledge, no public 360-based visual–inertial dataset is currently available. To address this gap, we have constructed a novel 360° camera dataset under different challenging environments to evaluate the performance of our proposed algorithm. The results of our experiments demonstrate the impressive robustness and accuracy of our method compared with other state-of-the-art VIO methods across diverse and challenging environments. To facilitate future research on 360-VIO, we will release the method, self-collected dataset and the calibration tool at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Gatsby23/360-VIO</uri> .

We developed a fast and robust head pose and gaze estimation system. This system can detect facial feature points and estimate 3D pose angles and gaze direction in various conditions including changes in facial expression, partial occlusion, etc. The system needs only one face image as input and doesn't need any special devices such as blinking LED or stereo camera. Moreover, no calibration process is needed. It shows 95% of head pose estimation accuracy and 81% of gaze estimation accuracy (when the error margin is 15 degrees). The processing time is approximately 15ms/frame (Pentium4 3.2 GHz). Acceptable range of facial pose is within +/- 60 degrees in yaw (left-right) and within +/- 30 degrees in pitch (up-down).

The authors focus on the robust 3D-3D single and multiple pose estimation problems. The robust 3D-3D single pose estimation problem was formulated by R. M. Haralick et al. (1989) as a general regression in terms of a contaminated Gaussian error noise model. The robust 3D-3D multiple pose estimation problem appears much more difficult. This problem is formulated as a series of general regressions involving successively decreasing data size, with each regression related to one particular pose of interest. Since the first few regressions may carry a highly contaminated Gaussian error noise model, a very robust estimator called the MF-estimator, given by X. Zhuang et al. (1992), is used to solve each regression for each desired pose estimation. Computer experiments with real imagery and simulated data were conducted and results are promising. >

Visual-inertial odometry (VIO), which fuses noisy inertial readings and camera measurements to provide 3D motion tracking, is a foundational component in many autonomous applications. With the increasing use of next-generation edge devices (e.g., AR/VR devices, nano drones, and mobile robotics) that are constrained by limited power, resources, and multitasking demands, balancing computational efficiency and accuracy in VIO estimators has become more critical than ever. Historically, state estimation algorithms have been developed using either optimization or filtering-based methods, with the key distinction being the ability to relinearize measurements and correct state estimates iteratively. It has been widely claimed that iterative methods improve accuracy by allowing for the reduction of error through relinearization at a higher computational demand. Conversely, filtering methods are more efficient but may suffer from significant linearization errors. However, these trade-offs have not been thoroughly examined in the context of visual-inertial motion tracking. In this paper, we conduct the first comprehensive study on the impact of iterative algorithms in sliding-window VIO. We analyze the relinearization of IMU and camera measurements separately, providing insights into how each affects system performance. By considering key factors such as system observability and measurement processes, we offer a deeper understanding of VIO estimator behavior. Our findings, backed by real-world tests, offer practical guidelines for balancing accuracy and efficiency, helping practitioners determine when to prioritize iterative methods or simpler filtering approaches while encouraging researchers and engineers to rethink VIO design for optimal resource allocation.

Visual-inertial odometry (VIO) is an important technology for autonomous\nrobots with power and payload constraints. In this paper, we propose a novel\napproach for VIO with stereo cameras which integrates and calibrates the\nvelocity-control based kinematic motion model of wheeled mobile robots online.\nIncluding such a motion model can help to improve the accuracy of VIO. Compared\nto several previous approaches proposed to integrate wheel odometer\nmeasurements for this purpose, our method does not require wheel encoders and\ncan be applied when the robot motion can be modeled with velocity-control based\nkinematic motion model. We use radial basis function (RBF) kernels to\ncompensate for the time delay and deviations between control commands and\nactual robot motion. The motion model is calibrated online by the VIO system\nand can be used as a forward model for motion control and planning. We evaluate\nour approach with data obtained in variously sized indoor environments,\ndemonstrate improvements over a pure VIO method, and evaluate the prediction\naccuracy of the online calibrated model.\n