IMU/Vehicle Calibration and Integrated Localization for Autonomous Driving

Event cameras, inspired by biological vision, offer high dynamic range, excellent temporal resolution, and minimal data redundancy. Precise calibration of event camera systems is essential for applications such as 3D vision. The cessation of extra gray frame production in popular models like the dynamic vision sensor (DVS) poses significant challenges to achieving high-accuracy calibration. Traditional calibration methods, which rely on motion to trigger events, are prone to movement-related errors. This paper introduces a motion-error-free calibration method for event cameras using a flashing target produced by a standard electronic display that elicits high-fidelity events. We propose an improved events-accumulator to reconstruct gray images with distinct calibration features and develop an optimization method that adjusts camera parameters and control point positions simultaneously, enhancing the calibration accuracy of event camera systems. Experimental results demonstrated higher accuracy compared to the traditional motion-based calibration method (reprojection error: 0.03 vs. 0.96 pixels). The 3D reconstruction error remained around 0.15 mm, significantly improving over the motion-based method's 8.00 mm. Additionally, the method's adaptability for hybrid calibration in event-based stereovision systems was verified (e.g., with frame cameras or projectors).

Biplanar deflectometry based on transparent screens (TSs) does not necessitate any movement of the screen or the addition of a spectroscope, mitigating the issues of complex calibration processes and reduced accuracy. This method usually only requires the calibration of the thickness <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$d$ </tex-math></inline-formula> and refractive index <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\eta $ </tex-math></inline-formula> of the TS. However, traditional refraction parameter calibration methods require repeatedly moving the camera or standard flat mirror (Sfm), making the calibration process complex; to address this issue, the current article proposes a novel calibration method for the refractive parameters of TSs, which analyzes the optical path model of camera detection light refracting on TSs through inverse ray tracing. The mathematical model of the incident vector related to TS thickness <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$d$ </tex-math></inline-formula> and refractive index <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\eta $ </tex-math></inline-formula> is also constructed. The refraction parameters of the TS can be obtained by inputting a known incident vector (constructed by a Sfm) into the model. This method does not necessitate repeated movement of the camera or the Sfm; the refraction parameters <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$d$ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\eta $ </tex-math></inline-formula> of the TS can be calibrated in the calibration stage, greatly reducing the calibration complexity. Finally, a spherical mirror with a radius of curvature of 1000 mm is measured. The root-mean-square error (RMSE) is <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$3.56~\mu \text{m}$ </tex-math></inline-formula> when considering the effect of refraction, and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$8.55~\mu \text{m}$ </tex-math></inline-formula> without considering the effect of refraction, which proves the effectiveness of the proposed method.

This article presents a high-precision single-camera inertial measurement unit (IMU) extrinsic calibration method by tightly fusing the visual information from other cameras. Specifically, multiple additional cameras are added to the monocular camera-IMU system for assisting calibration as we theoretically prove that more cameras used in calibration can lead to smaller lower bound on the covariance of the estimated extrinsic parameters, which then results in better calibration accuracy. Moreover, we provide two degenerative motion conditions in the resulting multicamera visual-inertial system, which impair the calibration accuracy and should be avoided in real application whenever possible. More importantly, we present the requirement of minimum motion for a reliable extrinsic calibration to provide the practical guideline. Finally, the full validation on both simulation and real-world data is demonstrated. By evaluating the Cramér-Rao lower bound on the covariance, the proposed camera-IMU calibration method is shown to be statistically efficient for accurate calibration with errors less than 0.01 m in translation and 0.5° in rotation, which is consistent with the theoretical analysis in this article.

In autonomous systems and robotic applications, accurate extrinsic calibration between light detection and ranging (LiDAR) sensors and cameras is crucial for reliable sensor fusion. Several techniques have been developed, including target-based and targetless calibration, but they are either impractical for real-world applications or limited in extracting complex and diverse features. This study presents TransCalib, an innovative deep-learning method for targetless and automatic extrinsic calibration. TransCalib predicts the misalignment between the camera and LiDAR by leveraging EfficientNetV2 to obtain features from the RGB camera image and LiDAR point cloud projection image (depth image), owing to its performance and parameter efficiency.We also developed an innovative feature-matching module that comprises a calibration convolutional feature aggregation block (Calib-CFAB) and a convolutional self-attention (CSA) transformer. Calib-CFAB enriches the combined feature map of the RGB and depth images, while the CSA transformer obtains the correlation in the feature maps. Trained and tested on the KITTI odometry dataset, TransCalib achieved a mean absolute rotation error of 0.14° and a mean translation error of 1.8 cm, outperforming existing methods. The proposed method allows for a robust fusion of LiDAR and camera data, improving the perception abilities of autonomous systems.

Accurately calibrate the extrinsic parameters between a camera and an Inertial Measurement Unit (IMU) is a prerequisite for achieving sensor data fusion in visual-inertial navigation systems. Due to delays introduced by triggering, transmission, and other factors, the sampled times of the camera and IMU do not align with the system timestamps, leading to a decrease in the accuracy of extrinsic calibration. Existing calibration methods are unable to avoid the temporal misalignment between the camera and IMU. This paper introduces a novel intermittent calibration sampling method to address temporal misalignment challenges. The approach entails detecting motion-stillness transition points in the IMU data as keyframe segmentation criteria, thereby decoupling data acquisition from system timestamps and aligning it with the sensor’s motion process. Subsequently, the robot hand-eye calibration method is applied to the extrinsic calibration of the Camera-IMU, combining the Random Sample Consensus algorithm to filter the poses of the camera and IMU. This process achieves precise calibration of extrinsic parameters. Experimental validation shows that the algorithm proposed in this paper enhances calibration accuracy when compared to the widely used Kalibr Camera-IMU calibration toolbox. The rotational extrinsic parameter accuracy increased by 181.25%, and the translational extrinsic parameter accuracy improved by 168.33%. These findings provide effective technical means for precise measurement of Camera-IMU extrinsic parameters.

In this paper, a new calibration method is proposed, which combines the extrinsic camera calibration and the robot hand-eye calibration. The advantage of this method is that it eliminates the errors and difficulties by solving complex equations such as AX=XB or AX=YB in traditional calibration methods. Error resources analysis and comparison with traditional methods are described in this paper. The experiment results on actual robot vision measurement system prove the validity and feasibility of this new method.

Calibration is vital to improving the accuracy of the strapdown inertial navigation system (SINS). Calibration is one of the crucial phases before the operation of SINS and requires high accuracy and speed. Traditional system-level calibration methods have the disadvantages of long calibration times and complicated path design. A fast calibration method for the lever arm and other error parameters is proposed in this article. To shorten the calibration time and improve the calibration accuracy, a backtracking calibration method in inertial frames based on the reduced-order Kalman filter is proposed. Simulations and experiments are given to illustrate the effectiveness of the novel calibration method. The calibration process can be completed in 10 min, which is greatly shortened compared with traditional methods. The calibration time is shortened, and the calibration accuracy is improved. Furthermore, navigation experiments show that the velocity and position accuracies of the SINS are improved significantly after compensation with the calibration results, fully illustrating the significance of the proposed calibration method in improving the navigation performance.

To obtain the 6 DOF extrinsic parameters (rotation and translation matrix) between a 3D ranging sensor and a thermal camera, previous methods require a high-resolution 3D ranging sensor to reliably detect features. Although sparse 3D LiDARs are widely used on autonomous robots, to the best of our knowledge, the extrinsic calibration between a sparse 3D LiDAR (particularly Velodyne VLP-16) and a thermal camera has not been considered in the literature. In this paper, we present a two-step method to address the problem, where a monocular visual camera is used to assist the process. The proposed method decomposes the problem into two steps: extrinsic calibration between a sparse 3D LiDAR and a visual camera; extrinsic calibration between a visual camera and a thermal camera. Experiments are conducted to demonstrate the effectiveness of the proposed two-step method.

We have developed a fast detector calibration method, which is capable of calibration of monolithic crystal detectors fast and conveniently. This method could potentially be used to achieve the DOI calibration information of a thick detector. Compared with traditional calibration method, this calibration technique doesnt require a well-collimated pencil beams, and can calibrate a detector in reduced time with fan beams. After the collection of the calibration reference dataset, we propose a data-processing method called common-data subset (CDS) method to acquire the reference datasets as if they were acquired using a pencil beam or even as if there is a single-point light source scanning inside the scintillator. If using fan beams, only 2N scans (in x and y directions) are necessary to calibrate the whole detector surface instead of doing N2 scans using pencil gamma-ray beam. For calibration of the DOI information, 3N scans are necessary, within which the last set of N scans are done with slant beams. Monte-Carlo simulation has been done to support the feasibility of this method, which enables simple, fast and robust calibration for monolithic crystal detectors.

For applications in robotics and autonomous vehicles, fusing 3D LiDAR, RGB camera, and thermal camera can improve the perception in day and night environments. To fuse the sensors, accurate extrinsic calibration is important. However, most existing methods use offline calibration, which is tedious, as it requires the use of special targets and human intervention. Meanwhile, among the online calibration methods, little attention has been paid to LiDAR and thermal camera online calibration. Thus, in this paper, an online extrinsic calibration method is proposed to solve the problem, called RGBDTCalibNet. It can achieve the extrinsic calibration between a 3D LiDAR and an RGB camera, as well as a 3D LiDAR and a thermal camera. It leverages CNN-based deep learning methods for feature extraction and feature matching. The proposed network only takes a pair of images and point cloud as the input and outputs the extrinsic parameters in a single shot. It is trained and tested on two datasets: KITTI360 and our own dataset collected in the NTU campus, using two types of LiDAR: Velodyne HDL-64 and Livox Horizon. Both demonstrate the effectiveness, robustness and accuracy of RGBDTCalibNet. The code will be open-sourced at https://github.com/sanatmharolkar/RGBDTCalibNet.git

With the advantages of high velocity measurement accuracy and fast dynamic response, the laser Doppler velocimeter (LDV) is expected to replace the odometer to be combined with a strapdown inertial navigation system (SINS) to form a higher precision integrated navigation system. However, LDV scale factor error and misalignment angles between LDV and inertial measurement unit will affect the accuracy of navigation. Considering that not all global navigation satellite system (GNSS) receivers can directly provide velocity information and current mainstream calibration methods are sensitive to the measurement noise and outliers of velocity and position information, a robust calibration method aided by GNSS is proposed in this paper, which is based on position observation. Different from current popular calibration methods, the attitude information of the GNSS/SINS integrated navigation system obtained by an adaptive Kalman filter is used to construct the observation vector together with LDV velocity outputs and GNSS position outputs in this method. The LDV scale factor error and the misalignment angle are determined by the ratio of two observation vector modulus and the Davenport's q-method method, respectively. The accuracy and robustness of the calibration method are verified by one vehicle test with normal GNSS signals and one vehicle test with GNSS signals with outliers. And the horizontal position error of dead reckoning of the calibrated LDV/SINS integrated system are less than 0.0314% and 0.1033% of the mileage, respectively.

With the rapid development of autonomous driving and SLAM technology, the perception system of a vehicle heavily relies on laser and image sensors to capture the real-world scenario and avoid obstacles autonomously. To achieve accurate and robust multi-sensor fusion computation, high-precision extrinsic calibration of camera and laser scanner is a necessary requirement. Traditional multi-sensor calibration methods based on manual features rely on specific scenarios and may not provide feature information over long distances. In this paper, we present a novel approach for robustly calibrating the extrinsic parameters of a solid-state(SS) lidar-camera system in a natural environment. Our proposed method begins with obtaining robust line feature information. we first innovatively employ a super-voxel clustering method to extract global 3D line features from the complete point cloud and then back-project these 3D line features into 2D space. Afterward, a transformer-based edge detection network, EDTER, is used to detect the edge features and estimate the probability pixel-by-pixel. To consider the uncertainty of two-dimensional line features and the inconsistency of residuals at different distances, we construct a line feature weight model for line feature residual calculation. Finally, we minimize the residual errors using least squares optimization to recover the relative pose of the camera and the lidar sensor. We conducted a performance study to compare our proposed method against existing targetless calibration methods on various natural scenarios. The experimental results demonstrate that our proposed method achieves higher robustness, accuracy, and consistency, making it suitable for real-world applications.

Roadside camera calibration is essential to intelligent traffic surveillance and still an unsolved problem. The commonly used pattern-based calibration methods are suitable for the laboratory environment rather than real traffic environment, since the calibration patterns (e.g., checkerboards) generally do not exist in traffic scenarios. In view of this, we propose a new framework for roadside camera calibration where the vehicle moving on the roadway is first introduced as a calibration pattern. Considering that the vehicles are main monitoring targets and inevitably appear in traffic scenarios, the proposed calibration method has a wide use range and is not limited to the structure information of traffic scenarios. Inspired by the traditional pattern-based calibration methods that utilize the matching of 3D-2D point correspondences, we utilize the 3D-2D vehicle matching for camera calibration. The key insight is to convert the camera calibration problem into a vehicle matching problem. To improve the accuracy of calibration results, a new measure function is provided to evaluate the vehicle matching degree and a dynamic calibration method using multi-frame information is proposed to correct camera parameters. Experiments on real traffic images demonstrate the effectiveness and practicability of the proposed calibration framework.

With the continuous development and popularization of sensor-fusion technology for mobile robots, the application of camera and light detection and ranging (LiDAR) fusion perception has become particularly important. Moreover, the calibration of extrinsic parameters between the camera and LiDAR is a crucial prerequisite for fusion. Although traditional target-based calibration methods have been widely adopted, their cumbersome operation and high costs necessitate the development of more efficient and flexible calibration methods. To address this problem, this study proposed a two-stage calibration method based on motion and edge matching. In the first stage, the preliminary estimation of the extrinsic parameters between the camera and LiDAR was performed by matching visual odometry and LiDAR odometry using a hand–eye target method. In the second stage, the calibration results from the first stage were further refined by matching the image edges and discontinuous depth point clouds. The calibration system was then tested in both simulated and actual environments. The experimental results showed that this method, which did not require specially structured targets, could achieve highly flexible and robust automated calibration. Compared to other advanced methods, the accuracy of the proposed method was higher.

Research on accurate calibration method of screen plane equation of sky screen vertical target