Targetless LiDAR–camera extrinsic calibration via semantic distribution alignment

For applications in robotics and autonomous vehicles, fusing 3D LiDAR, RGB camera, and thermal camera can improve the perception in day and night environments. To fuse the sensors, accurate extrinsic calibration is important. However, most existing methods use offline calibration, which is tedious, as it requires the use of special targets and human intervention. Meanwhile, among the online calibration methods, little attention has been paid to LiDAR and thermal camera online calibration. Thus, in this paper, an online extrinsic calibration method is proposed to solve the problem, called RGBDTCalibNet. It can achieve the extrinsic calibration between a 3D LiDAR and an RGB camera, as well as a 3D LiDAR and a thermal camera. It leverages CNN-based deep learning methods for feature extraction and feature matching. The proposed network only takes a pair of images and point cloud as the input and outputs the extrinsic parameters in a single shot. It is trained and tested on two datasets: KITTI360 and our own dataset collected in the NTU campus, using two types of LiDAR: Velodyne HDL-64 and Livox Horizon. Both demonstrate the effectiveness, robustness and accuracy of RGBDTCalibNet. The code will be open-sourced at https://github.com/sanatmharolkar/RGBDTCalibNet.git

In autonomous systems and robotic applications, accurate extrinsic calibration between light detection and ranging (LiDAR) sensors and cameras is crucial for reliable sensor fusion. Several techniques have been developed, including target-based and targetless calibration, but they are either impractical for real-world applications or limited in extracting complex and diverse features. This study presents TransCalib, an innovative deep-learning method for targetless and automatic extrinsic calibration. TransCalib predicts the misalignment between the camera and LiDAR by leveraging EfficientNetV2 to obtain features from the RGB camera image and LiDAR point cloud projection image (depth image), owing to its performance and parameter efficiency.We also developed an innovative feature-matching module that comprises a calibration convolutional feature aggregation block (Calib-CFAB) and a convolutional self-attention (CSA) transformer. Calib-CFAB enriches the combined feature map of the RGB and depth images, while the CSA transformer obtains the correlation in the feature maps. Trained and tested on the KITTI odometry dataset, TransCalib achieved a mean absolute rotation error of 0.14° and a mean translation error of 1.8 cm, outperforming existing methods. The proposed method allows for a robust fusion of LiDAR and camera data, improving the perception abilities of autonomous systems.

Accurately calibrate the extrinsic parameters between a camera and an Inertial Measurement Unit (IMU) is a prerequisite for achieving sensor data fusion in visual-inertial navigation systems. Due to delays introduced by triggering, transmission, and other factors, the sampled times of the camera and IMU do not align with the system timestamps, leading to a decrease in the accuracy of extrinsic calibration. Existing calibration methods are unable to avoid the temporal misalignment between the camera and IMU. This paper introduces a novel intermittent calibration sampling method to address temporal misalignment challenges. The approach entails detecting motion-stillness transition points in the IMU data as keyframe segmentation criteria, thereby decoupling data acquisition from system timestamps and aligning it with the sensor’s motion process. Subsequently, the robot hand-eye calibration method is applied to the extrinsic calibration of the Camera-IMU, combining the Random Sample Consensus algorithm to filter the poses of the camera and IMU. This process achieves precise calibration of extrinsic parameters. Experimental validation shows that the algorithm proposed in this paper enhances calibration accuracy when compared to the widely used Kalibr Camera-IMU calibration toolbox. The rotational extrinsic parameter accuracy increased by 181.25%, and the translational extrinsic parameter accuracy improved by 168.33%. These findings provide effective technical means for precise measurement of Camera-IMU extrinsic parameters.

With the extensive research and application of various unmanned systems, for instance, unmanned aerial vehicle and self-driving cars, people are increasingly aware of the importance of multi-sensor data fusion. In this paper, a novel calibration method and corresponding experimental setup are proposed to accurately estimate the extrinsic parameters between LiDAR and camera. Proposed method introduces the reflection intensity to extract LiDAR line features on a self-developed calibration board. Intersections of LiDAR point clouds are calculated with the extracted LiDAR line features and intersections of visual lines are derived from ArUco Marker. Therefore multiple intersection correspondences between LiDAR frame and camera frame are given to calculate extrinsic parameters. We proved through experiments using a binocular camera with known intrinsic parameters. The results show that extrinsic calibration errors are within 0.7\(^{\circ }\) over rotation and within 1 cm over translation. The method is compared to the state-of-the-art method, the accuracy and convergence improve to about 3 times.KeywordsExtrinsic calibrationLiDARCameraReflection intensity

Accurate extrinsic calibration of inertial sensors, such as Inertial Measurement Units (IMUs) and cameras is crucial for trajectory estimation of Uncrewed Aerial Vehicles (UAVs). While numerous calibration methods have been proposed, these techniques often rely on specialized equipment, planar targets, and an initial estimate of the calibration parameters. In this research, we propose a targetless calibration method designed for UAVs equipped with IMUs and RGBDepth (RGB-D) cameras. Our approach leverages deep-learning-based floor-segmentation to extract ground points from the depth channel of RGB-D images. Subsequently, the normal vector to these points is estimated. The known orientation of the normal to the floor segment and the gravity vector sensed in the accelerometer’s frame are utilized in a robust estimation approach to estimate the extrinsic calibration parameters. We illustrate that the developed method outperforms MATLAB’s Toolboxes and exhibits similar performance to Kalibr without the use of specialized checkerboard targets.

Targetless LiDAR-camera extrinsic calibration remains challenging due to unreliable cross-modal correspondences and sensitivity to initialization. We present a targetless extrinsic calibration framework based on class-agnostic boundary mask alignment in a shared image-plane representation. This scheme first constructs consistent LiDAR-camera mask pairs from image-plane depth and intensity projections of LiDAR data and camera images. It then obtains robust initial pose candidates through bounded rotation-only global initialization and refines them using a computationally efficient stochastic gradient approximation to estimate the optimal extrinsic parameters. Experiments on the KITTI benchmark demonstrate a superior accuracy-runtime trade-off compared with a segmentation-based global optimization baseline, while real-world driving tests confirm stable cross-modal alignment under vibration and inter-modal timing jitter.

Abstract. In the current state-of-the-art, geodetic deformation analysis of natural and artificial objects (e.g. dams, bridges,...) is an ongoing research in both static and kinematic mode and has received considerable interest by researchers and geodetic engineers. In this work, due to increasing the accuracy of geodetic deformation analysis, a terrestrial laser scanner (TLS; here the Zoller+Fröhlich IMAGER 5006) and a high resolution digital camera (Nikon D750) are integrated to complementarily benefit from each other. In order to optimally combine the acquired data of the hybrid sensor system, a highly accurate estimation of the extrinsic calibration parameters between TLS and digital camera is a vital preliminary step. Thus, the calibration of the aforementioned hybrid sensor system can be separated into three single calibrations: calibration of the camera, calibration of the TLS and extrinsic calibration between TLS and digital camera. In this research, we focus on highly accurate estimating extrinsic parameters between fused sensors and target- and targetless (mutual information) based methods are applied. In target-based calibration, different types of observations (image coordinates, TLS measurements and laser tracker measurements for validation) are utilized and variance component estimation is applied to optimally assign adequate weights to the observations. Space resection bundle adjustment based on the collinearity equations is solved using Gauss-Markov and Gauss-Helmert model. Statistical tests are performed to discard outliers and large residuals in the adjustment procedure. At the end, the two aforementioned approaches are compared and advantages and disadvantages of them are investigated and numerical results are presented and discussed.

The combination of a camera and a 2D laser rangefinder (LRF) is widely used in robotics, mapping, and unmanned driving to simultaneously obtain the 3D geometry and color texture of a scene. However, data misregistration between the camera and the LRF frequently occurs due to the difficulty of precise installation and alignment between them. Extrinsic calibration between the camera and the LRF is necessary. In this study, a photogrammetric control field is used to perform a robust and accurate calibration between the camera and the LRF which requires only one shot. With the use of the photogrammetric control field as the common reference, extrinsic calibration between two sensors is decoupled by calibrating each sensor separately. First, using the room corner of the control field, the LRF is calibrated with respect to the control field by solving a simplified perspective-three-point problem. Then, a large number of control points are used to obtain the robust and accurate extrinsic parameters of the camera with respect to the control field. Experiments with simulated and real data were performed. The experimental results show that the proposed scheme is accurate, precise, and robust under different noise levels, and the calibration results can be utilized in indoor and outdoor applications.

Both 3D mmWave Radar (3D: <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$x, y, z$</tex> ) and thermal camera are robust in harsh environments. Fusing them is beneficial for V2X road side units and unmanned vehicles to operate in all-weather conditions. To fuse the two sensors, accurate extrinsic calibration is indispensable. However, limited literatures can be found, since 3D radar is pushed into the market just for a short period. Most research are focused on 2D radar (2D: <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$x, y$</tex> ) and rgb camera calibration. Recently, one research is proposed for 3D radar and rgb camera, but it requires continuous movement of the sensors to perform hand-eye calibration. It is impractical for the sensors mounted to the road side in V2X, where the sensors are non-movable. To solve the problems, 3DRadar2ThermalCalib is proposed. 1) A novel calibration target, a spherical-trihedral, is introduced. We use the sphere center as the common feature. It can be directly detected by both sensors. 2) We propose the methods to automatically detect the sphere center from both sensors. 3) The optimal extrinsic parameter is obtained by minimizing the re-projection error. Both quantitative and qualitative analysis in real environments demonstrate the method is accurate. The re-projection error can reach 1.88 pixels.

This paper presents an improvement for error reduction of the cost function for non-linear optimization of extrinsic parameters estimation between single line scan LiDAR and RGB camera. The non-linear optimization utilizes a least square scheme by assigning equal weights to all LiDAR measurement points. With robust regression, we used all LiDAR measurement points and removed RANSAC outlier removal with a weighting scheme dependent on the defined geometric constraint. The methods aims to minimize the error from the inaccuracy of the LiDAR measurement points using robust regression with M-estimator. The methods are tested with 100 random trials with noise magnitude from 5 to 40mm and a 10 percent chance of outliers of 3 times the normal noise magnitude. The results show that M-estimator is more resistant to noise than current state of art.

Stereo-vision devices have rigorous requirements for extrinsic parameter calibration. In Stereo Visual Inertial Odometry (VIO), inaccuracy in or changes to camera extrinsic parameters may lead to serious degradation in estimation performance. In this manuscript, we propose an online calibration method for stereo VIO extrinsic parameters correction. In particular, we focus on Multi-State Constraint Kalman Filter (MSCKF [1]) framework to implement our method. The key component is to formulate stereo extrinsic parameters as part of the state variables and model the Jacobian of feature reprojection error with respect to stereo extrinsic parameters as sub-block of update Jacobian. Therefore we can estimate stereo extrinsic parameters simultaneously with inertial measurement unit (IMU) states and camera poses. Experiments on EuRoC dataset and real-world outdoor dataset demonstrate that the proposed algorithm produce higher positioning accuracy than the original S-MSCKF [2], and the noise of camera extrinsic parameters are self-corrected within the system.

Binocular stereovision system is regularly only calibrated once, that is each camera is calibrated using a picture of the calibration pattern laid in the cross field. This methods calibration accuracy is expected to be improved. A twice calibrating method is proposed here. Each cameras extrinsic and intrinsic parameters are calibrated in different fields. The extrinsic parameters are firstly calibrated in the cross field, then the intrinsic parameters are calibrated in the close-range field. By applying the method to one binocular stereovision system built in our lab, it is demonstrated that the proposed method can improve the calibration accuracy more than 3.4dB than the once calibrating method. Its accuracy makes this novel method more suitable for high accurate target tracking application.

In order to implement 3D measurement in different measurement systems with line structured-light sensor, the extrinsic matrix for transforming the position between the structured-light sensor and the measurement platform must be calibrated. An extrinsic calibration method, achieved by measuring the fixed point (the center of a standard ball) of the measuring system with the sensor on the different positions, was proposed. After collecting the images of the line structured-light stripe on the ball, the information of the slices can be extracted by direction template. The coordinate of the ball center in the laser plane frame can be calculated more accurately though image filtration and slice denoising. The extrinsic matrix can be solved by the conjugate pairs, which include the coordinates of both the ball center in the laser plane frame and the sensor in the measuring system. Experimental study was carried out on a four-axis CMM with a ball supplied by CIMCORE, of which the diameter was 25.40145 mm. 203 conjugate pairs were extracted and the extrinsic parameters were calculated. The ball was fitted with the calibrated parameters and the measured point cloud on the ball surface. The average error of the radius calculated was 0.0052mm. This method is simple; it can also be used in non-orthogonal measurement system.

In this paper we address the problem of estimating the intrinsic parameters of a 3D LIDAR while at the same time computing its extrinsic calibration with respect to a rigidly connected camera. Existing approaches to solve this nonlinear estimation problem are based on iterative minimization of nonlinear cost functions. In such cases, the accuracy of the resulting solution hinges on the availability of a precise initial estimate, which is often not available. In order to address this issue, we divide the problem into two least-squares sub-problems, and analytically solve each one to determine a precise initial estimate for the unknown parameters. We further increase the accuracy of these initial estimates by iteratively minimizing a batch nonlinear least-squares cost function. In addition, we provide the minimal identifiability conditions, under which it is possible to accurately estimate the unknown parameters. Experimental results consisting of photorealistic 3D reconstruction of indoor and outdoor scenes, as well as standard metrics of the calibration errors, are used to assess the validity of our approach.

In autonomous driving, the camera and LiDAR are complementary to each other through their strengths, and many autonomous vehicles and robot recognition systems utilize cameras and LiDAR. In order to effectively fuse the data of the camera and LiDAR attached to different positions and angles, we must perform camera&#x2013;LiDAR extrinsic calibration. Most existing camera&#x2013;LiDAR calibration methods infer the results by constructing a camera&#x2013;LiDAR feature pair using features acquired from a single frame. It has the disadvantage that it is challenging to draw results. In this paper, we used sequential LiDAR data to extract features by accumulating LiDAR frames effectively. By using the location information between the accumulated frame (global) and the single frame (local) to convert the features detected from the global to the local, the shortcomings of LiDAR&#x2019;s single frame with few features were supplemented. Methods for detecting feature points in LiDAR are described step-by-step, and the advantages and excellence of this camera&#x2013;LiDAR calibration system are shown through quantitative/qualitative evaluation methods. As a result, the proposed system outputs superior performance compared to other systems.