A Review of Deep Learning-Based LiDAR and Camera Extrinsic Calibration

Parallel lines, common geometric elements in daily environments, can form vanishing points in projective geometry. In this article, we introduce a novel method for the extrinsic calibration of high-resolution light detection and ranging (LiDAR) and cameras based on vanishing points. This method incorporates two main innovations: 1) the vanishing point is introduced into the extrinsic parameter calibration of camera and LiDAR and 2) it features a simple and intuitive mathematical model compared to the existing target-less extrinsic parameter calibration methods. These advantages confer three key characteristics on the method: 1) it eliminates the need for a specific calibration target, making it applicable to most scenes; 2) owing to its exclusion of an optimization process for fitting optimal matches for calibration, it avoids convergence and consistency considerations and exhibits rapid computational speed; and 3) the algorithm has theoretical applicability between any high-resolution visual sensors. Simulation and actual experiments conducted across diverse settings demonstrate that the method is reliable and accurate. The code has been open-sourced for learning and usage at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/ColeZhao/livox_camera_vanishing</uri>.

Many approaches for camera and LiDAR calibration are presented in literature but none of them estimates all intrinsic and extrinsic parameters simultaneously and therefore optimally in a probabilistic sense.In this work, we present a method to simultaneously estimate intrinsic and extrinsic parameters of cameras and LiDARs in a unified problem. We derive a probabilistic formulation that enables flawless integration of different measurement types without hand-tuned weights. An arbitrary number of cameras and LiDARs can be calibrated simultaneously. Measurements are not required to be time-synchronized. The method is designed to work with any camera model.In evaluation, we show that additional LiDAR measurements significantly improve intrinsic camera calibration. Further, we show on real data that our method achieves state-of-the-art calibration precision with high reliability.

A Comparison of Error Metrics for Extrinsic Calibration and Fusion of Camera and Multi-Layer LIDAR

The paper describes an approach that allows solving the problem of extrinsic calibration of a multi-beam lidar and a stereo camera. The approach does not impose any restrictions on the place in which calibration should be performed. Calibration is performed using a calibration board, which is a flat rectangle with special markers. Three-dimensional correspondences are used for calibration. First, a search for the three-dimensional coordinates of the corner points of the calibration board in the coordinate systems of the stereo pair cameras as well as in the coordinate system of the lidar is made. Next, using the optimization methods, calibration parameters are calculated. The results of a series of virtual and real experiments show that the algorithm allows the calibration to be performed with an accuracy comparable to that of sensors. The proposed approach allows one to improve the calibration accuracy due to the simultaneous use of information from two cameras of the stereo pair and is suitable for lidars with both the low and high point density.

The external parameter calibration of LiDAR and camera is the core link of multimodal perception fusion technology, and its accuracy directly determines the effectiveness of cross modal information fusion. This article proposes a method for high-precision calibration of laser radar and camera extrinsic parameters using a checkerboard calibration board based on spatial registration theory. In response to the difficulty of feature extraction caused by the sparsity of LiDAR point clouds, this method enhances feature information by accumulating multi view point clouds and combines filtering algorithms to improve feature saliency; A standardized experimental platform was built based on ROS and Matlab, utilizing the reflection intensity distribution pattern of the checkerboard point cloud to achieve accurate extraction of its three-dimensional corner coordinates. At the same time, the corresponding two-dimensional checkerboard corner points are detected in the camera image, and the relative pose transformation matrix between the LiDAR and the camera is solved by establishing an accurate matching relationship between the two-dimensional and three-dimensional corner points. The experimental results show that this method has high-precision calibration results and good repeatability, which can provide a reliable extrinsic basis for multi-sensor fusion perception systems.

Due to different modality and sampling density between LiDAR and camera, the correspondences between these two sensors are difficult to find. In this paper, we propose a novel method for Camera-LiDAR extrinsic parameters calibration based on the near-far dual targets. In our approach, a chessboard target is placed closer to the combined Camera-LiDAR device and a trihedron object of the structured scene is placed farther. Using the chessboard, camera motion can be estimated through the RGB images captured by the camera. The point cloud captured by LiDAR from the trihedron is then used for LiDAR motion estimation. By coordinating their motion constraints, our approach can reliably estimate the extrinsic parameters between two sensors by a classic hand-eye calibration process without facing the challenges of finding feature correspondences. Finally, the proposed method is evaluated on both simulated data and real data.

With the continuous development and popularization of sensor-fusion technology for mobile robots, the application of camera and light detection and ranging (LiDAR) fusion perception has become particularly important. Moreover, the calibration of extrinsic parameters between the camera and LiDAR is a crucial prerequisite for fusion. Although traditional target-based calibration methods have been widely adopted, their cumbersome operation and high costs necessitate the development of more efficient and flexible calibration methods. To address this problem, this study proposed a two-stage calibration method based on motion and edge matching. In the first stage, the preliminary estimation of the extrinsic parameters between the camera and LiDAR was performed by matching visual odometry and LiDAR odometry using a hand–eye target method. In the second stage, the calibration results from the first stage were further refined by matching the image edges and discontinuous depth point clouds. The calibration system was then tested in both simulated and actual environments. The experimental results showed that this method, which did not require specially structured targets, could achieve highly flexible and robust automated calibration. Compared to other advanced methods, the accuracy of the proposed method was higher.

This paper proposes an improved method for the calibration of a 2D LiDAR or a planar scanning laser rangefinder with respect to a camera, which uses the same checkerboard target as in the camera calibration. To address one of the drawbacks in the LiDAR calibration with a checkerboard, the sensitivity to checkerboard poses, laser points on the checkerboard are obtained from both the originally positioned LiDAR and the same LiDAR vertically rotated from the original position, in the proposed calibration. They construct the additional constraints with the normal vector of the checkerboard, and strengthen the basic point-on-plane constraints in the LiDAR calibration with a checkerboard. However, in practice, the LiDAR rotation can be imprecise. Thus, we further develop the calibration procedure and examine its calibration performance, in the presence of such impreciseness in the LiDAR rotation. A performance analysis is conducted with synthetic noisy data, and it is confirmed that the proposed calibration yields smaller error in the parameter estimation than a conventional baseline calibration method. Subsequently, we examine the calibration performance with real data, where the proposed calibration is carried out by building a calibration system. It is shown that the proposed calibration also has more consistency in the parameter estimation. For further study, we compare the proposed calibration based on point-on-plane constraints with a calibration based on point-to-point constraints, and show that the proposed calibration, with easier data acquisition, yields comparable results.

Roadside camera calibration is essential to intelligent traffic surveillance and still an unsolved problem. The commonly used pattern-based calibration methods are suitable for the laboratory environment rather than real traffic environment, since the calibration patterns (e.g., checkerboards) generally do not exist in traffic scenarios. In view of this, we propose a new framework for roadside camera calibration where the vehicle moving on the roadway is first introduced as a calibration pattern. Considering that the vehicles are main monitoring targets and inevitably appear in traffic scenarios, the proposed calibration method has a wide use range and is not limited to the structure information of traffic scenarios. Inspired by the traditional pattern-based calibration methods that utilize the matching of 3D-2D point correspondences, we utilize the 3D-2D vehicle matching for camera calibration. The key insight is to convert the camera calibration problem into a vehicle matching problem. To improve the accuracy of calibration results, a new measure function is provided to evaluate the vehicle matching degree and a dynamic calibration method using multi-frame information is proposed to correct camera parameters. Experiments on real traffic images demonstrate the effectiveness and practicability of the proposed calibration framework.

We address joint extrinsic calibration of lidar, camera and radar sensors. To simplify calibration, we propose a single calibration target design for all three modalities, and implement our approach in an open-source tool with bindings to Robot Operating System (ROS). Our tool features three optimization configurations, namely using error terms for a minimal number of sensor pairs, or using terms for all sensor pairs in combination with loop closure constraints, or by adding terms for structure estimation in a probabilistic model. Apart from relative calibration where relative transformations between sensors are computed, our work also addresses absolute calibration that includes calibration with respect to the mobile robot's body. Two methods are compared to estimate the body reference frame using an external laser scanner, one based on markers and the other based on manual annotation of the laser scan. In the experiments, we evaluate the three configurations for relative calibration. Our results show that using terms for all sensor pairs is most robust, especially for lidar to radar, when minimum five board locations are used. For absolute calibration the median rotation error around the vertical axis reduces from 1 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">°</sup> before calibration, to 0.33 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">°</sup> using the markers and 0.02 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">°</sup> with manual annotations.

The calibration problem of binocular stereo vision rig is critical for its practical application. However, most existing calibration methods are based on manual off-line algorithms for specific reference targets or patterns. In this paper, we propose a novel simultaneous localization and mapping (SLAM)-based self-calibration method designed to achieve real-time, automatic and accurate calibration of the binocular stereo vision (BSV) rig’s extrinsic parameters in a short period without auxiliary equipment and special calibration markers, assuming the intrinsic parameters of the left and right cameras are known in advance. The main contribution of this paper is to use the SLAM algorithm as our main tool for the calibration method. The method mainly consists of two parts: SLAM-based construction of 3D scene point map and extrinsic parameter calibration. In the first part, the SLAM mainly constructs a 3D feature point map of the natural environment, which is used as a calibration area map. To improve the efficiency of calibration, a lightweight, real-time visual SLAM is built. In the second part, extrinsic parameters are calibrated through the 3D scene point map created by the SLAM. Ultimately, field experiments are performed to evaluate the feasibility, repeatability, and efficiency of our self-calibration method. The experimental data shows that the average absolute error of the Euler angles and translation vectors obtained by our method relative to the reference values obtained by Zhang’s calibration method does not exceed 0.5˚ and 2 mm, respectively. The distribution range of the most widely spread parameter in Euler angles is less than 0.2˚ while that in translation vectors does not exceed 2.15 mm. Under the general texture scene and the normal driving speed of the mobile robot, the calibration time can be generally maintained within 10 s. The above results prove that our proposed method is reliable and has practical value.

Calibration of optical metrology stereophotogrammetric systems is vital to obtain accurate and precise three-dimensional (3-D) measurements. Despite its importance, the work pipeline of intrinsic and extrinsic camera calibration still remains manually laborious with high technical complexity. The use of a multilayer perceptron neural network to calibrate an optical metrology stereophotogrammetric system utilizing a statistical band-limited pattern projection system is demonstrated. Highly accurate, highly precise, and highly dense 3-D surface reconstructions are obtained solely from homologous corresponding pairs without the need for intrinsic and extrinsic camera calibration. Measurement performance in the typical optical metrology sense, where 3-D measurements were evaluated with respect to length and surface gauges, is shown.

Common single-line 2D LiDAR sensors and cameras have become core components in the field of robotic perception due to their low cost, compact size, and practicality. However, during the data fusion process, the randomness and complexity of real industrial scenes pose challenges. Traditional calibration methods for LiDAR and cameras often rely on precise targets and can accumulate errors, leading to significant limitations. Additionally, the semantic fusion of LiDAR and camera data typically requires extensive projection calculations, complex clustering algorithms, or sophisticated data fusion techniques, resulting in low real-time performance when handling large volumes of data points in dynamic environments. To address these issues, this paper proposes a semantic fusion algorithm for LiDAR and camera data based on contour and inverse projection. The method has two remarkable features: (1) Combined with the ellipse extraction algorithm of the arc support line segment, a LiDAR and camera calibration algorithm based on various regular shapes of an environmental target is proposed, which improves the adaptability of the calibration algorithm to the environment. (2) This paper proposes a semantic segmentation algorithm based on the inverse projection of target contours. It is specifically designed to be versatile and applicable to both linear and arc features, significantly broadening the range of features that can be utilized in various tasks. This flexibility is a key advantage, as it allows the method to adapt to a wider variety of real-world scenarios where both types of features are commonly encountered. Compared with existing LiDAR point cloud semantic segmentation methods, this algorithm eliminates the need for complex clustering algorithms, data fusion techniques, and extensive laser point reprojection calculations. When handling a large number of laser points, the proposed method requires only one or two inverse projections of the contour to filter the range of laser points that intersect with specific targets. This approach enhances both the accuracy of point cloud searches and the speed of semantic processing. Finally, the validity of the semantic fusion algorithm is proven by field experiments.

In image-based measurements, quantitative image data must be mapped to three-dimensional object space. Analytical photogrammetric methods, which may be used to accomplish this task, are discussed from the viewpoint of experimental fluid dynamicists. The Direct Linear Transformation (DLT) for camera calibration, used in pressure sensitive paint, is summarized. An optimization method for camera calibration is developed that can be used to determine the camera calibration parameters, including those describing lens distortion, from a single image. Combined with the DLT method, this method allows a rapid and comprehensive in-situ camera calibration and therefore is particularly useful for quantitative flow visualization and other measurements such as model attitude and deformation in production wind tunnels. The paper also includes a brief description of typical photogrammetric applications to temperature- and pressure-sensitive paint measurements and model deformation measurements in wind tunnels.

This paper proposes a calibration method for monocular cameras and 2D Lidar external parameters based on automatic detection of features. The triangle calibration target is used, and the point and line features are automatically extracted. The point line constraint and the proposed European transformation invariance constraint are applied simultaneously in the calibration process to obtain more accurate and effective external parameters. First, a line feature extraction method is proposed. The triangle region in the image is extracted by Hough transform and K-means clustering, and the 3D pose of the triangle calibration target is calculated. Second, we analyze the effects of image feature extraction and Lidar measurement range errors on rotating and translational external parameters, respectively, through simulation experiments. The error characteristics of camera-lidar external parameters are analyzed and compared with those of Zhang and Pless. Finally, real-data experiments show that our method can obtain accurate external calibration.