LGMMFusion: A LiDAR-guided multi-modal fusion framework for enhanced 3D object detection.

With the rapid development of the autonomous vehicles industry, there has been a dramatic proliferation of research concerned with related works, where road markings detection is an important issue. When there is no public open data in a field, we must collect road markings data and label them by ourselves manually, which is huge labor work and takes lots of time. Moreover, object detection often encounters the problem of small object detection. The detection accuracy often decreases when the detection distance increases. This is primarily because distant objects on the road take up few pixels in the image and object scales vary depending on different distances and perspectives. For the sake of solving the issues mentioned above, this paper utilizes a virtual dataset and open dataset to train the object detection model and cross-field testing in the field of Taiwan roads. In order to make the model more robust and stable, the data augmentation method is employed to generate more data. Therefore, the data are increased through the data augmentation method and homography transformation of images in the limited dataset. Additionally, Inverse Perspective Mapping is performed on the input images to transform them into the bird's eye view, which solves the "small objects at far distance" problem and the "perspective distortion of objects" problem so that the model can clearly recognize the objects on the road. The model testing on the front-view images and bird's eye view images also shows a remarkable improvement of accuracy by 18.62%.

Multi-sensor fusion is essential for an accurate and reliable autonomous driving system. Recently, BEVFusion has been proposed to integrate LiDAR and image features in a unified Bird's Eye View (BEV) representation. However, there is an issue with the loss of local image information during the extraction of global image information on the backbone network. In order to fully integrate local features with global features, this paper proposes a network called DS-BEV based on feature selection and refinement. It includes a Feature Selection Fusion module (FSM) and a Feature Refinement module (FRM). In the FSM, the features of different modal are first extracted by using specific networks and projected into a unified BEV representation space. Through channel and spatial learning, important information is selected from the initial features and fused to generate preliminary fusion features. Then,the image features extracted by a CNN network and the preliminary fusion features output by the FSM are sent to the FRM together. By combining the local features generated by CNN network, the fusion features are refined. We evaluate our model on the nuScenes dataset. Experiments show that our DS-BEV achieves 69.5% mAP and 72.3% NDS in detection accuracy.

Voxel is one of the common structural representation of 3D point cloud. Due to the sparsity of point cloud generated by light detection and ranging (LiDAR), there is the extreme imbalance in the foreground and background voxels. It decreases the accuracy of 3D object detection, has the negative effect on intelligent driving safety. To overcome this problem, we present a saliency prediction based 3D object detector SP-Det in this article. Although foreground voxels have the sufficient feature of object, it is difficult to localize the foreground region from voxel space with the larger background region. We design an auxiliary learning task, saliency prediction (SP). It benefits 3D detector in identifying the foreground region. SP task uses label diffusion to alleviate the label imbalance. It reduces the learning difficulty of saliency in voxel and bird's eye view (BEV) spaces. After that, to strengthen feature interaction from the sparse foreground region, we design saliency fusion (SF) module to fuse the learning result in SP task. It utilizes voxel and BEV saliency maps as progressive attention to resist the redundant feature from background region. To aggregate more foreground feature inside 3D and BEV region of interest (RoI), we design hybrid grid maps based RoI pooling (Hybrid-RoI pooling). Experiments are conducted in STF dataset. The adverse weather enlarges the sparsity of LiDAR point cloud, increasing the difficulty of object detection. SP-Det identifies and leverages foreground region, and achieves the performance better than the current methods. Hence, we believe that SP-Det benefits to LiDAR based 3D scene understanding in the adverse weather.

LiDAR-based Global Localization (LGL) is an essential ingredient for autonomous robots. However, existing LGL methods typically consider only partial information (e.g., geometric features) from LiDAR observations or are designed for homogeneous LiDAR sensors, overlooking the uniformity in LGL. In this work, a uniform LGL method is proposed, termed UniLGL, which simultaneously achieves spatial and material uniformity, as well as sensor-type uniformity. The key idea of the proposed method is to encode the complete point cloud, which contains both geometric and material information, into a pair of Bird's Eye View (BEV) images (i.e., a spatial BEV image and an intensity BEV image), thereby transforming the LGL problem into a cascaded LiDAR Place Recognition (LPR) and pose estimation problem from the perspective of image fusion. An end-to-end multi-BEV fusion network is designed to extract uniform features, equipping UniLGL with spatial and material uniformity. To ensure robust LGL across heterogeneous LiDAR sensors, a viewpoint invariance hypothesis is introduced, which replaces the conventional translation equivariance assumption commonly used in existing LPR networks and supervises UniLGL to achieve sensor type uniformity in both global descriptors and local feature representations. Moreover, UniLGL introduces a pipeline that leverages a pre-trained single-image Vision Foundation Model (VFM) for feature extraction to enhance the multi-BEV fusion LPR network, enabling strong generalization with only a few LiDAR data for fine-tuning. Finally, based on the mapping between local features on the 2D BEV image and the point cloud, a robust global pose estimator is derived that determines the global minimum of the global pose on <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\text{SE}(3)$</tex-math></inline-formula> without requiring additional registration. To validate the effectiveness of the proposed uniform LGL, extensive benchmarks are conducted in real-world environments, and the results show that the proposed UniLGL is demonstratively competitive compared to other State-of-the-Art (SOTA) LGL methods. Furthermore, UniLGL has been deployed on diverse platforms, including full-size trucks and agile Micro Aerial Vehicles (MAVs), to enable high-precision localization and mapping as well as multi-MAV collaborative exploration in port and forest environments, demonstrating the applicability of UniLGL in industrial and field scenarios. The code will be released at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/shenhm516/UniLGL</uri>.

Accurate detection of objects in perception system is a basic task for autonomous vehicles to operate reliably. The current mainstream paradigms for perception and prediction are able to decompose into three sequential subtasks: detection, tracking and prediction. Much of the previous works focus on one of these subtasks and internal relationship between the various subtasks are therefore ignored. In this work we propose an end-to-end model that is able to jointly carry out 3D detection and motion prediction in the context of autonomous vehicles, which can benefit from joint optimization of two tasks. Considering computational consumption and memory cost, our approach utilizes 2D convolutions across space over a bird's eye view (BEV) representation of the 3D world, not 3D convolutions which require more inference time for a single point cloud. In this paper, the key model is the sequential multi-frame point clouds channels based on BEV, which can help perform object detection and learn temporal information at the same time. Experiments on nuScenes dataset show that the proposed network achieves state-of-the-art results, and the trade-off between accuracy and efficiency of two tasks are obtained. Additionally, we validate the effectiveness of our models through the ablation studies. Specifically, by sharing network, parameters and detection results we can perform two tasks within 32 FPS.

Accurate object detection is a fundamental requirement for autonomous systems to operate in dynamic urban environments. Considering the intricacy of the environment and the occlusion of objects, various point cloud based three-dimensional (3D) object detection methods have been proposed, such as point-based or voxel-based methods. In this paper, a feature selection mechanism is proposed in a voxel-based method to generate bird’s eye view (BEV) feature maps from the original point cloud. For the 3D Backbone, the U-shaped structure and single scale feature selection module are combined. After combining high-level semantics and low-level fine-grained features, the optimized features after the BEV feature maps are applied to region of interest (RoI) refinement, so that the voxel features can better serve the subsequent 3D object detection. The experimental results on KITTI dataset show higher 3D object detection accuracy compared to the state-of-the-art 3D detection methods, which reflects the effectiveness of the proposed architecture.

3D object detection based on LiDAR point clouds is a crucial module in autonomous driving particularly for long range sensing. Most of the research is focused on achieving higher accuracy and these models are not optimized for deployment on embedded systems from the perspective of latency and power efficiency. For high speed driving scenarios, latency is a crucial parameter as it provides more time to react to dangerous situations. Typically a voxel or point-cloud based 3D convolution approach is utilized for this module. Firstly, they are inefficient on embedded platforms as they are not suitable for efficient parallelization. Secondly, they have a variable runtime due to level of sparsity of the scene which is against the determinism needed in a safety system. In this work, we aim to develop a very low latency algorithm with fixed runtime. We propose a novel semantic segmentation architecture as a single unified model for object center detection using key points, box predictions and orientation prediction using binned classification in a simpler Bird's Eye View (BEV) 2D representation. The proposed architecture can be trivially extended to include semantic segmentation classes like road without any additional computation. The proposed model has a latency of 4 ms on the embedded Nvidia Xavier platform. The model is 5X faster than other top accuracy models with a minimal accuracy degradation of 2% in Average Precision at IoU=0.5 on KITTI dataset.

This paper describes new vision-based registration methods utilizing not only cameras on a user's head-mounted display but also a bird's-eye view camera that observes the user from an objective viewpoint. Two new methods, the line constraint method (LCM) and global error minimization method (GEM), are proposed. The former method reduces the number of unknown parameters concerning the user's viewpoint by restricting it to be on the line of sight from the bird's-eye view. The other method minimizes the sum of errors, which is the sum of the distance between the fiducials on the view and the calculated positions of them based on the current viewing parameters, for both the user's view and the bird's-eye view. The methods proposed here reduce the number of points that should be observed from the user's viewpoint for registration, thus improving the stability. In addition to theoretical discussions, this paper demonstrates the effectiveness of our methods by experiments in comparison with methods that use only a user's view camera or a bird's-eye view camera.

In response to the problem that the detection precision of the current 3D object detection algorithm is low when the object is severely occluded, this study proposes an object detection algorithm based on the reconstruction of sparse point clouds in the viewing frustum. The algorithm obtains more local feature information of the sparse point clouds in the viewing frustum through dimensional expansion, performs the fusion of local and global feature information of the point cloud data to obtain point cloud data with more complete semantic information, and then applies the obtained data to the 3D object detection task. The experimental results show that the precision of object detection in both 3D view and BEV (Bird’s Eye View) can be improved effectively through the algorithm, especially object detection of moderate and hard levels when the object is severely occluded. In the 3D view, the average precision of the 3D detection of cars, pedestrians, and cyclists at a moderate level can be increased by 7.1p.p., 16.39p.p., and 5.42p.p., respectively; in BEV, the average precision of the 3D detection of car, pedestrians, and cyclists at hard level can be increased by 6.51p.p., 16.57p.p., and 7.18p.p., respectively, thus indicating the effectiveness of the algorithm.

This paper proposes an adaptive Inverse Perspective Mapping (IPM) algorithm to obtain accurate bird's-eye view images from the sequential images of forward looking cameras. These images are often distorted by the motion of the vehicle; even a small motion can cause a substantial effect on bird'seye view images. In this paper, we propose an adaptive model for the IPM to accurately transform camera images to bird'seye view images by using motion information. Using motion derived from the monocular visual simultaneous localization and mapping (SLAM), experimental result shows that the proposed approaches can provide stable bird's-eye view images, even with large motion during the drive.

The effective representation and feature extraction of 3-D scenes from sparse and unstructured point clouds pose a significant challenge in 3-D object detection. In this article, we propose TransMRE, a network that enables fully sparse multiple observation plane feature fusion using LiDAR point clouds as single-modal input. TransMRE achieves this by sparsely factorizing a 3-D voxel scene into three separate observation planes: XY, XZ, and YZ planes. In addition, we propose Observation Plane Sparse Fusion and Interaction to explore the internal relationship between different observation planes. The Transformer mechanism is employed to realize feature attention within a single observation plane and feature attention across multiple observation planes. This recursive application of attention is done during multiple observation plane projection feature aggregation to effectively model the entire 3-D scene. This approach addresses the limitation of insufficient feature representation ability under a single bird’s-eye view (BEV) constructed by extremely sparse point clouds. Furthermore, TransMRE maintains the full sparsity property of the entire network, eliminating the need to convert sparse feature maps into dense feature maps. As a result, it can be effectively applied to LiDAR point cloud data with large scanning ranges, such as Argoverse 2, while ensuring low computational complexity. Extensive experiments were conducted to evaluate the effectiveness of TransMRE, achieving 64.9 mAP and 70.4 NDS on the nuScenes detection benchmark, and 32.3 mAP on the Argoverse 2 detection benchmark. These results demonstrate that our method outperforms state-of-the-art methods.

Advanced driver assistance systems (ADASs) and autonomous vehicles rely on different types of sensors, such as camera, radar, ultrasonic, and LiDAR to sense the surrounding environment. Compared with the other types of sensors, millimeter-wave automotive radar has advantages in terms of cost and reliability under bad weather conditions (e.g., snow, rain, fog) and doesn't suffer from light condition variations (e.g., darkness). Typical radar devices used in today's commercial vehicles with ADAS features produce sparse point clouds in low angular resolution with a limited number of antennas. In this paper, we present a machine learning-aided signal processing chain to suppress the radar imaging blur effect introduced by the phase migration in time-division multiplexing multi-input multi-output (MIMO) radar, to generate low-level high-resolution radar bird's eye view (BEV) spectra with rich object's features. Compared with radar point clouds, there is no information loss in radar BEV spectra. We then propose a Temporal-fusion, Distance tolerant single stage object detection Network, termed as, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">TDRadarNet</i> , and an enhanced version, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">TDRadarNet+</i> to robustly detect vehicles in both long and short ranges on radar BEVs. We introduce a first-of-its-kind multi-model dataset, containing 14,800 frames of high-resolution low-level radar BEV spectra with synchronized stereo cameras RGB images and three-dimensional (3D) LiDAR point clouds. Our dataset achieves 0.39 m range resolution and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$1.2^\circ$</tex-math></inline-formula> degree azimuth angular resolution with 100 meters maximum detectable range. Moreover, we create a sub-dataset, the Doppler Unfolding dataset, containing 244,140 beam vectors extracted from the 3D radar data cube. With extensive testing and evaluation, we demonstrate that our Doppler unfolding network achieves 93.46% Doppler unfolding accuracy. Compared to YOLOv7, a state-of-the-art image-based object detection network, TDRadarNet achieves a 70.3% Average Precision (AP) for vehicle detection, demonstrating a 21.0% improvement; TDRadarNet+ achieves a 73.9% AP, showing a 24.6% improvement in performance.

The Medieval Church of St. Andrews. Edited by David McRoberts. Pp. xii + 178 + 24 illustrations + plan and a facsimile of the 16th-century ‘bird's eye view’ of St. Andrews. Glasgow: Burns, 1976. £7. - Volume 29 Issue 3

Purpose Sensors are vital for autonomous driving, but they are susceptible to weather noise. To address the performance degradation of point cloud object detection in foggy conditions, this paper aims to propose an efficient dynamic pillar-based point cloud detection network tailored for foggy traffic scenarios. Design/methodology/approach The point cloud data are first pillarized, and then the model adaptively adjusts the feature extraction process through DynamicPillarVFE to significantly enhance the processing capability for sparse and irregular point cloud data. Second, a bird's eye view (BEV) backbone network based on residual connections is used to effectively mitigate the gradient vanishing problem in deep networks. In addition, this paper introduces Triplet Attention (TA) in the feature enhancement part, which can focus on important features more comprehensively and suppress noise interference. Findings In this paper, we mixed a small portion of the Multifog KITTI dataset on top of the KITTI dataset. Experimental results demonstrate that under moderate difficulty 3D evaluation metrics, our method achieves accuracy improvements of 5.03%, 6.66% and 7.78% for cars, pedestrians and cyclists, respectively, compared to the PointPillars baseline, significantly enhancing point cloud object detection performance in foggy conditions. Furthermore, the DPSF-Net architecture achieves an inference speed of 32.36 ms per frame, fully meeting the real-time processing requirements for autonomous driving applications. Originality/value This method is more economical to simulate the influence of sensors under foggy conditions, and through the training of mixed data sets, the network model can better cope with foggy interference.

Large-scale scene point cloud registration with limited overlap is a challenging task due to computational load and constrained data acquisition. To tackle these issues, we propose a point cloud registration method, MT-PCR, based on Modality Transformation. MT-PCR leverages a Bird's Eye View (BEV) capturing the maximal overlap information to improve the accuracy and utilizes images to provide complementary spatial features. Specifically, MT-PCR converts 3D point clouds to BEV images and estimates correspondence by 2D image keypoints extraction and matching. Subsequently, the 2D correspondence estimates are then transformed back to 3D point clouds using inverse mapping. We have applied MT-PCR to Terrestrial Laser Scanning (TLS) and Aerial Laser Scanning (ALS) point cloud registration on the GrAco dataset, involving 8 low-overlap, square-kilometer scale registration scenarios. Experiments and comparisons with commonly used methods demonstrate that MT-PCR can achieve superior accuracy and robustness in large-scale scenes with limited overlap.