MENet: Map-enhanced 3D object detection in bird’s-eye view for LiDAR point clouds

Accurate object detection is a fundamental requirement for autonomous systems to operate in dynamic urban environments. Considering the intricacy of the environment and the occlusion of objects, various point cloud based three-dimensional (3D) object detection methods have been proposed, such as point-based or voxel-based methods. In this paper, a feature selection mechanism is proposed in a voxel-based method to generate bird’s eye view (BEV) feature maps from the original point cloud. For the 3D Backbone, the U-shaped structure and single scale feature selection module are combined. After combining high-level semantics and low-level fine-grained features, the optimized features after the BEV feature maps are applied to region of interest (RoI) refinement, so that the voxel features can better serve the subsequent 3D object detection. The experimental results on KITTI dataset show higher 3D object detection accuracy compared to the state-of-the-art 3D detection methods, which reflects the effectiveness of the proposed architecture.

Point cloud-based three-dimensional (3D) object detection is a critical task in autonomous driving, robotics, and augmented reality, where accurate localization and classification of objects are essential under diverse and challenging scenarios. This work introduces a point cloud-based multi-target 3D object detection framework using Light Detection and Ranging (LiDAR) sensors. The key contribution lies in improving the encoding technique to preserve spatial information and adapt to varying point densities, enabling efficient processing of raw LiDAR data for robust 3D detection. Specifically, we design an efficient network architecture that incorporates a single unified classification head to jointly handle positive and negative samples, simplifying the design and improving training stability. Our model detects all classes (car, pedestrian, cyclist) simultaneously within a single network, enhancing computational efficiency while avoiding separate networks for each class. Furthermore, the proposed feature extraction pipeline combines a voxel-based encoder with a sparsely embedded convolutional backbone and a feature pyramid network, facilitating multi-scale feature representation and effective detection of objects at different scales. This encoding method facilitates the efficient processing of raw LiDAR data, enabling accurate object detection across diverse scenarios. As a result, our model achieves state-of-the-art performance on the KITTI dataset, surpassing baseline methods such as PointPillars and VoxelNet by delivering superior Average Precision (AP) across all difficulty levels for the car class in both bird’s-eye view (BEV) and 3D object detection tasks. Specifically, it achieves AP40 scores of 92.84%, 88.63%, and 85.68% in BEV detection and 87.84%, 76.65%, and 73.60% in 3D detection for Easy, Moderate, and Hard levels, respectively. These results highlight the effectiveness of our encoding technique in enhancing model efficiency and detection accuracy across diverse scenarios.

In the wave of research on autonomous driving, 3D object detection from the Bird’s Eye View (BEV) perspective has emerged as a pivotal area of focus. The essence of this challenge is the effective fusion of camera and LiDAR data into the BEV. Current approaches predominantly train and predict within the front view and Cartesian coordinate system, often overlooking the inherent structural and operational differences between cameras and LiDAR sensors. This paper introduces CL-FusionBEV, an innovative 3D object detection methodology tailored for sensor data fusion in the BEV perspective. Our approach initiates with a view transformation, facilitated by an implicit learning module that transitions the camera’s perspective to the BEV space, thereby aligning the prediction module. Subsequently, to achieve modal fusion within the BEV framework, we employ voxelization to convert the LiDAR point cloud into BEV space, thereby generating LiDAR BEV spatial features. Moreover, to integrate the BEV spatial features from both camera and LiDAR, we have developed a multi-modal cross-attention mechanism and an implicit multi-modal fusion network, designed to enhance the synergy and application of dual-modal data. To counteract potential deficiencies in global reasoning and feature interaction arising from multi-modal cross-attention, we propose a BEV self-attention mechanism that facilitates comprehensive global feature operations. Our methodology has undergone rigorous evaluation on a substantial dataset within the autonomous driving domain, the nuScenes dataset. The outcomes demonstrate that our method achieves a mean Average Precision (mAP) of 73.3% and a nuScenes Detection Score (NDS) of 75.5%, particularly excelling in the detection of cars and pedestrians with high accuracies of 89% and 90.7%, respectively. Additionally, CL-FusionBEV exhibits superior performance in identifying occluded and distant objects, surpassing existing comparative methods.

High-Definition Maps (HD maps) are essential for the precise navigation and decision-making of autonomous vehicles, yet their creation and upkeep present significant cost and timeliness challenges. The online construction of HD maps using on-board sensors has emerged as a promising solution; however, these methods can be impeded by incomplete data due to occlusions and inclement weather, while their performance in distant regions remains unsatisfying. This paper proposes PriorDrive to address these limitations by directly harnessing the power of various vectorized prior maps, significantly enhancing the robustness and accuracy of online HD map construction. Our approach integrates a variety of prior maps uniformly, such as OpenStreetMap's Standard Definition Maps (SD maps), outdated HD maps from vendors, and locally constructed maps from historical vehicle data. To effectively integrate such prior information into online mapping models, we introduce a Hybrid Prior Representation (HPQuery) that standardizes the representation of diverse map elements. We further propose a Unified Vector Encoder (UVE), which employs fused prior embedding and a dual encoding mechanism to encode vector data. To improve the UVE's generalizability and performance, we propose a segment-level and point-level pre-training strategy that enables the UVE to learn the prior distribution of vector data. Through extensive testing on the nuScenes, Argoverse 2 and OpenLane-V2, we demonstrate that PriorDrive is highly compatible with various online mapping models and substantially improves map prediction capabilities. The integration of prior maps through PriorDrive offers a robust solution to the challenges of single-perception data, paving the way for more reliable autonomous driving.

Compared to standard navigation maps, HD maps contain precise additional information for automated vehicles. To exploit this information, a lane-level accurate localization estimate within the HD map needs to be available. Here, the computation and memory overhead of the localization algorithm needs to be as small as possible to enable a reasonable usage of HD map information. Since the accuracy of GNSS measurements is insufficient for an accurate localization at lane-level, HD maps commonly contain geo-referenced landmarks like lane markings and boundaries that can be detected by series-production camera sensors, especially on highways. These measurements are fused with the motion measurements provided by the ABS and ESP sensors of the vehicle within a Bayesian filtering framework. Here, the most popular localization algorithm coming from the field of robotics is the particle filter (PF). However, for a robust estimation, a sufficiently high number of particles is required which is in contradiction to the objective to minimize the computational and memory overhead of the localization algorithm. Hence, in this contribution a memory and run-time efficient Extended Kalman-Filter (EKF) based solution is proposed employing measurements from series-production vehicle sensors and a public HD map from two highways near Dusseldorf. Experimental results achieved with the EKF are comparable with the localization accuracy provided by the PF while run-time and memory consumption are considerably reduced.

Voxel is one of the common structural representation of 3D point cloud. Due to the sparsity of point cloud generated by light detection and ranging (LiDAR), there is the extreme imbalance in the foreground and background voxels. It decreases the accuracy of 3D object detection, has the negative effect on intelligent driving safety. To overcome this problem, we present a saliency prediction based 3D object detector SP-Det in this article. Although foreground voxels have the sufficient feature of object, it is difficult to localize the foreground region from voxel space with the larger background region. We design an auxiliary learning task, saliency prediction (SP). It benefits 3D detector in identifying the foreground region. SP task uses label diffusion to alleviate the label imbalance. It reduces the learning difficulty of saliency in voxel and bird's eye view (BEV) spaces. After that, to strengthen feature interaction from the sparse foreground region, we design saliency fusion (SF) module to fuse the learning result in SP task. It utilizes voxel and BEV saliency maps as progressive attention to resist the redundant feature from background region. To aggregate more foreground feature inside 3D and BEV region of interest (RoI), we design hybrid grid maps based RoI pooling (Hybrid-RoI pooling). Experiments are conducted in STF dataset. The adverse weather enlarges the sparsity of LiDAR point cloud, increasing the difficulty of object detection. SP-Det identifies and leverages foreground region, and achieves the performance better than the current methods. Hence, we believe that SP-Det benefits to LiDAR based 3D scene understanding in the adverse weather.

Traditionally, point cloud-based 3D object detectors are trained on annotated, non-sequential samples taken from driving sequences (e.g. the KITTI dataset). However, by doing this, the developed algorithms renounce to exploit any dynamic information from the driving sequences. It is reasonable to think that this information, which is available at test time when deploying the models in the experimental vehicles, could have significant predictive potential for the object detection task. To study the advantages that this kind of information could provide, we construct a dataset of dynamic occupancy grid maps from the raw KITTI dataset and find the correspondence to each of the KITTI 3D object detection dataset samples. By training a Lidar-based state-of-the-art 3D object detector with and without the dynamic information we get insights into the predictive value of the dynamics. Our results show that having access to the environment dynamics improves by 27% the ability of the detection algorithm to predict the orientation of smaller obstacles such as pedestrians. Furthermore, the 3D and bird's eye view bounding box predictions for pedestrians in challenging cases also see a 7% improvement. Qualitatively speaking, the dynamics help with the detection of partially occluded and far-away obstacles. We illustrate this fact with numerous qualitative prediction results.

Robust and accurate localization is an essential component for robotic navigation and autonomous driving. The use of cameras for localization with high definition map (HD Map) provides an affordable localization sensor set. Existing methods suffer from pose estimation failure due to error prone data association or initialization with accurate initial pose requirement. In this paper, we propose a cost-effective vehicle localization system with HD map for autonomous driving that uses cameras as primary sensors. To this end, we formulate vision-based localization as a data association problem that maps visual semantics to landmarks in HD map. Specifically, system initialization is finished in a coarse to fine manner by combining coarse GPS (Global Positioning System) measurement and fine pose searching. In tracking stage, vehicle pose is refined by implicitly aligning the semantic segmentation result between image and landmarks in HD maps with photometric consistency. Finally, vehicle pose is computed by pose graph optimization in a sliding window fashion. We evaluate our method on two datasets and demonstrate that the proposed approach yields promising localization results in different driving scenarios. Additionally, our approach is suitable for both monocular camera and multi-cameras that provides flexibility and improves robustness for the localization system.

In response to the problem that the detection precision of the current 3D object detection algorithm is low when the object is severely occluded, this study proposes an object detection algorithm based on the reconstruction of sparse point clouds in the viewing frustum. The algorithm obtains more local feature information of the sparse point clouds in the viewing frustum through dimensional expansion, performs the fusion of local and global feature information of the point cloud data to obtain point cloud data with more complete semantic information, and then applies the obtained data to the 3D object detection task. The experimental results show that the precision of object detection in both 3D view and BEV (Bird’s Eye View) can be improved effectively through the algorithm, especially object detection of moderate and hard levels when the object is severely occluded. In the 3D view, the average precision of the 3D detection of cars, pedestrians, and cyclists at a moderate level can be increased by 7.1p.p., 16.39p.p., and 5.42p.p., respectively; in BEV, the average precision of the 3D detection of car, pedestrians, and cyclists at hard level can be increased by 6.51p.p., 16.57p.p., and 7.18p.p., respectively, thus indicating the effectiveness of the algorithm.

3D object detection is a fundamental technique in autonomous driving. However, current LiDAR-based single-stage 3D object detection algorithms do not pay sufficient attention to the encoding of the inhomogeneity of LiDAR point clouds and the shape encoding of each object. This paper introduces a novel 3D object detection network called the spatial and part-aware aggregation network (SPANet), which utilizes a spatial aggregation network to remedy the inhomogeneity of LiDAR point clouds, and embodies a part-aware aggregation network that learns the statistic shape priors of objects. SPANet deeply integrates both 3D voxel-based features and point-based spatial features to learn more discriminative point cloud features. Specifically, the spatial aggregation network takes advantage of the efficient learning and high-quality proposals by providing flexible receptive fields from PointNet-based networks. The part-aware aggregation network includes a part-aware attention mechanism that learns the statistic shape priors of objects to enhance the semantic embeddings. Experimental results reveal that the proposed single-stage method outperforms state-of-the-art single-stage methods on the KITTI 3D object detection benchmark. It achieved a bird’s eye view (BEV) average precision (AP) of 91.59%, 3D AP of 80.34%, and heading AP of 95.03% in the detection of cars.

Autonomous driving technology is a core enabler for new energy vehicle industrial upgrading and a critical pillar for achieving sustainable development goals (SDGs), especially sustainable urban mobility, low-carbon transportation, and efficient intelligent transportation systems (ITS). However, unstable nighttime low-light perception severely restricts autonomous driving deployment, hindering sustainable transportation development—rooted in visual feature degradation and cross-modal imbalance that impair 3D object detection (autonomous driving’s core perception technology). To address this and advance sustainable autonomous driving, this paper proposes a Bird’s-Eye View (BEV)-based multi-modal 3D object detection approach tailored for nighttime scenarios, integrating low-light adaptive components while preserving the original BEV pipeline. Without modifying core inference, it enhances low-light robustness and cross-modal fusion stability, ensuring reliable perception for sustainable autonomous driving operation. Extensive experiments on the nuScenes nighttime subset quantify performance via rigorous metrics (NDS, mAP, mATE). Results show the method outperforms BEVFusion with negligible parameter/inference overhead, achieving 1.13% NDS improvement. This validates its effectiveness and provides a sustainable technical tool for autonomous driving perception, promoting new energy vehicle popularization, optimizing urban ITS efficiency, reducing perception-related accidents and carbon emissions, and directly contributing to transportation and socio-economic sustainability.

EPDet: Enhancing point clouds features with effective representation for 3D object detection

LiDAR-based 3D object detection and localization are crucial components of autonomous navigation systems, including autonomous vehicles and mobile robots. Most existing LiDAR-based 3D object detection and localization approaches primarily use geometric or structural feature abstractions from LiDAR point clouds. However, these approaches can be susceptible to environmental noise due to adverse weather conditions or the presence of highly scattering media. In this work, we propose an intensity-aware voxel encoder for robust 3D object detection. The proposed voxel encoder generates an intensity histogram that describes the distribution of point intensities within a voxel and is used to enhance the voxel feature set. We integrate this intensity-aware encoder into an efficient single-stage voxel-based detector for 3D object detection. Experimental results obtained using the KITTI dataset show that our method achieves comparable results with respect to the state-of-the-art method for car objects in 3D detection and from a bird's-eye view and superior results for pedestrian and cyclic objects. Furthermore, our model can achieve a detection rate of 40.7 FPS during inference time, which is higher than that of the state-of-the-art methods and incurs a lower computational cost.

Segmenting or detecting objects in sparse Lidar point clouds are two important tasks in autonomous driving to allow a vehicle to act safely in its 3D environment. The best performing methods in 3D semantic segmentation or object detection rely on a large amount of annotated data. Yet annotating 3D Lidar data for these tasks is tedious and costly. In this context, we propose a self-supervised pretraining method for 3D perception models that is tailored to autonomous driving data. Specifically, we leverage the availability of synchronized and calibrated image and Lidar sensors in autonomous driving setups for distilling self-supervised pre-trained image representations into 3D models. Hence, our method does not require any point cloud nor image annotations. The keyingredient of our method is the use of superpixels which are used to pool 3D point features and 2D pixel features in visually similar regions. We then train a 3D network on the self-supervised task of matching these pooled point features with the corresponding pooled image pixel features. The advantages of contrasting regions obtained by superpixels are that: (1) grouping together pixels and points of visually coherent regions leads to a more meaningful contrastive task that produces features well adapted to 3D semantic segmentation and 3D object detection; (2) all the different regions have the same weight in the contrastive loss regardless of the number of 3D points sampled in these regions; (3) it mitigates the noise produced by incorrect matching of points and pixels due to occlusions between the different sensors. Extensive experiments on autonomous driving datasets demonstrate the ability of our image-to-Lidar distillation strategy to produce 3D representations that transfer well on semantic segmentation and object detection tasks.

Easy, yet robust long-term localization is still an open topic in research. Existing approaches require either dense maps, expensive sensors, specialized map features or proprietary detectors.We propose using semantic segmentation on a monocular camera to localize directly in a HD map as used for automated driving. This combines lightweight, yet powerful HD maps with the simplicity of monocular vision and the flexibility of neural networks.The major challenges arising from this combination are data association and robustness against misdetections. Association is solved efficiently by applying distance transform on binary per-class images. This provides not only a fast lookup table for a smooth gradient as needed for pose-graph optimization, but also dynamic association by default.A sliding-window pose graph optimization combines single image detections with vehicle odometry, smoothing results and helping overcome even misclassifications in consecutive frames.Evaluation against a highly accurate 6D visual localization shows that our approach can achieve accuracy levels as required for automated driving, being one of the most lightweight and flexible methods to do so.