LiDAR-Based Intensity-Aware Outdoor 3D Object Detection.

3D object detection based on LiDAR point clouds is a crucial module in autonomous driving particularly for long range sensing. Most of the research is focused on achieving higher accuracy and these models are not optimized for deployment on embedded systems from the perspective of latency and power efficiency. For high speed driving scenarios, latency is a crucial parameter as it provides more time to react to dangerous situations. Typically a voxel or point-cloud based 3D convolution approach is utilized for this module. Firstly, they are inefficient on embedded platforms as they are not suitable for efficient parallelization. Secondly, they have a variable runtime due to level of sparsity of the scene which is against the determinism needed in a safety system. In this work, we aim to develop a very low latency algorithm with fixed runtime. We propose a novel semantic segmentation architecture as a single unified model for object center detection using key points, box predictions and orientation prediction using binned classification in a simpler Bird's Eye View (BEV) 2D representation. The proposed architecture can be trivially extended to include semantic segmentation classes like road without any additional computation. The proposed model has a latency of 4 ms on the embedded Nvidia Xavier platform. The model is 5X faster than other top accuracy models with a minimal accuracy degradation of 2% in Average Precision at IoU=0.5 on KITTI dataset.

MENet: Map-enhanced 3D object detection in bird’s-eye view for LiDAR point clouds

Accurate object detection is a fundamental requirement for autonomous systems to operate in dynamic urban environments. Considering the intricacy of the environment and the occlusion of objects, various point cloud based three-dimensional (3D) object detection methods have been proposed, such as point-based or voxel-based methods. In this paper, a feature selection mechanism is proposed in a voxel-based method to generate bird’s eye view (BEV) feature maps from the original point cloud. For the 3D Backbone, the U-shaped structure and single scale feature selection module are combined. After combining high-level semantics and low-level fine-grained features, the optimized features after the BEV feature maps are applied to region of interest (RoI) refinement, so that the voxel features can better serve the subsequent 3D object detection. The experimental results on KITTI dataset show higher 3D object detection accuracy compared to the state-of-the-art 3D detection methods, which reflects the effectiveness of the proposed architecture.

In the wave of research on autonomous driving, 3D object detection from the Bird’s Eye View (BEV) perspective has emerged as a pivotal area of focus. The essence of this challenge is the effective fusion of camera and LiDAR data into the BEV. Current approaches predominantly train and predict within the front view and Cartesian coordinate system, often overlooking the inherent structural and operational differences between cameras and LiDAR sensors. This paper introduces CL-FusionBEV, an innovative 3D object detection methodology tailored for sensor data fusion in the BEV perspective. Our approach initiates with a view transformation, facilitated by an implicit learning module that transitions the camera’s perspective to the BEV space, thereby aligning the prediction module. Subsequently, to achieve modal fusion within the BEV framework, we employ voxelization to convert the LiDAR point cloud into BEV space, thereby generating LiDAR BEV spatial features. Moreover, to integrate the BEV spatial features from both camera and LiDAR, we have developed a multi-modal cross-attention mechanism and an implicit multi-modal fusion network, designed to enhance the synergy and application of dual-modal data. To counteract potential deficiencies in global reasoning and feature interaction arising from multi-modal cross-attention, we propose a BEV self-attention mechanism that facilitates comprehensive global feature operations. Our methodology has undergone rigorous evaluation on a substantial dataset within the autonomous driving domain, the nuScenes dataset. The outcomes demonstrate that our method achieves a mean Average Precision (mAP) of 73.3% and a nuScenes Detection Score (NDS) of 75.5%, particularly excelling in the detection of cars and pedestrians with high accuracies of 89% and 90.7%, respectively. Additionally, CL-FusionBEV exhibits superior performance in identifying occluded and distant objects, surpassing existing comparative methods.

Existing learning methods for LiDAR-based applications use 3D points scanned under a pre-determined beam configuration, e.g., the elevation angles of beams are often evenly distributed. Those fixed configurations are task-agnostic, so simply using them can lead to sub-optimal performance. In this work, we take a new route to learn to optimize the LiDAR beam configuration for a given application. Specifically, we propose a reinforcement learning-based learning-to-optimize (RL-L2O) framework to automatically optimize the beam configuration in an end-to-end manner for different LiDAR-based applications. The optimization is guided by the final performance of the target task and thus our method can be integrated easily with any LiDAR-based application as a simple drop-in module. The method is especially useful when a low-resolution (low-cost) LiDAR is needed, for instance, for system deployment at a massive scale. We use our method to search for the beam configuration of a low-resolution LiDAR for two important tasks: 3D object detection and localization. Experiments show that the proposed RL-L2O method improves the performance in both tasks significantly compared to the baseline methods. We believe that a combination of our method with the recent advances of programmable LiDARs can start a new research direction for LiDAR-based active perception. The code is publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">github.com/vniclas/lidar_beam_selection</uri> .

&lt;p&gt;Two-Dimensional (2D) object detection has been an intensely discussed and researched field of computer vision. With numerous advancements made in the field over the years, we still need to identify a robust approach to efficiently conduct classification and localization of objects in our environment by just using our mobile devices. Moreover, 2D object detection limits the overall understanding of the detected object and does not provide any additional information in terms of its size and position in the real world. This work proposes an object localization solution in Three-Dimension (3D) for mobile devices using a novel approach. The proposed method works by combining a 2D object detection Convolutional Neural Network (CNN) model with Augmented Reality (AR) technologies to recognize objects in the environment and determine their real-world coordinates. We leverage the in-built Simultaneous Localization and Mapping (SLAM) capability of Google’s ARCore to detect planes and know the camera information for generating cuboid proposals fromanobject’s2Dboundingbox. Theproposedmethodisfastandefficientforidentifyingeveryday objects in real-world space and, unlike mobile offloading techniques, the method is well designed to work with limited resources of a mobile device.&lt;/p&gt; &lt;p&gt;&lt;br&gt;&lt;/p&gt;

Voxel is one of the common structural representation of 3D point cloud. Due to the sparsity of point cloud generated by light detection and ranging (LiDAR), there is the extreme imbalance in the foreground and background voxels. It decreases the accuracy of 3D object detection, has the negative effect on intelligent driving safety. To overcome this problem, we present a saliency prediction based 3D object detector SP-Det in this article. Although foreground voxels have the sufficient feature of object, it is difficult to localize the foreground region from voxel space with the larger background region. We design an auxiliary learning task, saliency prediction (SP). It benefits 3D detector in identifying the foreground region. SP task uses label diffusion to alleviate the label imbalance. It reduces the learning difficulty of saliency in voxel and bird's eye view (BEV) spaces. After that, to strengthen feature interaction from the sparse foreground region, we design saliency fusion (SF) module to fuse the learning result in SP task. It utilizes voxel and BEV saliency maps as progressive attention to resist the redundant feature from background region. To aggregate more foreground feature inside 3D and BEV region of interest (RoI), we design hybrid grid maps based RoI pooling (Hybrid-RoI pooling). Experiments are conducted in STF dataset. The adverse weather enlarges the sparsity of LiDAR point cloud, increasing the difficulty of object detection. SP-Det identifies and leverages foreground region, and achieves the performance better than the current methods. Hence, we believe that SP-Det benefits to LiDAR based 3D scene understanding in the adverse weather.

3D object detection is a fundamental technique in autonomous driving. However, current LiDAR-based single-stage 3D object detection algorithms do not pay sufficient attention to the encoding of the inhomogeneity of LiDAR point clouds and the shape encoding of each object. This paper introduces a novel 3D object detection network called the spatial and part-aware aggregation network (SPANet), which utilizes a spatial aggregation network to remedy the inhomogeneity of LiDAR point clouds, and embodies a part-aware aggregation network that learns the statistic shape priors of objects. SPANet deeply integrates both 3D voxel-based features and point-based spatial features to learn more discriminative point cloud features. Specifically, the spatial aggregation network takes advantage of the efficient learning and high-quality proposals by providing flexible receptive fields from PointNet-based networks. The part-aware aggregation network includes a part-aware attention mechanism that learns the statistic shape priors of objects to enhance the semantic embeddings. Experimental results reveal that the proposed single-stage method outperforms state-of-the-art single-stage methods on the KITTI 3D object detection benchmark. It achieved a bird’s eye view (BEV) average precision (AP) of 91.59%, 3D AP of 80.34%, and heading AP of 95.03% in the detection of cars.

This paper describes new vision-based registration methods utilizing not only cameras on a user's head-mounted display but also a bird's-eye view camera that observes the user from an objective viewpoint. Two new methods, the line constraint method (LCM) and global error minimization method (GEM), are proposed. The former method reduces the number of unknown parameters concerning the user's viewpoint by restricting it to be on the line of sight from the bird's-eye view. The other method minimizes the sum of errors, which is the sum of the distance between the fiducials on the view and the calculated positions of them based on the current viewing parameters, for both the user's view and the bird's-eye view. The methods proposed here reduce the number of points that should be observed from the user's viewpoint for registration, thus improving the stability. In addition to theoretical discussions, this paper demonstrates the effectiveness of our methods by experiments in comparison with methods that use only a user's view camera or a bird's-eye view camera.

3D object detection and localization based on only a monocular camera always faces its fundamental ill-posed issue to estimate 3D information. In combination with deep neural networks, recent researches have shown encouraging results to tackle this issue. However, most of them are only applied to street-view cameras based on several available small-size datasets and the 3D prediction accuracy of these methods is still low in comparison to traditional estimation methods using stereo-cameras. With the development of drone delivery applications in city spaces, it is also necessary to have a similar method to detect objects and estimate their 3D position from an aerial view. We proposed a novel Simulation to Reality approach to predict the object’s 3D position from an aerial view. An instance segmentation of an object is used as an intermediate representation not only to create a very large dataset for training by simulation but also to minimize the gap between simulation and reality. We designed a feed-forward neural network to predict the 3D position from instance segmentation and integrated it with a range-attention classification to improve accuracy, especially for 3D object detection at far distances. To evaluate our methods, we created two simulation datasets: one for cross-validation with other state-of-the-art methods and the other one for practical experiments on a real drone with a monocular camera. The experiment’s results demonstrate that we not only achieve better accuracy than the state-of-the-art methods using the monocular camera by testing on the same KITTI-3D dataset but also reach close to the accuracy of a stereo-based technique. Since our model is lightweight, we successfully deployed it on a companion computer on the real drone and the results of practical experiences are promising.

Autonomous driving technology is a core enabler for new energy vehicle industrial upgrading and a critical pillar for achieving sustainable development goals (SDGs), especially sustainable urban mobility, low-carbon transportation, and efficient intelligent transportation systems (ITS). However, unstable nighttime low-light perception severely restricts autonomous driving deployment, hindering sustainable transportation development—rooted in visual feature degradation and cross-modal imbalance that impair 3D object detection (autonomous driving’s core perception technology). To address this and advance sustainable autonomous driving, this paper proposes a Bird’s-Eye View (BEV)-based multi-modal 3D object detection approach tailored for nighttime scenarios, integrating low-light adaptive components while preserving the original BEV pipeline. Without modifying core inference, it enhances low-light robustness and cross-modal fusion stability, ensuring reliable perception for sustainable autonomous driving operation. Extensive experiments on the nuScenes nighttime subset quantify performance via rigorous metrics (NDS, mAP, mATE). Results show the method outperforms BEVFusion with negligible parameter/inference overhead, achieving 1.13% NDS improvement. This validates its effectiveness and provides a sustainable technical tool for autonomous driving perception, promoting new energy vehicle popularization, optimizing urban ITS efficiency, reducing perception-related accidents and carbon emissions, and directly contributing to transportation and socio-economic sustainability.

Object detection and localization is a crucial step for inspection and manipulation tasks in robotic and industrial applications. We present an object detection and localization scheme for 3D objects that combines intensity and depth data. A novel multimodal, scale- and rotation-invariant feature is used to simultaneously describe the object's silhouette and surface appearance. The object's position is determined by matching scene and model features via a Hough-like local voting scheme. The proposed method is quantitatively and qualitatively evaluated on a large number of real sequences, proving that it is generic and highly robust to occlusions and clutter. Comparisons with state of the art methods demonstrate comparable results and higher robustness with respect to occlusions.

Point cloud-based three-dimensional (3D) object detection is a critical task in autonomous driving, robotics, and augmented reality, where accurate localization and classification of objects are essential under diverse and challenging scenarios. This work introduces a point cloud-based multi-target 3D object detection framework using Light Detection and Ranging (LiDAR) sensors. The key contribution lies in improving the encoding technique to preserve spatial information and adapt to varying point densities, enabling efficient processing of raw LiDAR data for robust 3D detection. Specifically, we design an efficient network architecture that incorporates a single unified classification head to jointly handle positive and negative samples, simplifying the design and improving training stability. Our model detects all classes (car, pedestrian, cyclist) simultaneously within a single network, enhancing computational efficiency while avoiding separate networks for each class. Furthermore, the proposed feature extraction pipeline combines a voxel-based encoder with a sparsely embedded convolutional backbone and a feature pyramid network, facilitating multi-scale feature representation and effective detection of objects at different scales. This encoding method facilitates the efficient processing of raw LiDAR data, enabling accurate object detection across diverse scenarios. As a result, our model achieves state-of-the-art performance on the KITTI dataset, surpassing baseline methods such as PointPillars and VoxelNet by delivering superior Average Precision (AP) across all difficulty levels for the car class in both bird’s-eye view (BEV) and 3D object detection tasks. Specifically, it achieves AP40 scores of 92.84%, 88.63%, and 85.68% in BEV detection and 87.84%, 76.65%, and 73.60% in 3D detection for Easy, Moderate, and Hard levels, respectively. These results highlight the effectiveness of our encoding technique in enhancing model efficiency and detection accuracy across diverse scenarios.

The Medieval Church of St. Andrews. Edited by David McRoberts. Pp. xii + 178 + 24 illustrations + plan and a facsimile of the 16th-century ‘bird's eye view’ of St. Andrews. Glasgow: Burns, 1976. £7. - Volume 29 Issue 3

Our research objective is construction of 3D map and development of its practical applications. As an application, the authors had proposed 3D control interface for an operator who controls remote control robot. It is well know that an operator can control a robot easily using bird's-eye view. Using 3D control interface, an operator can control the robot at bird's-eye view. The bird's-eye view is constructed from 3D map and robot model. The operator can recognize relationship between the robot and the environment from 3D map and robot model following real robot motion. In this paper, we will describe 3D control interface and its implementation for HELIOS carrier. We will show an experimental result of 3D control interface. From result, we confirmed the effectiveness of the proposed method.