Real-Time Dynamic Object Detection for Autonomous Driving Using Prior 3D-Maps

This study aimed to explore how a three-dimensional (3D) map affects the acquisition of survey knowledge in complex virtual environments (VEs). Subject navigated in virtual and real subway stations, where the virtual station was augmented with either a 3D floor map, a 3D building map, or neither. After subject's navigation in the station, they made the judgment of spatial relative directions between objects. The analyses of response time showed that participants responded faster when they were assisted by the 3D floor or building map than without a map aid in virtual stations. There was no significant difference between survey knowledge acquired in the virtual station augmented with the 3D floor or building map and in the real station. Participants with the 3D floor map performed best in spatial judgments. Implications of these findings to a 3D map design were discussed.

Vehicle self-localization techniques, such as GNSS and Velodyne, can benefit greatly by accessing to an accurate 3D building map, especially in urban areas. Though, an accurate automatic 3D building map reconstruction is challenging in dense urban areas, in which buildings are located close to each other. In general, 3D building map can be generated from adding accurate height information of the building roofs to a detailed 2D map. But 2D building maps usually only provide footprint of the buildings which is not enough for 3D shape modeling of complicated buildings in urban areas such as Shinjuku, Tokyo, Japan. On the other hand, with the development of laser scanning technology, airborne laser scanning (ALS) has now become easily accessible for large scale 3D mapping. However, an automatic 3D map reconstruction using only airborne point cloud is not feasible for dense urban areas, because of the difficulty of defining boundary of each building individually. This paper presents an automatic 3D building map reconstruction technique which integrates airborne laser point cloud with 2D boundary map. Results from Shinjuku area is evaluated for the proposed technique.

Most existing Simultaneous Localization and Mapping (SLAM) systems rely on the assumption of static environments to achieve reliable and efficient mapping. However, such methods often suffer from degraded localization accuracy and mapping consistency in dynamic settings, as they lack explicit mechanisms to distinguish between static and dynamic elements. To overcome this limitation, we present BMP-SLAM, a vision-based SLAM approach that integrates semantic segmentation and Bayesian motion estimation to robustly handle dynamic indoor scenes. To enable real-time dynamic object detection, we integrate YOLOv5, a semantic segmentation network that identifies and localizes dynamic regions within the environment, into a dedicated dynamic target detection thread. Simultaneously, the data association Bayesian mobile probability proposed in this paper effectively eliminates dynamic feature points and successfully reduces the impact of dynamic targets in the environment on the SLAM system. To enhance complex indoor robotic navigation, the proposed system integrates semantic keyframe information with dynamic object detection outputs to reconstruct high-fidelity 3D point cloud maps of indoor environments. The evaluation conducted on the TUM RGB-D dataset indicates that the performance of BMP-SLAM is superior to that of ORB-SLAM3, with the trajectory tracking accuracy improved by 96.35%. Comparative evaluations demonstrate that the proposed system achieves superior performance in dynamic environments, exhibiting both lower trajectory drift and enhanced positioning precision relative to state-of-the-art dynamic SLAM methods.

Visual-inertial navigation system (VINS) is widely used for autonomous platforms but suffers from drifting over a long time. To remedy this situation, a lightweight 3D prior map-aided visual-inertial navigation system is presented in this paper, which tightly couples the visual-inertial data stream with a lightweight prior map involving 3D line information. To fill the gaps between 3D maps and 2D images, the mutual geometric feature of line segments is utilized to connect these two types of information in different dimensions. By detecting and matching line features in two data sources, the line pairs are utilized as constraints in the nonlinear optimization model and added to the existing factor graph framework in a tightly coupled form. Meanwhile, a fast line feature tracking strategy is employed to monitor and remove extreme outliers, which will further improve the reliability of this structural characteristic during the cross-modality localization. The effectiveness of the proposed method is evaluated by public indoor unmanned aerial vehicles (UAV) datasets, and outdoor unmanned ground vehicles (UGV) datasets generated by the CARLA simulator.

An improved lightweight deep neural network with knowledge distillation for local feature extraction and visual localization using images and LiDAR point clouds

Localization and mapping is an important factor for mobile robots. 3D map information is preferred even for mobile robots that move in a 2D space. In order to build 3D map of an unknown environment, we used only one laser scanner, and obtained 3D environmental information by continuously tilting the laser scanner itself using a pulse motor. We applied 2D scan matching method by extracting a horizontal space data of scan information in order to connect scan data for 3D map building. Then, we conducted 3D map building experiments and confirmed that 3D map was obtained in a quite high speed, and the resolution is the same as 2D map.

3D mapping of outdoor environments by scan matching and motion averaging

Conventional perception-planning pipelines of autonomous vehicles (AV) utilize deep learning (DL) techniques that typically generate deterministic outputs without explicitly evaluating their uncertainties and trustworthiness. Therefore, the downstream decision-making components may generate unsafe outputs leading to system failure or accidents, if the preceding perception component provides highly uncertain information. To mitigate this issue, this article proposes a coherent safe perception-planning framework that quantifies and transfers DL-based perception uncertainties. Following the Bayesian Deep Learning paradigm, we design a probabilistic 3D object detector that extracts objects from LiDAR point clouds while quantifying the corresponding aleatoric and epistemic uncertainty. A chance-constrained motion planner is designed to formulate an explicit link between DL-based perception uncertainties and operation risk and generate safe and risk-bounding trajectories. The proposed framework is validated through various challenging scenarios in the CARLA simulator. Experiment results demonstrate that our framework can effectively capture the uncertainties in DL, and generate trajectories that bound the risk under DL perception uncertainties. It also outperforms counterpart approaches without explicitly evaluating the uncertainties of DL-based perception.

Detection and tracking of on-road moving objects are crucial for vehicle perception. Data Fusion of perception sensors like cameras, lidar, and radar, allows the automotive perception to achieve a certain level of robustness and performance. The efficiency of vehicle perception determines its accuracy to take better decisions related to road safety. In this article, we propose two approaches for the detection and tracking of on-road moving objects by combining information from lidar and radar sensors. In this work, we propose to use a probabilistic occupancy grid for sensor fusion of radar and lidar point cloud, and a modified YOLO DeepSORT approach to detect, classify, and track the moving objects. Using both traditional and deep learning approaches, data fusion allows us to improve the robustness of the automotive perception pipeline. We finally used the nuScenes dataset to evaluate and compare the accuracy of our approaches with state-of-the-art trackers and detectors. We achieved an AMOTA score of 68.3% and a mAP score of 68.7% for the tracking and detection tasks respectively.

3D object detection based on LiDAR point clouds is a crucial module in autonomous driving particularly for long range sensing. Most of the research is focused on achieving higher accuracy and these models are not optimized for deployment on embedded systems from the perspective of latency and power efficiency. For high speed driving scenarios, latency is a crucial parameter as it provides more time to react to dangerous situations. Typically a voxel or point-cloud based 3D convolution approach is utilized for this module. Firstly, they are inefficient on embedded platforms as they are not suitable for efficient parallelization. Secondly, they have a variable runtime due to level of sparsity of the scene which is against the determinism needed in a safety system. In this work, we aim to develop a very low latency algorithm with fixed runtime. We propose a novel semantic segmentation architecture as a single unified model for object center detection using key points, box predictions and orientation prediction using binned classification in a simpler Bird's Eye View (BEV) 2D representation. The proposed architecture can be trivially extended to include semantic segmentation classes like road without any additional computation. The proposed model has a latency of 4 ms on the embedded Nvidia Xavier platform. The model is 5X faster than other top accuracy models with a minimal accuracy degradation of 2% in Average Precision at IoU=0.5 on KITTI dataset.

Real-time object detection in indoor settings is a challenging area of computer vision, faced with unique obstacles such as variable lighting and complex backgrounds. This field holds significant potential to revolutionize applications like augmented and mixed realities by enabling more seamless interactions between digital content and the physical world. However, the scarcity of research specifically fitted to the intricacies of indoor environments has highlighted a clear gap in the literature. To address this, our study delves into the evaluation of existing datasets and computational models, leading to the creation of a refined dataset. This new dataset is derived from OpenImages v7[14], focusing exclusively on 32 indoor categories selected for their relevance to real-world applications. Alongside this, we present an adaptation of a CNN detection model, incorporating an attention mechanism to enhance the model’s ability to discern and prioritize critical features within cluttered indoor scenes. Our findings demonstrate that this approach is not just competitive to existing state-of-the-art models in accuracy and speed but also opens new avenues for research and application in the field of real-time indoor object detection.

With the rapid development of autonomous driving, real-time object detection on 360° images becomes more and more important. In this paper, we propose a panoramic virtual dataset for training object detectors on 360° images. The most important feature of our dataset includes (1) an auto-generated city scene is created for rendering 360° dataset. (2) annotation work for this dataset is automatic. In addition, we propose a modified YOLOv3 model called Pano-YOLO for real-time panoramic object detection. Compared with YOLOv3, mAP of Pano-YOLO drops 0.39%. While speed is 32.47% faster. Experiments are performed to show that models trained on our virtual dataset can be applied in real world. And Pano-YOLO is capable of real-time object detection task on high-resolution 360° panoramic images and videos.

Real-time object detection and ranging of multiple objects on the road are the essential tasks in the field of autonomous driving. In this paper, we introduce a system for simultaneous detection and ranging of vehicles, people, non-motor vehicles and lanes based on RGB-D images. Among them, the detection of vehicles, people and non-motor vehicles belongs to general detection task and the lane detection belongs to segmentation task. In order to improve the accuracy and speed, we use two networks to complete these two tasks. We propose a real-time synchronization method with multi-GPU, which achieves separate training and simultaneous detection of lane detectior and vehicle, people and non-motor vehicle detector. We also propose a center-selective ranging module based on binocular ranging technology to distance the detected object. The system reaches nearly 15 FPS with four 1080Ti GPUs. We construct datasets about these problems including daytime and night in which the system achieves high accuracy. A real-time test of the system on the streets of Tianjin, China has been conducted by us, it has proved that the system can be applied to actual driving.

CAOM: Change-aware online 3D mapping with heterogeneous multi-beam and push-broom LiDAR point clouds

<div>The need for 3D mapping is on the rise to meet the requirements of a growing and diverse group of end-users. Existing 3D mapping systems, which have been classified according to the mode of operation as stationary, mobile and aerial, tend to serve one mode of operation only and are considered cost-prohibitive for many end-users. Unmanned aerial vehicles (UAVs) have experienced rapid growth since their introduction and their usage in 3D mapping is likewise accelerating at a rapid pace. This dissertation presents the design, development and implementation of a LiDAR-based generic 3D mapping system that can be used in the three mapping modes (stationary, mobile and UAV-based). The system provides direct georeferencing capabilities through optimized selected multimodal sensors. A fundamental part of this dissertation is the smart integration of the 3D mapping system components both on the hardware and software levels, along with a new mapping scheme that enables platform-independent deployment ability. </div><div>This research project also presents a rigorous non-linear uncertainty predictive model for the generic developed system and introduces a very low-cost variant of the system to be used in stationary and handheld mode. The developed multipurpose mapping system is tested in different environments for the three modes of operation, demonstrating its practicality, versatility and ease of deployment. To maximize the ease of deployment for diverse end-users, careful consideration is given to the mapping system components so that the developed system is ultra-lightweight, compact, and multipurpose. Additionally, this dissertation proposes a colorization workflow to make use of available optical imagery in the colorization process of the LiDAR point cloud. Lastly, the study compares two different 3D mapping approaches: 3D LiDAR-based mapping and a low-cost optical-based 3D structure from motion (SfM) workflow. The comparison is achieved by performing a real-world case study of digital surface model (DSM) generation by the two aforementioned approaches. Real-world testing that includes qualitative and quantitative validation against accurate state-of-the-art high-end LiDAR equipment proves the successful design, development and deployment of the developed crosscutting LiDAR-based 3D mapping system.</div>