Map Building Based on Sensor Fusion for Autonomous Vehicle

We investigate the deployment of a submersible platform with stereovision imaging capability for three-dimensional (3D) mapping of benthic habitats and other sea-floor structures over local areas. A complete framework is studied, comprising: 1) suitable trajectories to be executed for data collection; 2) data processing for positioning and trajectory followed by online frame-to-frame and frame-to-mosaic registration of images, as well as recursive global realignment of positions along the path; and 3) 3D mapping by the fusion of various visual cues, including motion and stereo within a Kalman filter. The computational requirements of the system are evaluated, formalizing how processing may be achieved in real time. The proposed scenario is simulated for testing with known ground truth to assess the system performance, to quantify various errors, and to identify how performance may be improved. Experiments with underwater images are also presented to verify the performance of various components and the overall scheme.

With the fast development of autonomous vehicle (AV) technology and possible popularity of AVs in the near future, a mixed-vehicle type driving environment where both AVs and their surrounding human-driving vehicles drive on the same road will exist and last for a long time. An AV measures its driving environments in real time and make control decisions to ensure driving safety. However, surrounding human-driving vehicles may conduct aggressive driving behaviors (e.g., sudden deceleration, sudden acceleration, sudden left or right lane change) in practice, which requires an AV to make correct control decisions to eliminate the effect of aggressive driving behaviors on its driving safety. In this paper, we propose a reinforcement learning based decision-making system (ReDS) which considers aggressive driving behaviors of surrounding human-driving vehicles during the decision making process. In ReDS, we firstly build a mixture density network based aggressive driving behavior detection method to detect possible aggressive driving behaviors among surrounding vehicles of an AV. We then build a reward function based on aggressive driving behavior detection results and incorporate the reward function into a reinforcement learning model to make optimal control decisions considering aggressive driving behaviors. We use a real-world traffic dataset from the United States Department of Transportation Federal Highway Administration to evaluate optimal control decision determination performance of ReDS in comparison with the state-of-the-art methods. The comparison results show that ReDS can improve optimal control decision success rate by 43% compared with existing methods, which demonstrates that ReDS has good optimal control decision determination performance.

Since several years our laboratory in ENEA Frascati Research Center is involved in development of laser scanners for Cultural Heritage investigation problems. Actually the best result obtained in this field by our laboratory is a 3D Red Green Blue Laser scanner, called RGB-ITR: the main feature of this scanner, further then measuring distances (up to 20m with a sub-millimetric resolution), is the ability to capture remotely color information by three calibrated laser sources: this information is collected for each point sampled by the instrument and is not affected by external light sources' influence. Moreover the ability to acquire color and distance information at the same time and for each point decrease drastically the post-production pipeline of a complete mesh. In this work the results of a complete scan of S. Peter Martyr in Rieti are shown, highlighting the efficiency and robustness of color calibration algorithms introduced for a correct color representation.

Advanced Driver Assistance Systems (ADAS) have emerged as a critical technology in enhancing vehicle safety, providing convenience to drivers, and facilitating the transition toward fully autonomous vehicles. The core of these systems relies heavily on computer vision (CV) technologies, which enable vehicles to interpret and understand their surrounding environment in real time. The integration of computer vision in ADAS allows vehicles to recognize and process various objects such as pedestrians, vehicles, traffic signs, and lane markings, significantly improving both safety and the driving experience. In this paper, we explore the pivotal role of computer vision in the development and implementation of ADAS in autonomous vehicles. We begin by reviewing the evolution of ADAS, detailing the different sub-systems such as lane departure warning, automatic emergency braking, adaptive cruise control, and parking assistance, all of which depend on sophisticated image processing techniques. Furthermore, the paper examines the intersection of computer vision with other emerging technologies such as machine learning, deep learning, and sensor fusion. We discuss how these innovations enable ADAS to function more effectively in complex, dynamic environments. The methodology section focuses on cutting-edge algorithms used for object detection, semantic segmentation, and real-time image processing, which are integral to the functionality of modern ADAS. Additionally, we evaluate the challenges that computer vision systems face in real-world applications, including issues with sensor calibration, environmental conditions, computational limitations, and the integration of data from multiple sensors (e.g., cameras, LiDAR, radar). The paper also presents an analysis of the results obtained from various real-world implementations of ADAS technologies, highlighting improvements in driver safety and the reduction of traffic-related accidents. We explore the potential of computer vision in enabling fully autonomous driving by addressing the current gaps in technology and the advancements needed to close these gaps. Finally, the paper concludes with a discussion on the future of ADAS and the role of computer vision in realizing fully autonomous, reliable, and efficient transportation systems. We argue that continued research and development in computer vision, combined with innovations in deep learning and sensor technologies, are critical to advancing the capabilities and reliability of autonomous vehicles in the near future.

The demand for construction site automation with mobile robots is increasing due to its advantages in potential cost-saving, productivity, and safety. To be realistically deployed in construction sites, mobile robots must be capable of navigating in unstructured and cluttered environments. Furthermore, mobile robots should recognize both static and dynamic obstacles to determine drivable paths. However, existing robot navigation methods are not suitable for construction applications due to the challenging environmental conditions in construction sites. This study introduces an autonomous as-is 3D spatial data collection and perception method for mobile robots specifically aimed for construction job sites with many spatial uncertainties. The proposed Simultaneous Localization and Mapping (SLAM)-based navigation and object recognition methods were implemented and tested with a custom-designed mobile robot platform, Ground Robot for Mapping Infrastructure (GRoMI), which uses multiple laser scanners and a camera to sense and build a 3D environment map. Since SLAM did not detect uneven surface conditions and spatiotemporal objects on the ground, an obstacle detection algorithm was developed to recognize and avoid obstacles and the highly uneven terrain in real time. Given the 3D real-time scan map generated by 3D laser scanners, a path-finding algorithm was developed for autonomous navigation in an unknown environment with obstacles. Overall, the 3D color-mapped point clouds of construction sites generated by GRoMI were of sufficient quality to be used for many construction management applications such as construction progress monitoring, safety hazard identification, and defect detection.

This paper proposes a new 3D gas distribution mapping technique based on Gaussian belief propagation, which is capable of resolving in real time, the concentration estimates in 3D space whilst accounting for the obstacle information within the scenario, the first of its kind in the literature. The gas mapping problem is formulated as a 3D factor graph of Gaussian potentials, the connections of which are conditioned on local occupancy values. The Gaussian belief propagation framework is introduced as the solver and a new hybrid message scheduler is introduced to increase the rate of convergence. The factor graph problem is then redesigned as a dynamically expanding inference task, coupling the information of consecutive gas measurements with local spatial structure obtained by the robot. The proposed algorithm is compared to the state of the art methods in 2D and 3D simulations and is found to resolve distribution maps orders of magnitude quicker than typical direct solvers. The proposed framework is then deployed onboard a ground robot in a 3D mapping and exploration task. The system is shown to be able to resolve multiple sensor inputs and output high resolution 3D gas distribution maps in a GPS denied cluttered scenario in real time. This online inference of complicated gas dispersion provides a new layer of contextual information over its 2D counterparts and enables autonomous systems to take advantage of real time estimates to inform potential next best sampling locations. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —The motivation of this work arises from the need to develop the robotic gas distribution mapping capability that can provide real-time situational awareness of the scenarios. The output distribution maps can be used to inform human first responders as to what areas of the environment contain a hazard, but looking towards autonomous robots, they can also be used by the robot itself to inform where should be measured next to gather more information about the environment. When performing these mapping tasks in unknown indoor environments, it is very important that the sensing robot can build up the knowledge of its physical surroundings together with how the obstacles in the environment affect the 3D gas distribution. The Gaussian belief propagation algorithm allows us to achieve all of this in real-time onboard the sensing robot, something that is yet to be achieved in the literature.

This paper presents an automatic guidance of an autonomous land vehicle based on DGPS and a 3D map. To compensate the positioning error of GPS, we used environmental sensing systems including video cameras, ultrasonic sensors and a 3D scanning laser rangefinder. The 3D map provides 3D environmental information and indicates landmarks for an image processing system to correct the positioning error of GPS. The image processing system also plays the role of obstacle detection. The 3D scanning laser rangefinder is used for speedy and accurate detection of obstacles. This paper describes the DGPS and a 3D map navigated driving system and presents the experimental results.

In contemporary development, autonomous vehicles (AVs) have emerged as a potential solution for sustainable and smart transportation to fulfill the increasing mobility demands whilst alleviating the negative impacts on society, the economy, and the environment. AVs completely depend on a machine to perform driving tasks. Therefore, their quality and safety are critical concerns for driving users. AVs use advanced driver assistance systems (ADASs) that heavily rely on sensors and camera data. These data are processed to execute vehicle control functions for autonomous driving. Furthermore, AVs have a voice communication system (VCS) to interact with driving users to accomplish different hand-free functions. Some functions such as navigation, climate control, media and entertainment, communication, vehicle settings, vehicle status, and emergency assistance have been successfully incorporated into AVs using VCSs. Several researchers have also implemented vehicle control functions using voice commands through VCSs. If a situation occurs when AV has lost control due to malfunctioning or fault in the installed computer, sensors and other associated modules, driving users can control the AV using voice notes to perform some driving tasks such as changing speeds, lanes, breaking, and directing the car to reach a safe condition. Furthermore, driving users need manual control over AV to perform these tasks in some situations, like lane changing or taking an exit due to divergence. These tasks can also be performed with the help of voice commands using VCSs. Therefore, finding the exact voice note used to instruct different actuators in risk situations is crucial. As a result, VCSs can greatly improve safety in critical situations where manual intervention is necessary. AVs’ functions and quality can be significantly increased by integrating a VCS with an ADAS and developing an interactive ADAS. Now, the driver functions are controlled by voice features. Therefore, natural language processing is utilized to extract the features to determine the user’s requirements. The extracted features control the vehicle functions and support driving activities. The existing techniques consume high computation while predicting the user command and causing a reduction in the AVs’ functions. This research issue is overcome by applying the variation continuous input recognition model. The proposed approach utilizes the linear training process that resolves the listening and time-constrained problems and uncertain response issues. The proposed model categorizes the inputs into non-trainable and trainable data, according to the data readiness and listening span. Then, the non-distinguishable data were validated by dividing it into the linear inputs used to improve the response in the AVs. Thus, effectively utilizing training parameters and the data decomposition process minimizes the uncertainty and increases the response rate. The proposed model has significantly improved the exact prediction of users’ voice notes and computation efficiency. This improvement enhances the VCS quality and reliability used to perform hand-free and vehicle control functions. The reliability of these functions ultimately improves the safety of AVs’ driving users and other road users.

Micro aerial vehicles (MAVs) pose specific constraints on onboard sensing, mainly limited payload and limited processing power. For accurate 3D mapping even in GPS denied environments, we have designed a light-weight 3D laser scanner specifically for the application on MAVs. Similar to other custom-built 3D laser scanners composed of a rotating 2D laser range finder, it exhibits different point densities within and between individual scan lines. When rotated fast, such nonuniform point densities influence neighborhood searches which in turn may negatively affect local feature estimation and scan registration. We present a complete pipeline for 3D mapping including pairwise registration and global alignment of 3D scans acquired in-flight. For registration, we extend a state-of-the-art registration algorithm to include topological information from approximate surface reconstructions. For global alignment, we use a graph-based approach making use of the same error metric and iteratively refine the complete vehicle trajectory. In experiments, we show that our approach can compensate for the effects caused by different point densities up to very low angular resolutions and that we can build accurate and consistent 3D maps in-flight with a micro aerial vehicle.

With the development of novel RGB-D visual sensors, data association has been a basic problem in 3D Visual Simultaneous Localization and Mapping (VSLAM). To solve the problem, a VSLAM algorithm based on Multiple Iterative Closest Point (MICP) is presented. By using both RGB and depth information obtained from RGB-D camera, 3D models of indoor environment can be reconstructed, which provide extensive knowledge for mobile robots to accomplish tasks such as VSLAM and Human-Robot Interaction. Due to the limited views of RGB-D camera, additional information about the camera pose is needed. In this paper, the motion of the RGB-D camera is estimated by a motion capture system after a calibration process. Based on the estimated pose, the MICP algorithm is used to improve the alignment. A Kinect mobile robot which is running Robot Operating System and the motion capture system has been used for experiments. Experiment results show that not only the proposed VSLAM algorithm achieved good accuracy and reliability, but also the 3D map can be generated in real time.

Research on autonomous and smart vehicles technology has been growing exponentially in recent years by integrating more and more smart software inside, those intelligent vehicles have several functionalities such as environment recognition, driver drowsiness detection and communication with other cars, etc. Thus, a supervisor is needed to control sensors and actuators, to manage the network and to process data tasks. In most of smart automobiles, embedded RTOS (Real Time Operating System) are used as supervisor to control all data coming from the different sensors, and to meet the real time requirements. In this paper, we investigate the real time software control in smart/autonomous through the design and prototyping of an ADAS (Advanced Driver Assistant System) real time application for vehicle environment recognition and objects detection. On one hand, we port the real time operating system FreeRTOS on the top of Multi/CPU-FPGA platform, and then we point out the advantages of the FreeRTOS to develop a real smart embedded system for environment car detection within a hardware prototype on a top of the FPGA Zynq hardware platform. On the other hand, a deep comparison between FreeRTOS and standard operating embedded Linaro/Linux, is presented. The entry point of our real time ADAS application system is a point cloud coming from LIDAR sensors as an input, and then several steps are defined in the application: data segmentation, objects searching, features extracts and classification. As a first step of validation, our ADAS application detects pedestrians using High-Definition range data.

Advancing research into autonomous micro aerial vehicle navigation requires data structures capable of representing indoor and outdoor 3D environments. The vehicle must be able to update the map structure in real time using readings from range-finding sensors when mapping unknown areas; it must also be able to look up occupancy information from the map for the purposes of localization and path-planning. Mapping models that have been used for these tasks include voxel grids, multi-level surface maps, and octrees. In this paper, we suggest a new approach to 3D mapping using a multi-volume occupancy grid, or MVOG. MVOGs explicitly store information about both obstacles and free space. This allows us to correct previous potentially erroneous sensor readings by incrementally fusing in new positive or negative sensor information. In turn, this enables extracting more reliable probabilistic information about the occupancy of 3D space. MVOGs outperform existing probabilistic 3D mapping methods in terms of memory usage, due to the fact that observations are grouped together into continuous vertical volumes to save space. We describe the techniques required for mapping using MVOGs, and analyze their performance using indoor and outdoor experimental data.

Sensors and their associated data fusion techniques, play a crucial role in Autonomous Vehicle (AV) decision-making applications. Accurately evaluate performance and reliability of the perception sources is an important task to be able to know the consistency of this data fusion. In this paper, a reference data generation framework for assessing perception sensors performances is proposed. Our approach relies on the complementary use of three data sources: a highly precise 3D map with semantic information, a High Density range finder sensor and a GNSS-RTK/INS localization unit. 3D map provides semantic knowledge of the environment and HD range finder precisely senses ego-vehicle surroundings. Finally, 3D map and HD scans are geometrically associated using positioning information in order to combine them and to infer reference data. Thorough experiments were conducted to evaluate and validate the proposed approach. As a proof of concept, performances of a LiDAR-based road plane detection method were evaluated, quantified and reported in terms of precision and recall.

The emergence of Mobile Mapping Systems (MMS) has set a marked paradigm in the photogrammetric and mapping community that has not only facilitated comprehensive 3D mapping of different environments but has also paved way for new aspects of applied research in this direction. Out of the many essential blocks that make these MMS a viable tool for mapping, the positioning and orientation module is considered to be a crucial yet an expensive component. The integration of such a module with mapping sensors has allowed for the extensive implementation of such systems to provide high-quality maps. However, while such systems do not lack in system robustness and performance in general, the deployment of these systems is restricted to applications and environments where a consistent availability of GNSS signals is assured. Extending these MMS to GNSS-denied areas, such as indoor environments, is therefore quite challenging and necessitates the development of an alternative module that can act as a viable substitute to GNSS/INS for system operation without having to resort to an exhaustive modification of the same to function in GNSS-denied locations. In this research, such a case has been considered for the implementation of an indoor MMS using an Unmanned Ground Vehicle (UGV) and a 3D laser scanner for the task of generating high density maps of GNSS-denied indoor areas. To mitigate the absence of GNSS data, this paper proposes a Pseudo-GNSS/INS module integrated framework which utilizes probabilistic Simultaneous Localization and Mapping (SLAM) techniques to estimate the platform pose and heading from 3D laser scanner data. This proposed framework has been implemented based on three major notions: (i) using geometric methods for sparse point cloud extraction to carry out real-time SLAM, (ii) generating position data and geo-referencing signals from these real-time SLAM pose estimates, and (iii) carrying out the entire operation through use of a single 3D mapping sensor. The final geo-referenced point cloud can then be generated through post-processing by the Iterative Closest Projected Point (ICPP) registration technique which also diminishes the effect of sensor measurement noise. The implementation, performance and results of the proposed MMS framework for an indoor mapping system have been presented in this paper that demonstrate the ability of this Pseudo-GNSS/INS framework to operate flexibly in GNSS-denied areas.

Visual understanding of 3-D environments in real time, at low power, is a huge computational challenge. Often referred to as simultaneous localization and mapping (SLAM), it is central to applications spanning domestic and industrial robotics, autonomous vehicles, and virtual and augmented reality. This paper describes the results of a major research effort to assemble the algorithms, architectures, tools, and systems software needed to enable delivery of SLAM, by supporting applications specialists in selecting and configuring the appropriate algorithm and the appropriate hardware, and compilation pathway, to meet their performance, accuracy, and energy consumption goals. The major contributions we present are: 1) tools and methodology for systematic quantitative evaluation of SLAM algorithms; 2) automated, machine-learning-guided exploration of the algorithmic and implementation design space with respect to multiple objectives; 3) end-to-end simulation tools to enable optimization of heterogeneous, accelerated architectures for the specific algorithmic requirements of the various SLAM algorithmic approaches; and 4) tools for delivering, where appropriate, accelerated, adaptive SLAM solutions in a managed, JIT-compiled, adaptive runtime context.