Removing Movable Objects from Grid Maps of Self-Driving Cars Using Deep Neural Networks

SUMMARY Mobile robots need Simultaneous Localization and Mapping (SLAM) for autonomous movement in human living environments. The occupancy grid map used in SLAM is a conventional method which makes a map by an occupancy probability in each grid. This method renews a map based on whether an object is observed or not. In order to remove moving objects from a map, an additional method is required. However, conventional methods deal only with actually moving objects, and potential moving objects (e.g., standing humans) are mapped as static objects. Furthermore, only binary states, used or not used, are given to each object in map updating. This paper proposes the immobility area grid map to represent a map by an immobility probability in each grid. The proposed method renews a map based on the identification of observed objects by a robot's sensors, in addition to whether an object is observed or not. We introduce the map update parameter, which is set adaptively from the certainty of identification result of the object. Observed objects can take continuous states, truly static—unknown—truly moving, according to the parameter value. Potential moving objects are not mapped if the parameter takes values corresponding to moving objects. The experimental results show robust mapping in dynamic environments including potential moving objects.

<div class="section abstract"><div class="htmlview paragraph">In the domain of advanced driver assistance systems and autonomous vehicles, precise perception and interpretation of the vehicle's environment are not merely requirements they are the very foundation upon which every aspect of functionality and safety is constructed. One prevalent method of representing the environment is through the use of an occupancy grid map. This map segments the environment into distinct grid cells, each of which is evaluated to determine if it is occupied or free. This evaluation operates under the assumption that each grid cell is independent of the others. The underlying mathematical structure of this system is the binary Bayes filter (BBF). The BBF integrates sensor data from various sources and can incorporate measurements taken at different times. The occupancy grid map does not rely on the identification of individual objects, which allows it to depict obstacles of any shape. This flexibility is a key advantage of this approach. Traditional occupancy grid maps fall short when it comes to predicting dynamic environments due to their lack of a process model. A notable enhancement to this static model is the Bayesian Occupancy Filter (BOF), which, unlike its predecessor, estimates a velocity distribution for each grid cell's occupancy using a histogram filter. However, the BOF's computational demands are high. To address this, research propose representing the dynamic state of grid cells using particles. This method enables the computation of dynamic grid maps in real-time applications, even with larger grid cell sizes and higher resolution. Despite these advancements, dynamic occupancy grid maps remain a relatively new field of study, especially when compared to more established object-tracking approaches. Until now, the BOF has been treated as a distinct research area with minimal overlap with other tracking methodologies. This methodology aims to bridge that gap and foster a more integrated approach to dynamic environment estimation.</div><div class="htmlview paragraph">This study introduces a novel approach to dynamic grid mapping, conceptualized as an approximation of a Random Finite Set (RFS) filter. An RFS is a probabilistic representation of a finite, random collection of objects and their respective states. Finite Set Statistics (FISST) provide a framework for Bayesian filtering of these random finite sets and form the foundation for several multi-object tracking methodologies, such as the Probability Hypothesis Density (PHD) filter. By characterizing the grid as an RFS, we can apply sophisticated concepts from the well-established domain of RFS filtering to dynamic grid mapping. The research develops a filter known as the Probability Hypothesis Density/Multi-Instance Bernoulli (PHD/MIB) filter. This filter alternately represents and propagates the dynamic grid map as a PHD and as multiple instances of Bernoulli filters, thereby offering a more integrated and efficient approach to dynamic environment estimation.</div><div class="htmlview paragraph">Furthermore, this research introduces a Sequential Monte Carlo (SMC) implementation of the PHD/MIB filter, as well as an approximation within the Dempster-Shafer framework, termed the Dempster-Shafer PHD/MIB (DS-PHD/MIB) filter. This DS-PHD/MIB filter necessitates fewer particles than the original PHD/MIB filter, thereby enhancing computational efficiency. The study provides a comprehensive description of an efficient, massively parallel implementation of the DS-PHD/MIB filter. The algorithm's pseudo code is also outlined, offering a clear and concise understanding of its workings. This approach further strengthens the integrated and efficient methodology for dynamic environment estimation.</div><div class="htmlview paragraph">In conclusion, the research delineates the attributes of the DS-PHD/MIB filter and debates its pros and cons in comparison to object-based tracking methodologies, using practical examples for illustration. A quantitative assessment using real-world data demonstrates that the DS-PHD/MIB filter yields consistent state estimation outcomes. It effectively models both the stochastic multi-object transition process and the stochastic multi-object observation process. Moreover, the evaluation affirms the real-time capability of the parallelized implementation of the DS-PHD/MIB filter. It validates its utility for state estimation in dynamic vehicle environments, thereby underscoring its potential for practical applications in dynamic environment estimation. This comprehensive approach offers a promising avenue for future research and development in this field.</div></div>

Moving Object Detection (MOD) from satellite videos plays one of the most fundamental roles in satellite video surveillance. Training a supervised moving object detector typically requires boundary box annotations for object instances, however, this annotation process is time-consuming for satellite videos. In this paper, we propose a weakly supervised method for sidestepping this process, where the supervised information for detecting moving objects on a frame is instead provided by the unsupervised method based on motion knowledge across a video. We adopt the Extended Low-rank and Structured Sparse Decomposition (E-LSD) approach for generating pixel-wise pseudo labels for moving objects. Then the extracted pseudo labels are used for training a Deep Convolutional Neural Network (DCNN) following the Encoder-Decoder architecture with lateral connections for segmenting moving objects from a new frame. We demonstrate the effectiveness of the proposed method on a satellite video dataset, and, compared with five state-of-the-art MOD methods tested, it achieves both improved detection accuracy and promising frame rate.

We present a technique that generates high-quality functional in-field self-tests specifically targeting deep learning (DL) accelerators. These functional tests can be applied in the field during normal operation of a DL accelerator, which is crucial to ensure that the safety and/or reliability requirements are met for any given application, including safety-critical applications such as self-driving cars, robotics, and more.Our technique takes advantage of special architectural characteristics and application properties to achieve high functional test coverage while incurring minimal system-level costs. Moreover, we devise different strategies for the compute units (which support computation operations) and the control units (which control data movement) because these two types of units exhibit different properties. For the compute units of a DL accelerator, we first use combinational ATPG to generate test patterns with high test coverage, which is possible because these units do not contain complex sequential logic. Next, we map the ATPG patterns to one or more equivalent deep neural networks (DNNs) that can be directly executed on the accelerator, which is possible given the well-defined dataflow/reuse algorithm of a DL accelerator. For the control units, we leverage the property that typically only one or a few fixed DNNs are deployed at a time in many application domains (e.g., self-driving cars). Thus, it is sufficient to target only the faults that can directly affect the correctness of the DNNs that are currently deployed. This is done by executing different layers of each target DNN using carefully-crafted input and weight values to maximize test coverage while minimizing test time.We apply our technique using Nvidia’s open-source accelerator as a case study to demonstrate its efficacy. Our results show that our technique achieves high test coverage. For the compute units, 99.9% single stuck-at functional test coverage is achieved. For the control units, we are able to prove that, given any target DNN, 100% coverage can be achieved for a large class of single and multiple fault models. The in-field functional self-test time is also very low, < 17 ms for various representative DNNs. These functional tests can be applied during boot-up, reset, and even concurrently with normal operation by executing DNN test programs directly on the accelerator, without requiring any test support in the hardware.

ABSTRACTOccupancy grid mapping in the field of mobile robotics is an integrative and promising solution for the various prerequisite such as path planning, autonomous navigation, localization, SLAM (Simultaneously Localization And Mapping) etc. The reliability of the occupancy grid mapping depends upon diverse parameters such as sensor accuracy, size of a grid cell, intrinsic and extrinsic parameters, sensor registration, sensor modelling, scanning angle, ambient conditions, etc. This research reveals the uncertainty in the generation of the laser occupancy grid map with the implementation of conventional laser geometry technique when the detected obstacle/target is on the perimeter of the grid cell. During autonomous navigation, the obstacle and the mobile robot are in a dynamic state that consequence in the wrong perception of the environment by identifying the wrong grid cell as occupied in the occupancy grid map when the obstacle is on the perimeter of the grid cell. The examined error is reduced by a newly designed Perimeter-based SLAM (P-SLAM) technique based on the vector algebra, laser geometry, Inverse Sensor Model and coordinate system. The obtained results with the implementation of P-SLAM are validated w.r.t conventional approaches with qualitative and quantitative analysis by performing real-world experiments.

Autonomous vehicles are considered to be one of the most celebrated applications of artificial intelligence techniques. Self-driving cars are driven by complex deep learning neural networks. This book chapter proposes an end—end framework, which can make driving decisions, based on the surroundings, also accounting for the traffic light. Using images, which will be provided by a camera onboard, the framework will make predictions, which will allow it to efficiently and safely maneuver the vehicle. The framework consists of a traffic signal detection model, followed by a deep neural network which takes the input from traffic signal detection along with the images from the camera to generate the next navigation controls.

Deep neural networks (DNNs) have achieved unprecedented success across many scientific and engineering fields in the last decades. Despite its empirical success, unfortunately, recent studies have shown that there are various failure modes and blindspots in DNN models which may result in unexpected serious failures and potential harms, e.g. the existence of adversarial examples and small perturbations. This is not acceptable especially for safety critical and high stakes applications in the real-world, including healthcare, self-driving cars, aircraft control systems, hiring and malware detection protocols. Moreover, it has been challenging to understand why and when DNNs will fail due to their complicated structures and black-box behaviors. Lacking interpretability is one critical issue that may seriously hinder the deployment of DNNs in high-stake applications, which need interpretability to trust the prediction, to understand potential failures, and to be able to mitigate harms and eliminate biases in the model. To make DNNs trustworthy and reliable for deployment, it is necessary and urgent to develop methods and tools that can (i) quantify and improve their robustness against adversarial and natural perturbations, and (ii) understand their underlying behaviors and further correct errors to prevent injuries and damages. These are the important first steps to enable Trustworthy AI and Trustworthy Machine Learning. In this talk, I will survey a series of research efforts in my lab contributed to tackling the grand challenges in (i) and (ii). In the first part of my talk, I will overview our research effort in Robust Machine Learning since 2017, where we have proposed the first attack-agnostic robustness evaluation metric, the first efficient robustness certification algorithms for various types of perturbations, and efficient robust learning algorithms across supervised learning to deep reinforcement learning. In the second part of my talk, I will survey a series of exciting results in my lab on accelerating interpretable machine learning and explainable AI. Specifically, I will show how we could bring interpretability into deep learning by leveraging recent advances in multi-modal models. I'll present recent works in our group on automatically dissecting neural networks with open vocabulary concepts, designing interpretable neural networks without concept labels, and briefly overview our recent efforts on demystifying black-box DNN training process, automated neuron explanations for Large Language Models and the first robustness evaluation of a family of neuron-level interpretation techniques.

The technique of simultaneous localization and mapping has received much attention recently in mobile robotics. When map is being built, robot memorizes environmental information on the plane of grid or topology. Several approaches about this research have been presented so far, but most of them use mapping technique as either grid-based map or topology-based map. This paper proposes some algorithms integrating grid and topology map and its performance is faster and more stable. The proposed scheme uses an occupancy grid map in representing the environment and then formulate topological information in path finding by A* algorithm. The localization process and the dynamic obstacle avoidance can be accomplished by topological information on grid map. On simulator mapping process is shown and the shortest path is decided on grid based map. Then topological information such as direction, distance is calculated on simulator program then transmitted to robot hardware devices. Three ultrasonic sensors and electronic compass are implemented on our mobile robot and two DC geared motor wheels are individually controlled by PD control method. Simulations and experimental results show the performance and efficiency of the proposed scheme.

Moving object segmentation (MOS) is one of the important and wellstudied computer vision tasks that is used in a variety of applications, such as video surveillance systems, human tracking, self-driving cars, and video compression. While traditional approaches to MOS rely on hand-crafted features or background modeling, deep learning methods using Convolution Neural Networks (CNNs) have been shown to be more effective in extracting features and achieving better accuracy. However, most deep learning-based methods for MOS offer scene-dependent solutions, leading to reduced performance when tested on previously unseen video content. Because spatial features are insufficient to represent the motion information, the spatial and temporal features should be used together to succeed in unseen videos. To address this issue, we propose the MOS-Net deep framework, an encoder-decoder network that combines spatial and temporal features using the flux tensor algorithm, 3D CNNs, and ConvLSTM in its different variants. MOS-Net 2.0 is an enhanced version of the base MOS-Net structure, where additional ConvLSTM modules are added to 3D CNNs for extracting long-term spatiotemporal features. In the final stage of the framework the output of the encoder-decoder network, the foreground probability map, is thresholded for producing a binary mask where moving objects are in the foreground and the rest forms the background. In addition, an ablation study has been conducted to evaluate different combinations as inputs to the proposed network, using the ChangeDetection2014 (CDnet2014) which includes challenging videos such as those with dynamic backgrounds, bad weather, and illumination changes. In most approaches, the training and test strategy are not announced, making it difficult to compare the algorithm results. In addition, the proposed method can be evaluated differently as video-optimized or video-agnostic. In video-optimized approaches, the training and test set is obtained randomly and separated from the overall dataset. The results of the proposed method are compared with competitive methods from the literature using the same evaluation strategy. It has been observed that the introduced MOS networks give highly competitive results on the CDnet2014 dataset. The source code for the simulations provided in this work is available online.

Moving object segmentation in video plays an important role in many computer vision applications such as outdoor video surveillance, robotics, self-driving cars, etc. Existing work comprises complicated training procedures considering more number of history frames, fine-tuning on test sequences, and pre-trained models adopted from other applications. Therefore, this paper proposes an end-to-end robust inter-frame change exposure (RIChEx) framework for segmenting moving objects using three consecutive frames. The RIChEx consists inter-frame change gradient (IFCG) aggregation approach responsible for extracting the relevant foreground features to capture substantial changes and spatio-temporal structural dependencies between the consecutive frames. This module consists of multi-scale addition (MSA) blocks for extracting multi-scale features in early layers. Furthermore, to enhance the temporal learning capability of the module, previous recurrent responses and optical flow information are integrated to estimate probable foreground availability and salient object motions. We have evaluated the proposed framework on the benchmark DAVIS-16 dataset to validate the model’s performance.

Occupancy grid state mapping is a key process in robotics and autonomous driving systems. It divides the environment into grid cells that contain information states. In this paper, we propose a modified SMC-BOF method to map and predict occupancy grids. The original SMC-BOF has been widely used in the occupancy grid mapping because it has lower computational costs than the BOF method. However, there are some issues related to conflicting information in dynamic situations. The original SMC-BOF cannot completely control an elongated vehicle that has conflicting information caused by the height difference between backward of vehicle and ground. To overcome this problem, we add confidence weights onto a part of the grid mapping process of the original SMC-BOF using the Linear Opinion Pool. We evaluate our method by LIDAR and stereo vision data in the KITTI dataset.

Moving object detection in video satellite image is studied. A detection algorithm based on deep learning is proposed. The small scale characteristics of remote sensing video objects are analyzed. Firstly, background subtraction algorithm of adaptive Gauss mixture model is used to generate region proposals. Then the objects in region proposals are classified via the deep convolutional neural network. Thus moving objects of interest are detected combined with prior information of sub-satellite point. The deep convolution neural network employs a 21-layer residual convolutional neural network, and trains the network parameters by transfer learning. Experimental results about video from Tiantuo-2 satellite demonstrate the effectiveness of the algorithm.

In this paper, we present an adaptive multi-feature matching method for target detection and target tracking based on an occupancy grid map in dynamic outdoor environments on a moving vehicle equipped with laser radar. The raw scanning points are divided into blocks and the matching strategy of ICP (Iterated Closest Points) algorithm is modified by using the multi-feature information of objects. We introduce a method to adaptively correct the weighting coefficients of the multi-feature in the similarity function. After matching the objects in the two adjacent frames, the online grid map is updated by Bayesian theory and the probability of inverse sensor model. The moving objects are detected with grid map matching. Then, the moving targets are finally tracked by tracker management coupled with Kalman filter. In the experiment, the online grid map is established and the stable results for targets tracking are also achieved.

Nowadays deep neural networks have been applied widely in many applications of computer vision including medical diagnosis and self-driving cars. However, deep neural networks are threatened by adversarial examples usually in which image pixels were perturbed unnoticeable to humans but enough to fool the deep networks. Compared to 2D image adversarial examples, 3D adversarial models are less invasive in the process of attacks, and thus more realistic. There have been many research works on generating 3D adversarial examples. In this paper, we study the robustness of 3D adversarial attacks when the victim camera is placed at different viewpoints. In particular, we find a method to create 3D adversarial examples that can achieve 100% attack success rate from all viewpoints with any integer spherical coordinates. Our method is simple as we only perturb the texture space. We create 3D models with realistic textures using 3D reconstruction from multiple uncalibrated images. With the help of a differentiable renderer, we then apply gradient based optimization to compute texture perturbations based on a set of rendered images, i.e., training dataset. Our extensive experiments show that even only including 1% of all possible rendered images in training, we can still achieve 99.9% attack success rate with the trained texture perturbations. Furthermore, our thorough experiments show high transferability of the multiview robustness of our 3D adversraial attacks across various state-of-the-art deep neural network models.

Lack of transparency in deep neural networks (DNNs) make them susceptible to backdoor attacks, where hidden associations or triggers override normal classification to produce unexpected results. For example, a model with a backdoor always identifies a face as Bill Gates if a specific symbol is present in the input. Backdoors can stay hidden indefinitely until activated by an input, and present a serious security risk to many security or safety related applications, e.g. biometric authentication systems or self-driving cars. We present the first robust and generalizable detection and mitigation system for DNN backdoor attacks. Our techniques identify backdoors and reconstruct possible triggers. We identify multiple mitigation techniques via input filters, neuron pruning and unlearning. We demonstrate their efficacy via extensive experiments on a variety of DNNs, against two types of backdoor injection methods identified by prior work. Our techniques also prove robust against a number of variants of the backdoor attack.