Multimodal Trajectory Predictions for Autonomous Driving without a Detailed Prior Map

The future of transportation is tightly connected to Autonomous Driving (AD). While a lot of progress has been made in recent years, there are still obstacles to overcome. One of the most critical issues is the safety verification of AD. A scenario-based verification approach that shifts tests from the fields to a virtual environment seems like a sophisticated approach to tackle the safety verification as tests need to be revised whenever changes are made to the AD. However, collecting and labelling data that can be used to construct scenarios is expensive and time-consuming to compute. In this work, we propose a unified framework for trajectory generation and validation in a consistent and principled way. We first explore methods to generate artificial trajectories that resemble the previously captured ones. More specifically, we consider two architectures based on Generative Adversarial Networks (GANs): recurrent GANs and a recurrent Autoencoder in combination with GANs. Moreover, we investigate the use of different metrics to evaluate the quality of generated trajectories which is a nontrivial task.

In recent years, deep neural networks have demonstrated high efficiency in solving the problems associated with image processing and analysis. The latest studies in generative adversarial neural networks open up broad prospects for solving problems in image style transfer, cross-domain adaptation, and the generation of new data for a target probability distribution. The ability of generative networks to implement complex and weakly formalized transformations allows the information systems based on generative adversarial technologies to solve problems in machine learning in a manner similar to human creative thinking. This paper presents the results of an experimental study on the design, training and deployment of an intelligent control system for a CNC laser engraver based on the generative adversarial network. In order to train deep models, large amounts of data are required. Therefore, one of the main challenges in this experimental study is a lack of labelled training dataset. The proposed control system for the laser engraver based on the deep generative model allows decreasing the time of the manufacturing process.

Segmenting or detecting objects in sparse Lidar point clouds are two important tasks in autonomous driving to allow a vehicle to act safely in its 3D environment. The best performing methods in 3D semantic segmentation or object detection rely on a large amount of annotated data. Yet annotating 3D Lidar data for these tasks is tedious and costly. In this context, we propose a self-supervised pretraining method for 3D perception models that is tailored to autonomous driving data. Specifically, we leverage the availability of synchronized and calibrated image and Lidar sensors in autonomous driving setups for distilling self-supervised pre-trained image representations into 3D models. Hence, our method does not require any point cloud nor image annotations. The keyingredient of our method is the use of superpixels which are used to pool 3D point features and 2D pixel features in visually similar regions. We then train a 3D network on the self-supervised task of matching these pooled point features with the corresponding pooled image pixel features. The advantages of contrasting regions obtained by superpixels are that: (1) grouping together pixels and points of visually coherent regions leads to a more meaningful contrastive task that produces features well adapted to 3D semantic segmentation and 3D object detection; (2) all the different regions have the same weight in the contrastive loss regardless of the number of 3D points sampled in these regions; (3) it mitigates the noise produced by incorrect matching of points and pixels due to occlusions between the different sensors. Extensive experiments on autonomous driving datasets demonstrate the ability of our image-to-Lidar distillation strategy to produce 3D representations that transfer well on semantic segmentation and object detection tasks.

LiDAR sensors are critical for 3D scene perception in autonomous driving, providing high spatial accuracy but suffering from limited temporal resolution. Point cloud interpolation, which reconstructs intermediate frames from sparsely captured inputs, effectively enhances temporal continuity. However, the challenges posed by non-linear and non-rigid motion in dynamic scenarios remain unresolved by existing methods, which often sacrifice accuracy for efficiency. To solve the above problems, we propose MSTSINet (Multiscale Transformer and SSM Interpolation Network), a novel framework that combines multiscale forward flow prediction with non-linear trajectory compensation for fast and accurate interpolation. Unlike conventional bidirectional flow-based methods, our forward flow-guided approach reduces inference time while improving robustness to complex motion. MSTSINet integrates multiscale motion modeling through a novel hierarchical feature extraction and semantic-aware cross-frame attention, progressively improving forward flow estimation for accurate non-linear trajectory prediction. To further improve motion fidelity and temporal coherence, we introduce a point-wise compensation mechanism during inference and design a novel multiscale loss function that supervises forward flow learning across varying temporal resolutions. Experiments on autonomous driving benchmarks demonstrate that MSTSINet significantly outperforms existing state-of-the-art methods, achieving up to a 26.39% reduction in Chamfer Distance and substantially improving inference speed. These advantages make it well-suited for real-time deployment in autonomous driving scenarios.

Autonomous driving requires robust and highly accurate perception technologies. Various deep learning algorithms based on only image processing satisfy this requirement, but few such algorithms are based on LiDAR. However, images are only one part of the perceptible sensors in an autonomous driving vehicle; LiDAR is also essential for the recognition of driving environments. The main reason why there exist few deep learning algorithms based on LiDAR is a lack of data. Recent translation technology using generative adversarial networks (GANs) has been proposed to deal with this problem. However, these technologies focus on only image-to-image translation, although a lack of data occurs more often with LiDAR than with images. LiDAR translation technology is required not only for data augmentation, but also for driving simulation, which allows algorithms to practice driving as if they were commanding a real vehicle, before doing so in the real world. In other words, driving simulation is a key technology for evaluating and verifying algorithms which are practically applied to vehicles. In this paper, we propose a GAN-based LiDAR translation algorithm for autonomous driving and driving simulation. It is the first LiDAR translation approach that can deal with various types of weather that are based on an empirical approach. We tested the proposed method on the JARI data set, which was collected under various adverse weather scenarios with diverse precipitation and visible distance settings. The proposed method was also applied to the real-world Spain data set. Our experimental results demonstrate that the proposed method can generate realistic LiDAR data under adverse weather conditions.

Evolutionary conditional GANs for supervised data augmentation: The case of assessing berry number per cluster in grapevine

This paper proposes a novel approach for generating 3-dimensional complex geological facies models based on deep generative models. It can reproduce a wide range of conceptual geological models while possessing the flexibility necessary to honor constraints such as well data. Compared with existing geostatistics-based modeling methods, our approach produces realistic subsurface facies architecture in 3D using a state-of-the-art deep learning method called Generative Adversarial Networks (GANs). GANs couple a generator with a discriminator and each uses a deep Convolutional Neural Network (CNN). The networks are trained in an adversarial manner until the generator can create "fake" images that the discriminator cannot distinguish from "real" images. We extend the original GAN approach to 3D geological modeling at the reservoir scale. The GANs are trained using a library of 3D facies models. Once the GANs have been trained, they can generate a variety of geologically realistic facies models constrained by well data interpretations. This geomodelling approach using GANs has been tested on models of both complex fluvial depositional systems and carbonate reservoirs that exhibit progradational and aggradational trends. The results demonstrate that this deep learning-driven modeling approach can capture more realistic facies architectures and associations than existing geostatistical modeling methods, which often fail to reproduce heterogeneous nonstationary sedimentary facies with apparent depositional trend.

Medical image reconstruction is typically an ill-posed inverse problem. In order to address such ill-posed problems, the prior distribution of the sought after object property is usually incorporated by means of some sparsity-promoting regularization. Recently, prior distributions for images estimated using generative adversarial networks (GANs) have shown great promise in regularizing some of these image reconstruction problems. In this work, we apply an image-adaptive GAN-based reconstruction method (IAGAN) to reconstruct high fidelity images from incomplete medical imaging data. It is observed that the IAGAN method can potentially recover fine structures in the object that are relevant for medical diagnosis but may be oversmoothed in reconstructions with traditional sparsity-promoting regularization.

Widely used recommendation systems do not meet all industry requirements, so the search for more advanced methods for creating recommendations continues. The proposed new methods based on Generative Adversarial Networks (GAN) have a theoretical comparison with other recommendation algorithms; however, real-world comparisons are needed to introduce new methods in the industry. In our work, we compare recommendations from the Generative Adversarial Network with recommendation from the Deep Semantic Similarity Model (DSSM) on real-world case of airflight tickets. We found a way to train the GAN so that users receive appropriate recommendations, and during A/B testing, we noted that the GAN-based recommendation system can successfully compete with other neural networks in generating recommendations. One of the advantages of the proposed approach is that the GAN training process avoids a negative sampling, which causes a number of distortions in the final ratings of recommendations. Due to the ability of the GAN to generate new objects from the distribution of the training set, we assume that the Conditional GAN is able to solve the cold start problem.

The article discusses the importance of selfdriving cars to improve road safety and reduce the number of accidents caused by human error. Self-driving cars not only reduce human error but also help reduce driver fatigue. We further explore the use of computer vision in autonomous cars, with previous research relying on deep learning algorithms with LiDAR sensors which can be expensive. The authors propose a more cost-effective approach using simple computer vision algorithms such as color space transformation, Canny edge detection, and Hough line transformation to detect lane lines and steer the car accordingly. This approach requires less operational hardware and can be implemented using affordable boards like Raspberry Pi and Nvidia Jetson Nano. The article also highlights the reconstruction of a remote-controlled car that had a 95% accuracy using a certain set of parameters was a tool for understanding autonomous cars better.

Nowadays, closed-set perception methods for autonomous driving perform well on datasets containing normal scenes. However, they still struggle to handle anomalies in the real world, such as unknown objects that have never been seen while training. The lack of public datasets to evaluate the model performance on anomaly and corner cases has hindered the development of reliable autonomous driving systems. Therefore, we propose a multimodal Synthetic Dataset for Anomaly and Corner case detection, called SDAC, which encompasses anomalies captured from multi-view cameras and the LiDAR sensor, providing a rich set of annotations for multiple mainstream perception tasks. SDAC is the first public dataset for autonomous driving that categorizes anomalies into object, scene, and scenario levels, allowing the evaluation under different anomalous conditions. Experiments show that closed-set models suffer significant performance drops on anomaly subsets in SDAC. Existing anomaly detection methods fail to achieve satisfactory performance, suggesting that anomaly detection remains a challenging problem. We anticipate that our SDAC dataset could foster the development of safe and reliable systems for autonomous driving.

In the recent era, a lot of interest is attracted by the autonomous vehicles which can sense surroundings and navigate without human intervention. Object detection and recognition form a major part of autonomous driving systems. Lidar sensors can be used to capture point clouds of driving environment. Detecting multiple 3D objects in point clouds in real time and defining their boundaries with the help of 3D bounding boxes are critical in motion planning by self driving systems. This paper proposes a LiDAR-based 3D object detection system that operates in real-time, with emphasis on autonomous driving scenarios. A state-of-the-art 2D standard object detector for RGB images, YOLOv4, is used as the base for object detection. The multi-class 3D bounding boxes are generated using a complex regression approach. An Euler-Region-Proposal Network (E-RPN) is used to predict the pose of the object. The proposed model receives point cloud data as input and outputs 3D bounding boxes with classes in real-time. The experiments done on the KITTI benchmark dataset proves that the proposed system outperforms existing methods in terms of accuracy and performance.

Generative Adversarial Networks (GANs) have proven to be a preferred method of synthesizing fake images of ob-jects, such as faces, animals, and automobiles. It is not surprising these models can also generate ISO-compliant, yet synthetic iris images, which can be used to augment training data for iris matchers and liveness detectors. In this work, we trained one of the most recent GAN mod-els (StyleGAN3 [15]) to generate fake iris images with two primary goals: (i) to understand the GAN's ability to produce "never-before-seen" irises, and (ii) to investigate the phenomenon of identity leakage as a function of the GAN's training time. Previous work has shown that personal biometric data can inadvertently flow from training data into synthetic samples, raising a privacy concern for subjects who accidentally appear in the training dataset. This paper presents analysis for three different iris matchers at varying points in the GAN training process to diagnose where and when authentic training samples are in jeopardy of leaking through the generative process. Our results show that while most synthetic samples do not show signs of identity leak-age, a handful of generated samples match authentic (training) samples nearly perfectly, with consensus across all matchers. In order to prioritize privacy, security, and trust in the machine learning model development process, the re-search community must strike a delicate balance between the benefits of using synthetic data and the corresponding threats against privacy from potential identity leakage.

One of the world's most public serious issues is road fatalities. Every year, millions of people die or suffer non-fatal injuries or impairments in car accidents worldwide. Using helmets, wearing seat belts, avoiding drunk driving, and adhering to traffic laws can all help to reduce fatalities and injuries. Road accidents are caused by driver's poor decisions or actions while driving. Furthermore, accidents can occur as a result of vehicle component failures or the drivers' health. Controlling speed and monitoring the health of the driver/passengers can help to improve vehicle and passenger safety. Automobiles may be equipped with driver/passenger health monitoring systems with sensors to help prevent accidents. The proposed work suggests a sensor-based prototype for autonomous vehicles with smart seat belts, and road rules such as speed control, stopping the vehicle when the traffic signal shows red light, parking/no parking control, and horn/no horn control systems using Radio Frequency (RF) communication. The prototype developed works in both manual and autonomous modes. Furthermore, this article also suggests including the YOLOR (You Only Learn One Representation) for real-time obstacle detection for self-driving and autonomous cars, as YOLOR demonstrates 5 percent higher performance than YOLOv4 (You Only Look Once version 4).

This paper presents a robust system for mitigating adversarial and natural GPS disruptions by presenting: (1) a software-based defense mechanism against spoofing attacks using generative adversarial networks (GANs), The system detects unauthorized or spoofed GPS signals from a hardware based spoofer, and (2) deep neural network models to infer positioning information in GPS-degraded /denied environments using the novel idea of GPS satellite constellation fingerprint. As the GAN and Satellite constellation fingerprinting are used together in a unified framework, we call it the “GANSAT positioning system.” Intuitively, the GANSAT neural networks implicitly learn a representation of the aggregation of the hardware fingerprints of the satellite’s in the GPS constellation at a given location and time. To demonstrate the approach, raw GPS signals were collected from the satellite transmitters using a software defined radio (SDR) at five different locations in the Florida panhandle area of the United States. Additionally, a GPS spoofer is implemented using a SDR and an open source software and used in an uncontrolled laboratory environment for spoofing the GPS signals at the aforementioned locations. In our experiments, the GANSAT framework yields ~99.5% accuracy for the task of identifying and filtering the spoofed GPS signals from real ones. It also achieves ~100% accuracy for the task of location estimation.