Autonomous Driving using Safe Reinforcement Learning by Incorporating a Regret-based Human Lane-Changing Decision Model

Autonomous driving has attracted significant research interests in the past two decades as it offers many potential benefits, including releasing drivers from exhausting driving and mitigating traffic congestion, among others. Despite promising progress, lane-changing remains a great challenge for autonomous vehicles (AV), especially in mixed and dynamic traffic scenarios. Recently, reinforcement learning (RL) has been widely explored for lane-changing decision makings in AVs with encouraging results demonstrated. However, the majority of those studies are focused on a single-vehicle setting, and lane-changing in the context of multiple AVs coexisting with human-driven vehicles (HDVs) have received scarce attention. In this paper, we formulate the lane-changing decision-making of multiple AVs in a mixed-traffic highway environment as a multi-agent reinforcement learning (MARL) problem, where each AV makes lane-changing decisions based on the motions of both neighboring AVs and HDVs. Specifically, a multi-agent advantage actor-critic (MA2C) method is proposed with a novel local reward design and a parameter sharing scheme. In particular, a multi-objective reward function is designed to incorporate fuel efficiency, driving comfort, and the safety of autonomous driving. A comprehensive experimental study is made that our proposed MARL framework consistently outperforms several state-of-the-art benchmarks in terms of efficiency, safety, and driver comfort.

Self-Preserving Genetic Algorithms (SPGA) combine the evolutionary strategy of a genetic algorithm with safety assurance methods commonly implemented in safe reinforcement learning (SRL), a branch of reinforcement learning (RL) that accounts for safety in the exploration and decision-making process of the agent. Safe learning approaches are especially important in safety-critical environments, where failure to account for the safety of the controlled system could result in the loss of millions of dollars in hardware or bodily harm to people working nearby, as is true of many cyber-physical systems. While SRL is a viable approach to safe learning, there are many challenges that must be taken into consideration when training agents, such as sample efficiency, stability, and exploration---an issue that is easily addressed by the evolutionary strategy of a genetic algorithm. By combining GAs with the safety mechanisms used with SRL, SPGA offers a safe learning alternative that is able to explore large areas of the solution space, addressing SRL's challenge of exploration. This work implements SPGA with both action masking and run time assurance safety strategies to evolve safe controllers for three types of discrete action space environments applicable to cyber physical systems (control, routing, and operations) and under various safety conditions. Training and testing evaluation metrics are compared with results from SRL trained controllers to validate results. SPGA and SRL controllers are trained across 5 random seeds and evaluated on 500 episodes to calculate average wall time to train, average expected return, and percentage of safe action evaluation metrics. SPGA achieves comparable reward and safety performance results with significantly improved training efficiency (55x faster on average), demonstrating the effectiveness of this safe learning approach.

To explore the impact of autonomous vehicles (AVs) on human-driven vehicles (HDVs), a solution for AV to coexist harmoniously with HDV during the car following period when AVs are in low market penetration rate (MPR) was provided. An extension car following framework with two possible soft optimization targets was proposed in this article to improve the experience of HDV followers with different following strategies by deep deterministic policy gradient (DDPG) algorithm. The pretreated Next Generation Simulation (NGSIM) dataset was used for the experiments. 1027 car following events with being redefined were extracted from it, in which 600 of the events were used for training and 427 of the events were used for testing. The different driving strategies obtained from the classical car following models were embedded into virtual environment built by OpenAI gym. The reward function combined safety, efficiency, jerk, and stability was used to encourage the agent with DDPG algorithm to maximize it. The final result reveals that disturbance of HDV followers decreases by 2.362% (strategy a), 8.184% (strategy b), and 13.904% (strategy c), respectively. The disturbance of HDV follower decreases by 14.961% (strategy a), 12.020% (strategy b), and 13.425% (strategy c), respectively. HDV followers with different strategies get less jerk in both soft optimizations. AV passengers get a loss on jerk and efficiency, but safety is enhanced. Also, AV car following performs better than HDV car following in both soft and brutal optimizations. Moreover, two possible solutions for harmonious coexistence of HDVs and AVs when AVs are in low MPR are proposed.

Human-driven Vehicles (HVs) and Autonomous Vehicles (AVs) will inevitably coexist in the future, resulting in mixed traffic conditions. Modeling car-following behavior in such mixed traffic is crucial for better understanding the differences in driving behavior between AVs and HVs, thereby facilitating the development of AV driving strategies to enhance traffic safety. In such mixed traffic, the car-following modes typically include AVs following HVs, HVs following HVs and HVs following AVs. Previous studies have compared the driving behavior of HVs following HVs and HVs following AVs, but few have considered the driving behavior of AVs following HVs in the comparison. To bridge this gap, this paper first decomposes car-following events in mixed traffic extracted from a Lyft level-5 dataset into driving primitives using the Hidden Markov Model with Gaussian Mixture Model (GMM-HMM). Then, the Latent Dirichlet Allocation with GMM (GMM-LDA) is introduced to cluster the driving primitives into different car-following behavior patterns. Based on the identified patterns, the differences across the three car-following modes are further explored. The results show that the driving behavior under three car-following modes differs from each other. Specifically, compared to the HVs following HVs mode, in the AVs following HVs mode, AVs tend to maintain car-following behavior patterns with a longer distance gap and adopt a lower speed to ensure safety. This study offers a comprehensive understanding of car-following behavior in mixed traffic, which will facilitate the development of human-like driving strategies for AVs and enhance the safety of mixed traffic.

A dynamic lane-changing decision and trajectory planning model of autonomous vehicles under mixed autonomous vehicle and human-driven vehicle environment

Reinforcement learning (RL) algorithms have been successfully applied to control tasks associated with unmanned aerial vehicles and robotics. In recent years, safe RL has been proposed to allow the safe execution of RL algorithms in industrial and mission-critical systems that operate in closed loops. However, if the system operating conditions change, such as when an unknown fault occurs, typical safe RL algorithms cannot adapt while retaining past knowledge. Continual RL algorithms have been proposed to address this issue. However, the impact of continual adaptation on the system’s safety is an understudied problem. In this paper, we study the intersection of safe and continual RL. First, we empirically demonstrate that a popular continual RL algorithm, elastic weight consolidation, does not satisfy safety constraints in nonlinear systems subject to varying operating conditions. Specifically, we study the MuJoCo HalfCheetah and Ant environments with velocity constraints and sudden joint loss non-stationarity. Then, we show that an agent trained using constrained policy optimization, a safe RL algorithm, experiences catastrophic forgetting in continual learning settings. With this in mind, we explore a simple reward-shaping method to ensure that elastic weight consolidation prioritizes remembering both safety and task performance for safety-constrained, nonlinear, and non-stationary systems.

Velocity control in car-following behavior with autonomous vehicles using reinforcement learning

With the development of intelligent connected vehicle and artificial intelligence technologies, mixed traffic scenarios where autonomous and human-driven vehicles coexist are becoming increasingly common. Autonomous vehicles need to accurately predict the lane change behavior of preceding vehicles to ensure safety. However, lane change behavior of human-driven vehicles is influenced by both environmental factors and driver preferences, which increases its uncertainty and makes prediction more difficult. To address this challenge, this paper focuses on the mining of driving preferences and the prediction of lane change behavior. We clarify the definition of driving preference and its relationship with driving style and construct a representation of driving operations based on vehicle dynamics parameters and statistical features. A preference feature extractor based on the SimCLR contrastive learning framework is designed to capture high-dimensional driving preference features through unsupervised learning, effectively distinguishing between aggressive, normal, and conservative driving styles. Furthermore, a dual-branch lane change prediction model is proposed, which fuses explicit temporal features of vehicle states with implicit driving preference features, enabling efficient integration of multi-source information. Experimental results on the HighD dataset show that the proposed model significantly outperforms traditional models such as Transformer and LSTM in lane change prediction accuracy, providing technical support for improving the safety and human-likeness of autonomous driving decision-making.

Mixed traffic capacity estimation of autonomous vehicles impact based on empirical data

Based on a survey of 696 visitors to US national parks, we found higher preferences for working/studying/reading (+77%), using social media (+63%), and entertaining (+34%) activities when traveling in an autonomous vehicle (AV) compared to a human-driven vehicle (HV). A multinomial logit-based ranked-choice analysis (between HVs, AVs, and teleportation) suggests that while most travelers enjoy spending time traveling by manually driving and/or engaging in activities of interest in HVs and AVs, some would prefer teleportation instead of spending time traveling. The choice of AVs is significantly influenced by preferences for working/studying/reading and eating/caring activities.

We propose a computationally efficient approach to safe reinforcement learning (RL) for frequency regulation in power systems with high levels of variable renewable energy resources. The approach draws on set-theoretic control techniques to craft a neural network-based control policy that is guaranteed to satisfy safety-critical state constraints, without needing to solve a model predictive control or projection problem in real time. By exploiting the properties of robust controlled-invariant polytopes, we construct a novel, closed-form "safety-filter" that enables end-to-end safe learning using any policy gradient-based RL algorithm. We then apply the safety filter in conjunction with the deep deterministic policy gradient (DDPG) algorithm to regulate frequency in a modified 9-bus power system, and show that the learned policy is more cost-effective than robust linear feedback control techniques while maintaining the same safety guarantee. We also show that the proposed paradigm outperforms DDPG augmented with constraint violation penalties.

Previous empirical behavior analysis on Autonomous Vehicles (AV) mainly focused on vehicles with Adaptive Cruise Control (ACC) system due to the lack of high-level AV dataset. Recently released SAE Level-4 AV datasets such as the Waymo Open Dataset provide great opportunities to evaluate their behavioral impact on traffic flow. In this study, we aim to characterize the empirical Car Following (CF) behaviors of the Waymo autonomous vehicle and compare its feature with human-driven Vehicles (HV), and capture such behavioral differences using the IDM CF model. Our main findings include: (a) AV is much safer than HV, based on our analysis using surrogate safety measures, as time headways and jam spacings of the AV are significantly larger than HV; (b) the response time of AV is also significantly larger than that of HV in response to various types of stimuli; (c) despite the short length of trajectories in the Waymo Open Dataset, we have confirmed that these trajectories are suitable for calibrating some of the IDM parameters; and the calibration results of IDM are consistent with our empirical analysis. Moreover, the modelling results, reveal that the proportion of string unstable behavior of AV is less than that of HV; and (d) for HV, there is generally no significant difference between following AV and following HV except a smaller jam spacing when following AV. Overall, we conclude that currently AV behaves in a conservative way to ensure its safety at the cost of traffic efficiency.

Along with the rapid development of autonomous driving technology, autonomous vehicles are showing a trend of practicality and popularity. Autonomous vehicles perceive environmental information through sensors to provide a basis for the decision making of vehicles. Based on this, this paper investigates the lane-changing decision-making behavior of autonomous vehicles. First, the similarity between autonomous vehicles and moving molecules is sought based on a system-similarity analysis. The microscopic lane-changing behavior of vehicles is analyzed by the molecular-dynamics theory. Based on the objective quantification of the lane-changing intention, the interaction potential is further introduced to establish the molecular-dynamics lane-changing model. Second, the relationship between the lane-changing initial time and lane-changing completed time, and the dynamic influencing factors of the lane changing, were systematically analyzed to explore the influence of the microscopic lane-changing behavior on the macroscopic traffic flow. Finally, the SL2015 lane-changing model was compared with the molecular-dynamics lane-changing model using the SUMO platform. SUMO is an open-source and multimodal traffic experimental platform that can realize and evaluate traffic research. The results show that the speed fluctuation of autonomous vehicles under the molecular-dynamics lane-changing model was reduced by 15.45%, and the number of passed vehicles was increased by 5.93%, on average, which means that it has better safety, stability, and efficiency. The molecular-dynamics lane-changing model of autonomous vehicles takes into account the dynamic factors in the traffic scene, and it reasonably shows the characteristics of the lane-changing behavior for autonomous vehicles.

Autonomous vehicle (AV) uses the artificial intelligence (AI) technologies to control the vehicle without human intervention. The implementation of AV has the advantages over the human-driven vehicle such as reducing the road traffic deaths that caused by human errors, increasing the traffic efficiency and minimizing the carbon emission to save the environment. The main objective of this paper is to develop an AV that keeps a safe distance while following the lead car and remains at the centerline of road. The proposed Deep Reinforcement Learning (DRL) algorithm for the autonomous driving simulation is Deep Deterministic Policy Gradient (DDPG). In this paper, the DDPG model for the path following control, reward function, actor network and critic network are created. The DDPG agent has been trained until 1650-episode rewards have been received. After the training, the proposed DDPG agent has been simulated to verify the performance. Then, the values of the two hyperparameters, which are mini-batch size and actor learning rate, are tuned to obtain the shortest training time.

Transitioning to a driverless city: Evaluating a hybrid system for autonomous and non-autonomous vehicles