Combining multi-agent deep deterministic policy gradient and rerouting technique to improve traffic network performance under mixed traffic conditions

A mixed traffic car-following behavior model

Freeway merging areas are frequent sources of traffic bottlenecks, but connected and autonomous technologies may offer solutions for optimizing merging operations and reducing impediments to the flow of traffic. In this study, we develop a mixed traffic flow model for merging areas with four types of vehicles: human-driven vehicles (HVs), connected vehicles (CVs), autonomous vehicles (AVs), and connected autonomous vehicles (CAVs). There are two parts to the model. First, the concepts of autonomous penetration (Pa) and connected penetration (Pc) are introduced to characterize vehicle demands, and the extent to which the different proportions of vehicle types impact the traffic flow in the merging area is examined. Second, the driving behaviors of the vehicles are modeled to capture the various distinctions between them. In addition, two critical features, CAV degradation and merging control strategies under the connected environment, are incorporated. The SUMO platform was utilized to model the mixed traffic flow and conduct simulation experiments on the impacts of Pa, Pc and merging control strategies. The results show that (1) increasing Pa consistently enhances merging area performance at any level of Pc, especially when Pc=1; (2) when Pa<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\geq$</tex-math></inline-formula>0.7 and Pc<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\leq$</tex-math></inline-formula>0.7, the operational performance of mixed traffic flow decrease with increasing Pc; (3) when Pa<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\leq$</tex-math></inline-formula>0.2, the implementation of a merging control strategy reduces the operational performance of the merging area with increasing Pc; and (4) the utilization of a merging control strategy enhances the performance of the merging area when Pa<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\geq$</tex-math></inline-formula>0.3 and Pc<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\geq$</tex-math></inline-formula>0.8, and the enhancement increases significantly with Pa.

Connected Autonomous Vehicles (CAVs) are going to share the road environment with human drivers until their full market deployment achievement. In this context, this paper tackles and solves the problem of decentralized crossing at unsignalized intersections for mixed traffic flows, composed of both connected human-driven vehicles (CHVs) and CAVs. To this aim, a novel cooperative fully-distributed control protocol for CAVs is proposed, augmenting the Adaptive Cruise Control (ACC) action based on proximity sensors measurements, with an additional networked protocol exploiting Vehicle-to-Vehicle (V2V) information for the cooperative evaluation of the Time-to-Intersection of all incoming vehicles within the communication range. This further collaborative action automatically adapts the CAVs motion at intersection avoiding both collisions with other vehicles and intersection deadlocks, while improving the intersection throughput. Safety and mobility performances are evaluated via the purposely designed Virtual Testing platform - Mixed Traffic Simulator (MiTraS)- combining the Matlab/Simulink environment with the SUMO traffic simulator. The analysis is carried out for an exemplary two-lane four-way unsignalized intersection considering both different traffic demands and different CAVs penetration rates. The unavoidable presence of variable delays in information delivery due to the wireless communication network has been also explicitly accounted for in both control design and validation. The extensive simulation analysis confirms how the inclusion of CAVs, equipped with the proposed control algorithm, within the mixed traffic flow strongly improves both the safety and mobility performances of the intersection. Lastly, we disclose how the benefits of the proposed controller increase with CAV penetration.

This paper investigates the problem of age of information (AoI) aware radio resource management for a platooning system. Multiple autonomous platoons exploit the cellular wireless vehicle-to-everything (C-V2X) communication technology to disseminate the cooperative awareness messages (CAMs) to their followers while ensuring timely delivery of safety-critical messages to the Road-Side Unit (RSU). To lower the computational load at the RSU and cope with the challenges of dynamic channel conditions, we exploit a distributed resource allocation framework based on multi-agent reinforcement learning (MARL), where each platoon leader (PL) acts as an agent and interacts with the environment to learn its optimal policy. Motivated by the existing literature in RL, we propose two novel MARL frameworks based on the multi-agent deep deterministic policy gradient (MADDPG), named Modified MADDPG, and Modified MADDPG with task decomposition. Both algorithms train two critics with the following goals: A global critic which estimates the global expected reward and motivates the agents toward a cooperating behavior and an exclusive local critic for each agent that estimates the local individual reward. Furthermore, based on the tasks each agent has to accomplish, in the second algorithm, the holistic individual reward of each agent is decomposed into multiple sub-reward functions where task-wise value functions are learned separately. Numerical results indicate our proposed algorithms' effectiveness compared with other contemporary RL frameworks, e.g., federated reinforcement learning (FRL) in terms of AoI performance and CAM message transmission probability.

Variable speed limits (VSLs) have a potential to improve mobility and safety of motorway through harmonisation of traffic flow. However, the success of VSL is highly dependent on drivers’ compliance to the displayed speed limits. Emerging technologies in the field of connected autonomous vehicles (CAVs) is likely to revolutionise the way VSL will be operated in the near future. VSL can be integrated with CAVs where CAVs will automatically obey the displayed speed limits. However, 100% penetration rate of CAVs is a long‐term goal. At the initial stage of deployment, CAVs will coexist with manually driven vehicles on motorways. This study explores ways to implement VSL under a mixed traffic condition where CAVs and manually driven vehicles coexist in the traffic stream. A hybrid approach combining particle swarm optimisation with proportional‐integrated‐derivative is proposed to improve VSL implementation under a range of mixed traffic conditions. The simulation results showed that the proposed method successfully achieved the desired speed limits and rapidly stabilised the system at the initial and transient points at penetration rates of 50 and 75%, whereas the method encountered serious overshoots at lower penetration rates.

This study evaluates the effects of various market penetration rates of connected autonomous vehicles (CAV) on traffic operations at roundabouts. We have utilized a simulation-and an optimization-based approach for this purpose. The simulation-based approach included calibrated car following models with different driving behavior parameters for a mixed fleet of conventional vehicles and CAVs with various penetration rates ranging from 0% to 100%. We also used an optimization-based approach for the 100% CAV market penetration rate case to evaluate operations while trajectories of CAVs were optimized by a central controller. The simulation results showed that CAVs improved traffic operations in under-and semi-saturated flow conditions. Nevertheless, the optimization of CAV trajectories resulted in significant delay reductions and improvements in the roundabout performance. These results indicate that CAVs have great potentials for improving traffic operations once an effective control algorithm is available.

Multiagent reinforcement learning (RL) is widely used and can successfully solve many problems in the real world. In the multiagent RL system, a global critic network is used to guide each agent’s strategy to update to learn the most beneficial strategy for the collective. However, the global critic network also makes the current agent’s learning be affected by other agents’ strategies, which leads to unstable learning. To solve this problem, we propose dynamic decomposed multiagent deep deterministic policy gradient (DD-MADDPG): a new network that considers both global and local evaluations and adaptively adjusts the agent’s attention to the two evaluations. Besides, the use of the experience replay buffer by multiagent deep deterministic policy gradient (MADDPG) produces outdated experience, and the outdated strategies of other agents further affect the learning of the current agent. To reduce the influence of other agents’ outdated experience, we propose TD-Error and Time-based experience sampling (T2-PER) based on DD-MADDPG. We evaluate the proposed algorithm’s performance according to the learning stability and the average return obtained by the agents. We have conducted experiments in the MPE environment. The results show that the proposed method has better stability and higher learning efficiency than MADDPG and has a certain generalization ability.

With increasing travel demand, highway congestion and accidents have become more frequent. As an essential component of intelligent transportation systems (ITS), Hard Shoulder Running (HSR) provides a dynamic strategy to mitigate congestion by temporarily opening shoulder lanes, yet traditional methods often fail to adapt to real-time traffic changes and overlook the shoulder’s critical role in ensuring emergency vehicle access. This study proposes a novel HSR optimization framework based on the Long Short-Term Memory (LSTM) and Multi-Agent Deep Deterministic Policy Gradient (MADDPG) reinforcement learning algorithm, integrated with an improved A* algorithm for EV lane clearing. LSTM is used to extract temporal features from traffic data to support intelligent decision-making in MADDPG, enhanced with prioritized experience replay and importance sampling. The EV lane-clearing task is formulated using graph theory, and the improved A* algorithm determines the optimal path for clearing. A simulation case was developed using Simulation of Urban MObility (SUMO) for a section of the Jinan City Ring Highway, China, evaluating four levels of traffic service. Results show the proposed method reduces total travel time by 14.6%, Time Integrated Time-to-Collision (TIT) by 45.2%, and CO2 emissions by 11.9%. Additionally, with EV intervention, braking times are reduced by up to 376.3% and travel time by 18.1% using the improved A* strategy. These findings demonstrate that the integrated LSTM-MADDPG and A* approach effectively enhances highway traffic efficiency, safety, and sustainability under complex real-world conditions.

Introduction To address the urgent need for enhanced grid flexibility in high-penetration distributed photovoltaic (PV) systems, this paper proposes a novel two-stage framework integrating multi-objective cluster partitioning and multi-agent reinforcement learning (MARL) for cooperative peak shaving. Methods First, a comprehensive multi-objective optimization model defines PV cluster partitioning using three domain-specific metrics: available peak shaving margin (APSM), peak shaving synergy (PSS), and node correlation degree (NCD). This approach replaces conventional clustering algorithms, ensuring partitions balance electrical compactness, regulation capacity, and operational coordination. Subsequently, each partitioned cluster acts as an autonomous agent in a hierarchical MARL framework. Leveraging the multi-agent deep deterministic policy gradient (MADDPG) algorithm, agents collaboratively optimize active/reactive power allocation among clusters while coordinating unit-level control within clusters. Results Validated on the IEEE-33 node system, the strategy achieves a 29.39% reduction in peak-to-valley difference and 54.44% improvement in power balance deviation, significantly smoothing net load curves and enhancing voltage stability. Discussion This work demonstrates how MARL-driven coordination unlocks the inherent flexibility of distributed PV clusters, providing a scalable solution for grid-edge resource participation in system-level peak shaving.

• A grid-aware multi-agent reinforcement learning was developed for EV charging. • Public charging data worked with domestic demand to build a real environment. • A dual-head actor optimised the acceptance and cross-site redirection of EV charging. • A centralised attention-based critic and delayed replay buffer ensured stable learning. • The RL policy reduced peak load variability by 40 % and threshold violations by 37 %. As electric vehicles (EV) become more widespread, managing charging demand is critical for grid stability and efficient resource allocation. However, prior work often optimises single sites or treats demand as exogenous, without explicitly modelling grid capacity constraints or the aggregate effects of redirection, and it rarely offers an integrated capacity-aware view that links domestic and public charging under system-level coordination. This study, therefore, develops a multi-agent reinforcement learning framework with cross-site redirection to optimise EV charging operations. The research area focuses on the Perth and Kinross region of Scotland in the UK, using real public charging records from the council database as well as domestic charging statistics from the UK Department for Transport. Charging sites are modelled as cooperative agents using the Multi-Agent Deep Deterministic Policy Gradient algorithm, trained in a spatiotemporal environment that integrates public charging data and simulated domestic demand. We contribute by unifying public and home charging demand in one learning environment, introducing a cross-site actor trained under centralised training and decentralised execution, and incorporating a delayed replay buffer with short-horizon forecasts so redirection aligns with future congestion. The learned policy reduces peak-hour load standard deviation by up to 40 % and lowers cumulative threshold violations by 37 % compared to the baseline. Distinct weekday and weekend strategies emerge, enabling adaptive coordination under varying demand patterns. The study provides interpretable control for EV networks, balancing peak demand and service quality across sites while addressing the system-level coordination.

Financial investigation decision is a complex affair that highly relies on expert knowledge. In the era of big data, it remains practical to make intelligent financial investigation decisions via computing technology. As a consequence, this work introduces reinforcement learning to search for effective decision schemes that are adaptive to the diverse financial environment. Therefore, an intelligent financial investment decision model based on multi-agent reinforcement learning is proposed in this paper. First, the long short-term memory (LSTM) model is utilized to strengthen financial investment data. Then, the hidden Markov model (HMM) is utilized to make a preliminary judgment on financial investment strategy. Next, a multi-agent deep deterministic policy gradient (MADDPG) is employed to set the goal of agent reinforcement and determine decision indicators based on the financial market environment. Multiple agents (investors) form a cooperative strategy through learning and interaction to achieve the optimal investment decision in the financial market. The experimental results on realistic data indicate that the proposal can play an important role in investment decision-making.

It is interesting to realize that traffic intersections provide an essential mechanism to handle traffic flows in different directions while ironically traffic bottlenecks, gridlocks, and accidents tend to occur in the vicinity of the intersections. Meanwhile, with the recent developments in internet of things (IoT) technologies, there is a great potential for integrating them for improving operational efficiencies of road infrastructure and connected autonomous vehicles (CAVs). This paper aims to leverage the capabilities of both autonomous intersection manager (AIM) and CAVs for more energy-saving and safe-traffic management. A mixed-traffic environment where human-driven vehicles (HDVs) and CAVs sharing the same road is considered. A two-layer framework is adopted to handle signal and vehicle controls effectively. The first layer is a signal control layer where the AIM receives the traffic network states, trains with the data through machine learning (ML), and outputs a set of optimal green times for each intersection phase. The second layer is a decentralized-vehicle control layer where the CAVs receive the signal phase and timing (SpaT) information from the AIM to compute the optimal speed values. The proposed solution helps the CAVs to minimize idling at red signals or to speed up to safely arrive at and pass through a green signal. Our proposed framework is designed to optimize intersection efficiency and minimize vehicle average delay and fuel consumption. All experiments have been conducted in a microscopic traffic simulation environment, the PTV-VISSIM, simulating real-world dynamics of vehicles and drivers’ behaviors based on decades of field-data.

Multi-agent combat is a combat scenario in multiagent reinforcement learning (MARL). In this combat, agents use reinforcement learning methods to learn optimal policies. Actually, policy may be changed, which leads to a non-stationary environment. In this case, it is difficult to predict opponents' policies. Many reinforcement learning methods try to solve nonstationary problems. Most of the previous works put all agents into a frame and model their policies to deal with non-stationarity of environments. But, in a combat environment, opponents can not be in the same frame as our agents. We group opponents and our agents into two frames, only considering opponents as a part of the environment. In this paper, we focus on the problem of modelling opponents' policies in non-stationary environments. To solve this problem, we propose an algorithm called Additional Opponent Characteristics Multi-agent Deep Deterministic Policy Gradient (AOC-MADDPG) with the following contributions: (1) We propose a new actor-critic framework to deal with nonstationarity of environments in MARL, so that agents can adapt to more complex environments. (2) A model for opponents' policies is built by introducing observations and actions of the opponents into the critic network as additional characteristics. We evaluate our AOC-MADDPG algorithm in two multi-agent combat environments. As a result, our approach significantly outperforms the baseline. Agents trained by our method can get higher rewards in non-stationary environments.

This paper combines millimeter-wave radar (MMW) data with connected autonomous vehicle (CAV) trajectory data to estimate queue length on a second-by-second basis. Firstly, queued vehicles on multiple lanes with the same traffic movement are mapped to a virtual lane. Then, the presence of CAV or human-driven vehicle (HDV) for any given queueing index is determined. A Bayesian joint probability model is subsequently established for queue length expectation, considering the existence of the queued vehicle type as a condition. The average time headway and dissipation speed distribution are derived from departure timestamps, which allows for the calculation of the prior probability ratio of queue length. Lastly, the maximum likelihood estimation (EM) algorithm is employed for iteratively estimating the CAV penetration rate. Simulation results demonstrate the method offers a compelling trade-off between precision and second-based real-time performance, while field test results further confirm the wide applicability under various traffic conditions.

The lateral control of the vehicle is significant for reducing the rollover risk of high-speed cars and improving the stability of the following vehicle. However, the existing car-following (CF) models rarely consider lateral control. Therefore, a CF model with combined longitudinal and lateral control is constructed based on the three degrees of freedom vehicle dynamics model and reinforcement learning method. First, 100 CF segments were selected from the OpenACC database, including 50 straight and 50 curved road trajectories. Afterward, the deep deterministic policy gradient (DDPG) car-following model and multi-agent deep deterministic policy gradient (MADDPG) car-following model were constructed based on the deterministic policy gradient theory. Finally, the models are trained with the extracted trajectory data and verified by comparison with the observed data. The results indicate that the vehicle under the control of the MADDPG model and the vehicle under the control of the DDPG model are both safer and more comfortable than the human-driven vehicle (HDV) on straight roads and curved roads. Under the premise of safety, the vehicle under the control of the MADDPG model has the highest road traffic flow efficiency. The maximum lateral offset of the vehicle under the control of the MADDPG model and the vehicle under the control of the DDPG model in straight road conditions is respectively reduced by 80.86% and 71.92%, compared with the HDV, and the maximum lateral offset in the curved road conditions is lessened by 83.67% and 78.95%. The proposed car following model can provide a reference for developing an adaptive cruise control system considering lateral stability.