A Deep Reinforcement Learning Agent for Traffic Intersection Control Optimization

The efficiency of traffic flows in urban areas is known to crucially depend on signal operation. Here, elements of signal control are discussed, based on the minimization of overall travel times or vehicle queues. Interestingly, we find different operation regimes, some of which involve a "slower-is-faster effect", where a delayed switching reduces the average travel times. These operation regimes characterize different ways of organizing traffic flows in urban road networks. Besides the optimize-one-phase approach, we discuss the procedure and advantages of optimizing multiple phases as well. To improve the service of vehicle platoons and support the self-organization of "green waves", it is proposed to consider the price of stopping newly arriving vehicles.

To reduce the traffic congestion, ensuring intersection control strategy efficiency is a priority in urban transportation system. Reinforcement Learning (RL)-based method provides a new possibility for solving this complex problem. In this paper, a novel region adaptive traffic signal control (RA-TSC) strategy is proposed. As an intelligent agent, the traffic signal light can learn the optimal control policy via deep reinforcement learning algorithm, namely Double Dueling Deep Q Network (3DQN). Different from the previous researches, the proposed method takes traffic characteristics into account, and two approaches are proposed to improve the performance of intersection control: trust region state (TRS) and standardized reward (SR). Besides, prioritized experience replay is adopted to guarantee algorithm stability and optimize data exploitation. Simulation results show that the proposed RA-TSC algorithm can reduce vehicle waiting length by 20% and 15.1% respectively when compared with fixed time and original 3DQN control algorithms.

Traffic signal control is a complex problem and it is difficult to determine an optimal strategy to control multi-directional traffic at multiple intersections. Recent years have witnessed numerous successes of deep learning neural networks in the fields of artificial intelligence. Motivated by the dominant performance of neural networks, this study attempts to develop a novel adaptive signal control approach by fusing deep learning (DL) and reinforcement learning (RL), i.e., deep reinforcement learning (DRL), for arterial signal coordination. DRL can considerably improve the ability to deal with large amounts of data processing, systematic perception and expression, which is key to coordinated control of arterial intersections. The proposed algorithm is implemented by utilizing real-time traffic detection data and aims to optimize the hybrid global and local reward functions. The experimental results obtained by traffic simulation software SUMO demonstrate the advantage of the proposed approach, as well as its efficiency and effectiveness compared with fixed-time and actual signal control methods.

Intelligent Traffic Lights: Green Time Period Negotiaton

One method to reduce vehicle congestion in a road traffic network is to appropriately control traffic signals. One control scheme for traffic signals is a distributed control scheme in which individual traffic signals cooperate locally with other geographically close traffic signals. Deep reinforcement learning has been actively studied to appropriately control traffic signals. In distributed control, it is important to select appropriate cooperative partners. In this study, we propose a method for selecting appropriate cooperative partners using deep reinforcement learning to the distributed traffic signal control.

To decrease the impact of Partially Observable MDP on deep reinforcement learning performance of intersection signal control, A deep reinforcement learning is proposed in this paper with utilizing the real-time GPS data as well as learning the control of the traffic lights in single intersection. We integrate deep reinforcement learning network (DRQN) with recurrent neural network (RNN) and apply deep network, experience pool and greedy strategy in deep reinforcement learning strategy. It solves the problem of overestimation of target Q value and insufficient long-term experience learning in the standard reinforcement learning of traffic signal control. The comparison of performance was made between the proposed method and standard Deep Q-Network (DQN) on the partial observation of traffic situations. The experimental results show that both DQN and DRQN methods can adjust their traffic signal timing control strategies according to the specific traffic conditions as well as calculating a lower average delay time of vehicle than that of fixed-time control. Besides, the simulation effect of DRQN learning method is better than that of DQN learning method in different probe vehicle proportion environment.

Recently, the use of deep reinforcement learning techniques (DRL) has attracted increasing interest due to its ability of dynamical traffic signal control for multiple intersections. Only a few researches use the centralized control with single-agent to intelligently control all the signals, because the major problem, i.e., the curse of dimensionality, has not been successfully solved. We propose a novel centralized control method based on sequence-to-sequence model and attention mechanism to deal with this problem. The idea is similar to the Divide and Conquer paradigm. We mitigate the difficulty of searching in the huge space by dividing the state and action space into sub-spaces through sequence-to-sequence model. In addition, we greatly facilitate the communication and cooperation of traffic signals among intersections by introducing the attention mechanism. The DRL agent is trained by an efficient off-policy learning method - Proximal Policy Optimization. To the best of our knowledge, we are the first to use sequence-to-sequence method to deal with the huge search space problem in the traffic control. The comprehensive experiments demonstrate that our method can efficiently solve the curse of dimensionalitly problem and outperforms the traditional methods and other centralized control methods based on DRL.

With the rapid growth of traffic demand, the urban traffic conditions are becoming more dynamic and uncertain. Traditional traffic control methods cannot cope with such complex traffic conditions, which cause traffic congestion and environmental pollution. In this paper, we propose an intelligent intersection traffic signal control method based on deep Qlearning network algorithm. The simulation experiment is carried out by using SUMO traffic simulation software. The performance of the proposed method is verified based on the traffic data from Pingxiang City, Jiangxi Province and it shows that the proposed method can effectively alleviate the aggregate delay of the early and late traffic peaks compared with fixed time.

Traffic management at intersections is a challenging problem within the transport system. Various traffic signal control strategies have been used to manage the traffic at intersections in real time but are not able to deal with the congestion at road intersections. They are intended to acknowledge the continuous flow of vehicles on heavy traffic routes. However, the process of networking traffic lights of adjacent junctions is a complex matter because of many boundaries. Variable flows entering the junctions are not controlled by traditional systems. Moreover, there is no common intervention in the current traffic signal framework between the adjacent traffic light systems, the difference in the movement of cars over time, injuries, the passage of emergency vehicles, and pedestrian crossings. This paper provides a systematic literature review of the existing methods and algorithms for traffic signal control at intersections. Furthermore, this paper discusses the open challenges with the purpose of synchronizing Traffic Light for Intelligent Vehicles.KeywordsTraffic signal control (TSC)IntersectionsTraffic light synchronization

The integration of autonomous systems in traffic management has become increasingly important as urban populations and vehicle numbers continue to rise, leading to significant congestion. Traditional traffic signal control systems, which rely on fixed timing, are no longer sufficient to handle the dynamic and complex nature of urban traffic. To address these challenges, the proposed explainable Deep Reinforcement Learning (DRL) framework aims to optimize traffic signal control by dynamically adjusting traffic signals based on real-time data. This approach enhances traffic flow efficiency, reduces congestion, and improves overall system performance. The framework leverages Vehicle-to-Everything (V2X) communication, which enables real-time data exchange between vehicles, infrastructure, and other road users, extending the perception range of autonomous vehicles and providing valuable insights for traffic signal optimization. Additionally, the integration of smart infrastructure, such as smart intersections, plays a crucial role in enabling adaptive traffic management and facilitating better coordination across multiple intersections. One of the key advantages of the proposed system is its transparency, achieved through the implementation of explainable AI (XAI) techniques. These mechanisms provide clear insights into the decision-making processes, ensuring that traffic management authorities and system users can understand the rationale behind the system’s decisions. Although challenges such as data accuracy, scalability, and cybersecurity risks remain, the proposed DRL framework shows great promise in revolutionizing traffic management systems. Future research directions include enhancing data collection methods, improving the scalability of the system for larger cities, and further developing explainability features to improve trust and adoption in real-world applications.

Emergency vehicles (EVs) play a pivotal role in promptly responding to time-sensitive situations, such as traffic accidents, medical emergencies, and fires in urban areas. Conventional traffic control methodologies primarily minimize EV travel time by granting them the highest road-use priority, potentially leading to delays for other nearby traffic participants and disrupting the flow of normal traffic. Connected and automated vehicles (CAVs) can help achieve the optimization of the transportation system. Thus, this work proposes an innovative heuristic approach aimed at reducing both EV travel time and the adverse impact on normal traffic by detouring CAVs adjacent to an EV’s path. Specifically, it periodically and dynamically establishes a subnet for each road segment along the EV route. Within the subnetwork, CAVs would be prohibited from utilizing the EV path, thus expediting emergency service delivery. Furthermore, the approach incorporates mechanisms for rerouting CAVs to alleviate congestion. Comprehensive experimental results demonstrate that compared with the heuristic traffic clear-out coordination algorithm (HTCC), the proposed approach reduces the negative impact of EVs on normal traffic by 8.5%, 8.1%, 6.2%, and 7.4%, respectively, across four metrics, including average travel time, average time loss, average waiting time, and average number of stops for EVs and regular vehicles. In addition, sensitivity analyses are conducted to evaluate the impact of penetration rates of CAVs and varying traffic densities. The results indicate that under different penetration rates of CAVs, the proposed approach outperforms the benchmark methods across all four metrics, with advantages becoming more pronounced at higher penetration rates of CAVs. Compared with the noncontrol strategy, our approach and HTCC reduce the response time of EVs by 73.1% and 72.6%, 76.3% and 74.7%, and 77.6% and 74.7% in low, medium, and high traffic densities, respectively. Our approach demonstrates overall superior performance to HTCC, particularly under high traffic densities. This work and its findings enhance the deployment of smart EV management systems.

Efficient traffic signal control (TSC) for multiple intersections is an important way to solve traffic congestion. With the development of deep reinforcement learning (DRL), an increasing number of DRL methods are applied to TSC. However, the prevailing methods in DRL-based TSC does not adequately address the issue of missing data within the agent state space. Furthermore, these methods often insufficiently account for the intricate interactions and relationships among the agents involved. Therefore, a graph deep reinforcement learning traffic signal control for multiple intersections considering missing data is proposed in this paper. Firstly, we propose an agent state space estimation method based on wasserstein generative adversarial network (WGAN). This method is adept at addressing the issue of diverse types of missing data within the state space to ensure its integrity. Secondly, we propose a graph deep reinforcement learning based on two-stage attention network and GraphSage (TAGGRL) to improve TSC efficiency for multiple intersections. A dynamic interaction graph based on two-stage attention network is constructed to facilitate effective interactions among agents. GraphSAGE is constructed for aggregating multi-agent state features. Then, the decision network outputs Q-values based on the extracted features, which guide agents in executing phase-specific actions to enhance the smoothness of multiple intersections. Finally, the experimental results confirm that the agent state space estimation based on WGAN successfully solves the problem of missing data within the state space, thereby enhancing the robustness of TSC for multiple intersections. Furthermore, the TAGGRL model surpasses the baseline model in terms of TSC efficiency for multiple intersections.

To improve the effectiveness of model predictive control (MPC) in dynamic traffic signal control strategies, it has been combined with graph convolutional networks (GCNs) and deep reinforcement learning (DRL) technologies. In this study, a neural-network-based traffic signal control optimization method under the MPC framework is proposed. A dynamic correlation matrix is introduced in the predictive model to adapt to the dynamic changes in correlations between nodes over time. The signal control optimization strategy is solved using DRL, where the agent explores the optimal control strategy based on pre-set constraints in the future road environment. The geometric structure and traffic flow data of a real intersection were selected as the simulation validation environment, and a joint simulation was conducted using Python and SUMO. The experimental results indicate that in low-traffic scenarios, the queue length is reduced by more than 2 vehicles compared to the selected comparison methods; in high-traffic scenarios, the queue length is reduced by an average of 17 vehicles. Under the actual traffic data of the intersection, the average speed is increased by 6.4% compared to the fixed timing method; compared to the inductive signal control method, it increases from 9.76 m/s to 11.69 m/s, an improvement of 19.7%, effectively enhancing the intersection signal control performance.

The urban population growth and increase in a number of vehicles have affected the travel demand on Jordan Streets. The study aims to apply the Transportation Demand Management (TDM) policies to reduce the traffic flow in urban areas and improve the Level of Service (LOS). A group of a combination between TDM and Transportation System Management (TSM) strategies have been conducted, when TDM strategies were not successful to improve LOS. TSM concept refers to any group of actions that increase the capacity of roads network. Synchro 8 software was used to analyse the intersection conditions as important point, connecting two main arterial street in Amman area. A significant reduction in delay and fuel consumption was measured, but there was no real enhancement in LOS. The LOS was improved, when traffic demand was reduced by 20% and capacity was increased with 6 lanes on each approach for the existing conditions. The percentage of saving in fuel consumption and delay was observed to be around 64% and 63%, respectively for the next 5 years.

This paper introduces an integrated solution to the road congestion problem by modelling the road network, using real-time traffic data and drivers' parameters to compute the proposed congestion index for each road segment and generating recommendations to avoid the most congested trajectories. The proposed framework combines the benefits of the multi-agent systems, traffic data from static sensors and big data tools in order to optimise the traffic flow in urban areas. The congestion indexes are used in the road network generation which is represented by a weighted graph. The edges' costs are computed based on the congestion indexes and the edge's properties and vary when new traffic records are retrieved. In this study, the Hadoop framework is used in the data gathering and analysis along with an improved version of Dijkstra for the least congested path finding which allows the proposed framework to reach a higher level of performance.