Explainable Deep-Reinforcement Learning Framework for Autonomous Traffic Signal Control Integrating V2X Data and Smart Infrastructure

Infrastructure‐to‐infrastructure (I2I) communication enables the exchange of traffic data between intersections, which brings a new challenge to urban traffic control. This paper proposes a novel deep reinforcement learning (DRL) framework for urban traffic signal control within the vehicle‐to‐everything (V2X) ecosystem. The framework incorporates a joint‐state representation integrating traffic data from both I2I and vehicle‐to‐infrastructure (V2I) communications, while considering communication range effects. The reward function is designed to optimize both local intersection conditions and global network performance, facilitating adaptive signal coordination. A simulation case study in Huzhou, China, evaluates the proposed model against conventional pretimed, actuated, nonjoint‐state DRL, and joint‐state reinforcement learning (RL) models. Results demonstrate superior performance of the joint‐state DRL model in episode rewards, average speed, and average time loss, particularly during peak traffic periods. To address data limitations, Monte Carlo cross‐validation (MCCV) is employed, further validating the model’s robustness. Results show consistent performance advantages in average speed and time loss across peak and off‐peak periods, with slight variations compared to joint‐state RL models in certain intervals. The impacts of the communication range are also discussed with the proposed model. Pearson correlation analysis reveals a strong positive correlation between the communication range and convergence time across all traffic periods. Meanwhile, correlations between the communication range and reward, average speed, and average time loss vary by traffic period. Findings highlight the transformative potential of integrating DRL with V2X communication technologies for enhancing traffic signal control in complex urban environments. The proposed model offers a flexible, adaptive approach to traffic management, optimizing flow while maintaining safety standards, with implications for future smart city developments.

The acceleration of urbanization has left cities around the world facing significant challenges, including traffic congestion and inefficient management. Intelligent transportation systems provide new technological means to address these issues, especially in traffic signal control, where advanced communication technologies play an important role in facilitating the process. The potential of long range radio (LoRa) technology in intelligent traffic signal control systems is attracting growing interest, largely due to its capacity for long-range transmission and low power consumption. This paper explores the potential benefits of LoRa technology in traffic flow monitoring and signal control by reviewing existing studies, especially in the field of long-distance transmission and low-power communication. The study indicates that LoRa technology can effectively boost the efficacy of traffic management, reduce traffic congestion, and provide technical support for the development of future intelligent transportation systems. Nevertheless, the stability of LoRa technology in high-density urban environments and its integration with other technologies still present challenges. Further research can investigate the anti-jamming ability, communication rate optimization, and its potential for wider application of LoRa technology in Intelligent Transport Systems (ITS) to promote more efficient urban traffic management

Adaptive multi-objective reinforcement learning with hybrid exploration for traffic signal control based on cooperative multi-agent framework

Traffic congestion remains a major challenge in modern urban areas, leading to delays, increased fuel consumption, and environmental pollution. Traditional traffic signal systems rely on fixed schedules and often fail to adapt to real-time traffic dynamics. This paper proposes an adaptive traffic signal control framework using Deep Reinforcement Learning (DRL), specifically the Proximal Policy Optimization (PPO) algorithm, to optimize signal phases based on real-time traffic conditions. The control problem is formulated as a Markov Decision Process, where the agent learns optimal actions from vehicle-level metrics such as waiting time, queue length, speed, and distance to the intersection. The system is trained and evaluated in a SUMO-based simulation environment and benchmarked against a conventional fixed-time controller. Experimental results show that the RL-based controller reduces average waiting time and queue lengths by up to 37.5% and 33.3%, respectively. It also achieves moderate improvements in speed, fuel consumption, and CO<inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</inf>emissions. These results highlight the potential of DRL for intelligent traffic management in smart city infrastructures.

Efficient traffic signal control (TSC) for multiple intersections is an important way to solve traffic congestion. With the development of deep reinforcement learning (DRL), an increasing number of DRL methods are applied to TSC. However, the prevailing methods in DRL-based TSC does not adequately address the issue of missing data within the agent state space. Furthermore, these methods often insufficiently account for the intricate interactions and relationships among the agents involved. Therefore, a graph deep reinforcement learning traffic signal control for multiple intersections considering missing data is proposed in this paper. Firstly, we propose an agent state space estimation method based on wasserstein generative adversarial network (WGAN). This method is adept at addressing the issue of diverse types of missing data within the state space to ensure its integrity. Secondly, we propose a graph deep reinforcement learning based on two-stage attention network and GraphSage (TAGGRL) to improve TSC efficiency for multiple intersections. A dynamic interaction graph based on two-stage attention network is constructed to facilitate effective interactions among agents. GraphSAGE is constructed for aggregating multi-agent state features. Then, the decision network outputs Q-values based on the extracted features, which guide agents in executing phase-specific actions to enhance the smoothness of multiple intersections. Finally, the experimental results confirm that the agent state space estimation based on WGAN successfully solves the problem of missing data within the state space, thereby enhancing the robustness of TSC for multiple intersections. Furthermore, the TAGGRL model surpasses the baseline model in terms of TSC efficiency for multiple intersections.

Adaptive traffic signal control (ATSC) based on deep reinforcement learning (DRL) has shown promising prospects to reduce traffic congestion. Most existing methods keeping traffic signal phases fixed adopt two agent actions to match a four-phase suffering unstable performance and undesirable operation in a four-phase signalized intersection. In this paper, a Double Deep Q-Network (DDQN) with a dual-agent algorithm is proposed to obtain a stable traffic signal control policy. Specifically, two agents are denoted by two different states and shift the control of green lights to make the phase sequence fixed and control process stable. State representations and reward functions are presented by improving the observability and reducing the leaning difficulty of two agents. To enhance the feasibility and reliability of two agents in the traffic control of the four-phase signalized intersection, a network structure incorporating DDQN is proposed to map states to rewards. Experiments under Simulation of Urban Mobility (SUMO) are carried out, and results show that the proposed traffic signal control algorithm is effective in improving traffic capacity.

Traffic signal control is a complex problem and it is difficult to determine an optimal strategy to control multi-directional traffic at multiple intersections. Recent years have witnessed numerous successes of deep learning neural networks in the fields of artificial intelligence. Motivated by the dominant performance of neural networks, this study attempts to develop a novel adaptive signal control approach by fusing deep learning (DL) and reinforcement learning (RL), i.e., deep reinforcement learning (DRL), for arterial signal coordination. DRL can considerably improve the ability to deal with large amounts of data processing, systematic perception and expression, which is key to coordinated control of arterial intersections. The proposed algorithm is implemented by utilizing real-time traffic detection data and aims to optimize the hybrid global and local reward functions. The experimental results obtained by traffic simulation software SUMO demonstrate the advantage of the proposed approach, as well as its efficiency and effectiveness compared with fixed-time and actual signal control methods.

The volume and velocity of traffic data have in-creased dramatically due to the wide adoption of new technologies such as cameras, Internet-of-Thing devices, and vehicular net-works. That data can help us to optimize Traffic Signal Controls (TSCs) by using adaptive algorithms. Some direct applications of these algorithms are reducing the CO2 emission, fuel consumption, and traveling time. Recently, Deep Reinforcement Learning (DRL) methods are the de-facto solution due to its ability to handle big data with high performance. However, most open source codes and frameworks for TSCs using DRL algorithms have limited flexibility. That causes a difficulty to reuse the codebases for new contexts. Therefore, it will be difficult to have a benchmark for TSCs using different optimization algorithms. For this reason, our paper introduces TSLib – a Python framework for fast prototyping TSCs. Specifically, TSLib is designed as a modular system with high reusability so that researchers can quickly implement and evaluate new ideas of TSCs. Moreover, our work offers a comprehensive implementation of some well known TSCs algorithm including both traditional and DRL-based methods as well as their performance measurements.

Dynamic traffic assignment (DTA) applications require traffic signal control data that are typically difficult to obtain and cumbersome to code in the required format. In addition, the evaluation of future scenarios requires future traffic signal settings consistent with the forecast demand. These signal settings are not known a priori and are costly to estimate. Intuitively, future signal timings need to be reasonably optimized so as to represent what the traffic management agency will do. In the literature, integration between traffic control and DTA models has been formulated as a bi-level or single-level optimization problem with system or user optimal constraints. Most existing solution procedures require certain nested structure with an inner-loop algorithm solving the problem of user-equilibrium or system-optimal assignment and the outer-loop algorithm searching for the optimal signal-timing settings. Most of these solution approaches remain only research tools without practical use because of computational intractability. This research proposes an efficient solution algorithm to the problem. An adaptive traffic signal control model is embedded in a simulation-based DTA model. For each inbound approach at an intersection of interest, the adaptive model uses upstream information and a dynamic rolling-horizon approach to project traffic flow conditions for a dynamic but short (projection) period. The adaptive model provides the signal settings during the entire process of traffic flow simulation and for every iteration of the solution algorithm. Thus, during the entire solution process, the experienced travel times and resulting traffic assignment flows are based on the adaptive (demand-responsive) signal settings, allowing the DTA flows and the adaptive signal settings to be generated simultaneously in a single-loop algorithmic structure. Simulation experiments illustrate the capabilities of the proposed approach.

The traffic congestion problem in urban areas is worsening since traditional traffic signal control systems cannot provide efficient traffic control. Therefore, dynamic traffic signal control in intelligent transportation system (ITS) recently has received increasing attention. This study devised an adaptive and cooperative multi-agent fuzzy system for a decentralized traffic signal control. To achieve this goal we have worked on a model, which has three levels of control. Every intersection is controlled by its own traffic situation, its neighboring intersections recommendations and a knowledge base, which provides the traffic pattern of each intersection in any particular day of the week and hour of the day. The proposed architecture comprises a knowledge base, prediction module and a traffic observer that provide data to real traffic data preparation module, then a decision-making layer takes decision to how long should the intersection green light be extended. The proposed architecture can achieve dynamic traffic signal control. We have also developed a NetLogo-based traffic simulator to serve as the agents' world. Our approach is tested with traffic control of a large connected junction and the result obtained is promising; The average delay time can be reduced by 21.76% compared to the conventional fixed sequence traffic signal and 14.77% compared to the vehicle actuated traffic signal control strategy.

To decrease the impact of Partially Observable MDP on deep reinforcement learning performance of intersection signal control, A deep reinforcement learning is proposed in this paper with utilizing the real-time GPS data as well as learning the control of the traffic lights in single intersection. We integrate deep reinforcement learning network (DRQN) with recurrent neural network (RNN) and apply deep network, experience pool and greedy strategy in deep reinforcement learning strategy. It solves the problem of overestimation of target Q value and insufficient long-term experience learning in the standard reinforcement learning of traffic signal control. The comparison of performance was made between the proposed method and standard Deep Q-Network (DQN) on the partial observation of traffic situations. The experimental results show that both DQN and DRQN methods can adjust their traffic signal timing control strategies according to the specific traffic conditions as well as calculating a lower average delay time of vehicle than that of fixed-time control. Besides, the simulation effect of DRQN learning method is better than that of DQN learning method in different probe vehicle proportion environment.

Cooperative traffic signal control using Multi-step return and Off-policy Asynchronous Advantage Actor-Critic Graph algorithm

Traffic signal control plays a pivotal role in the management of urban traffic flow. With the rapid advancement of reinforcement learning, the development of signal control methods has seen a significant boost. However, a major challenge in implementing these methods is ensuring that signal lights do not change abruptly, as this can lead to traffic accidents. To mitigate this risk, a time-delay is introduced in the implementation of control actions, but usually has a negative impact on the overall efficacy of the control policy. To address this challenge, this paper presents a novel Traffic Signal Control Framework (PRLight), which leverages an On-policy Traffic Control Model (OTCM) and an Online Traffic Prediction Model (OTPM) to achieve efficient and real-time control of traffic signals. The framework collects multi-source traffic information from a local-view graph in real-time and employs a novel fast attention mechanism to extract relevant traffic features. To be specific, OTCM utilizes the predicted traffic state as input, eliminating the need for communication with other agents and maximizing computational efficiency while ensuring that the most relevant information is used for signal control. The proposed framework was evaluated on both simulated and real-world road networks and compared to various state-of-the-art methods, demonstrating its effectiveness in preventing traffic congestion and accidents.

Traffic related problems include not just traffic congestion due to increase in vehicular density, but also difficulty for passage of emergency vehicles, violation of rules such as red signal jumps, vehicle breakdowns and accidents causing blockage of roads and loss of lives. Previously proposed systems for smart traffic management focus on vehicular density based traffic signal control and provision for emergency vehicle passage. The capability of such systems are limited to one or two focus areas and most systems are not cost effective for real time implementation. In order to build a smart city, a comprehensive system for traffic management needs to be built that addresses all traffic related issues, not just traffic congestion. In this paper, a Comprehensive Traffic Management System (CTMS) using Radio Frequency Identification (RFID) and analytics for real time implementation has been proposed. The system is both cost effective and easy to implement. The outcomes are dynamic traffic signal timers that operate based on vehicular density, deviation of vehicles at previous junctions in case of blockage, traffic signal control for passage of emergency vehicles, detecting red signal violations, detecting road accidents, vehicle breakdowns for providing immediate assistance and tracking of vehicles.

Traffic congestion has substantially induced significant mobility and energy inefficiency. Many research challenges are identified in traffic signal control and management associated with artificial intelligence (AI)‐based models. For example, developing AI‐driven dynamic traffic system models that accurately capture high‐resolution traffic attributes and formulate robust control algorithms for traffic signal optimization is difficult. Additionally, uncertainties in traffic system modeling and control processes can further complicate traffic signal system controllability. To partially address these challenges, this study presents a novel, hybrid neural network model enhanced with a probability density function kernel shaping technique to formulate traffic system dynamics better and improve comprehensive traffic network modeling and control. The numerical experimental tests were conducted, and the results demonstrate that the proposed control approach outperforms the baseline control strategies and reduces overall average delays by 11.64% on average. By leveraging the capabilities of this innovative model, this study aims to address major challenges related to traffic congestion and energy inefficiency toward more effective and adaptable AI‐based traffic control systems.