Deep Reinforcement Learning-Based Traffic Signal Control for Urban Congestion

Traffic congestion remains a major challenge in modern urban areas, leading to delays, increased fuel consumption, and environmental pollution. Traditional traffic signal systems rely on fixed schedules and often fail to adapt to real-time traffic dynamics. This paper proposes an adaptive traffic signal control framework using Deep Reinforcement Learning (DRL), specifically the Proximal Policy Optimization (PPO) algorithm, to optimize signal phases based on real-time traffic conditions. The control problem is formulated as a Markov Decision Process, where the agent learns optimal actions from vehicle-level metrics such as waiting time, queue length, speed, and distance to the intersection. The system is trained and evaluated in a SUMO-based simulation environment and benchmarked against a conventional fixed-time controller. Experimental results show that the RL-based controller reduces average waiting time and queue lengths by up to 37.5% and 33.3%, respectively. It also achieves moderate improvements in speed, fuel consumption, and CO<inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</inf>emissions. These results highlight the potential of DRL for intelligent traffic management in smart city infrastructures.

To decrease the impact of Partially Observable MDP on deep reinforcement learning performance of intersection signal control, A deep reinforcement learning is proposed in this paper with utilizing the real-time GPS data as well as learning the control of the traffic lights in single intersection. We integrate deep reinforcement learning network (DRQN) with recurrent neural network (RNN) and apply deep network, experience pool and greedy strategy in deep reinforcement learning strategy. It solves the problem of overestimation of target Q value and insufficient long-term experience learning in the standard reinforcement learning of traffic signal control. The comparison of performance was made between the proposed method and standard Deep Q-Network (DQN) on the partial observation of traffic situations. The experimental results show that both DQN and DRQN methods can adjust their traffic signal timing control strategies according to the specific traffic conditions as well as calculating a lower average delay time of vehicle than that of fixed-time control. Besides, the simulation effect of DRQN learning method is better than that of DQN learning method in different probe vehicle proportion environment.

The intersection management system can increase traffic capacity, vehicle safety, and the smoothness of all vehicle movement. Platoons of connected vehicles (CVs) use communication technologies to share information with each other and with infrastructures. In this paper, we proposed a deep reinforcement learning (DRL) model that applies to vehicle platooning at an isolated signalized intersection with partial detection. Moreover, we identified hyperparameters and tested the system with different numbers of vehicles (1, 2, and 3) in the platoon. To compare the effectiveness of the proposed model, we implemented two benchmark options, actuated traffic signal control (ATSC) and max pressure (MP). The experimental results demonstrated that the DRL model has many outstanding advantages compared to other models. Through the learning process, the average waiting time of vehicles in the DRL method was improved by 20% and 28% compared with the ATSC and MP options. The results also suggested that the DRL model is effective when the CV penetration rate is over 20%.

Recent studies show that Deep Reinforcement Learning (DRL) models are vulnerable to adversarial attacks, which attack DRL models by adding small perturbations to the observations. However, some attacks assume full availability of the victim model, and some require a huge amount of computation, making them less feasible for real world applications. In this work, we make further explorations of the vulnerabilities of DRL by studying other aspects of attacks on DRL using realistic and efficient attacks. First, we adapt and propose efficient black-box attacks when we do not have access to DRL model parameters. Second, to address the high computational demands of existing attacks, we introduce efficient online sequential attacks that exploit temporal consistency across consecutive steps. Third, we explore the possibility of an attacker perturbing other aspects in the DRL setting, such as the environment dynamics. Finally, to account for imperfections in how an attacker would inject perturbations in the physical world, we devise a method for generating a robust physical perturbations to be printed. The attack is evaluated on a real-world robot under various conditions. We conduct extensive experiments both in simulation such as Atari games, robotics and autonomous driving, and on real-world robotics, to compare the effectiveness of the proposed attacks with baseline approaches. To the best of our knowledge, we are the first to apply adversarial attacks on DRL systems to physical robots.

Infrastructure‐to‐infrastructure (I2I) communication enables the exchange of traffic data between intersections, which brings a new challenge to urban traffic control. This paper proposes a novel deep reinforcement learning (DRL) framework for urban traffic signal control within the vehicle‐to‐everything (V2X) ecosystem. The framework incorporates a joint‐state representation integrating traffic data from both I2I and vehicle‐to‐infrastructure (V2I) communications, while considering communication range effects. The reward function is designed to optimize both local intersection conditions and global network performance, facilitating adaptive signal coordination. A simulation case study in Huzhou, China, evaluates the proposed model against conventional pretimed, actuated, nonjoint‐state DRL, and joint‐state reinforcement learning (RL) models. Results demonstrate superior performance of the joint‐state DRL model in episode rewards, average speed, and average time loss, particularly during peak traffic periods. To address data limitations, Monte Carlo cross‐validation (MCCV) is employed, further validating the model’s robustness. Results show consistent performance advantages in average speed and time loss across peak and off‐peak periods, with slight variations compared to joint‐state RL models in certain intervals. The impacts of the communication range are also discussed with the proposed model. Pearson correlation analysis reveals a strong positive correlation between the communication range and convergence time across all traffic periods. Meanwhile, correlations between the communication range and reward, average speed, and average time loss vary by traffic period. Findings highlight the transformative potential of integrating DRL with V2X communication technologies for enhancing traffic signal control in complex urban environments. The proposed model offers a flexible, adaptive approach to traffic management, optimizing flow while maintaining safety standards, with implications for future smart city developments.

This paper investigates the use of Deep Reinforcement Learning (DRL) to control a profit-seeking storage device trading in the European Continuous Intra-day Electricity Market (CIM). The main objective is to study whether model-free DRL can profitably trade on the CIM. Two DRL agents are compared: Twin Delayed Deep Deterministic Policy Gradients (TD3), and TD3 with behavior cloning. The agents are trained and evaluated in a simulated CIM environment, which uses historical market data to simulate other market participants. A Rolling Intrinsic (RI) algorithm is used as a benchmark. Results indicate that the agents are profitable and occasionally outperform RI, in one instance obtaining 162.03% of RI profit. However, none of the agents consistently outperforms the baseline. These results suggest that DRL has the potential to increase profitability considerably compared to RI, but that the observation provided to the agent is not descriptive enough of the CIM to learn a robust policy. Future research could use additional features in the observation, or model-based DRL to improve performance.

Traffic congestion alleviation around intersections has been a growing challenge, and a competent traffic signal control scheme plays a pivotal role in intelligent transportation systems. Recent studies using deep reinforcement learning techniques have shown promising results for traffic signal control, but they only focus on extracting features from traffic conditions of isolated or adjacent intersections. In this work, we employed navigation information for traffic signal control, greatly enriching the features for traffic signal control with deep reinforcement learning. In addition, we are the first to propose a novel scheme DeepNavi to exploit the temporal-spatial relations from numerous navigation routes and extract dynamic real-time and future traffic features. We tested our scheme on a challenging real-world traffic dataset with 16 intersections in a residential district of Hangzhou, China. Extensive experiments were conducted and the results demonstrated that our DeepNavi scheme achieves superior performance over five popular and state-of-the-art baseline methods on different metrics, including queue length, speed, travel time and accumulative waiting time. In addition, with our method, vehicles suffer the least red lights and enjoy the most green waves, which further validates that our scheme greatly relieves the congestion and provides excellent experience for drivers. Simulations with different penetration levels of navigation routes showed that even with only part of navigation routes available in the traffic network, our scheme can obtain superior performance, further demonstrating the effectiveness and feasibility of DeepNavi.

Uneven food distribution across various regions in Indonesia often results in supply-demand imbalances, leading to price surges, stock shortages, and overall market instability. This challenge is compounded by the limitations of conventional distribution systems, which are ill-equipped to respond to rapidly changing market dynamics. In response, this study introduces a novel, AI-driven approach by implementing Deep Reinforcement Learning (DRL) to optimize food distribution policies using real-world data. Specifically, we perform a comparative evaluation of four emerging DRL models—Double Deep Q-Network (Double DQN), Dueling DQN, Proximal Policy Optimization (PPO), and Advantage Actor-Critic (A2C)—to determine their effectiveness in learning adaptive distribution strategies from national food logistics data provided by Indonesia’s Central Bureau of Statistics (BPS). Each model was trained within a custom simulation environment based on the Markov Decision Process (MDP) framework and evaluated using five core performance metrics: cumulative reward, average reward, success rate, sample efficiency, and best reward. The results reveal that A2C consistently outperformed the other models, delivering the highest average reward and most stable training performance, while PPO demonstrated strong efficiency and success rate. These findings underscore the potential of policy-gradient methods—particularly A2C—as robust and intelligent solutions for dynamic food logistics management. This research offers one of the first comparative benchmarks of DRL methods in the food distribution domain and highlights their applicability for future integration into national AI-powered logistics systems.

We have adopted a new tool to invert magnetotelluric data for the 1D model based on deep Q-networks (DQN), which works as a stochastic optimization method. By transforming the inversion problem into a Markov decision process, the tool learns by trial and error to find the optimal path for updating the model to fit the observed data. The DQN method converges to the target through different paths (e.g., Bayesian or other stochastic methods) and can partially provide the probability distribution of the inversion results, which can be used for uncertainty estimation. The DQN search space gradually decreases as the learning experience progresses, accelerating the single inversion and approximating the optimal result. To check the effectiveness of the DQN inversion, the five- and eight-layer models were separately designed to test the robustness of the DQN for the initial model and the noise level. A further comparison with Occam’s method and the Bayesian method indicated that our DQN obtained more robust inversion results for data contaminated by different noise levels. The inversion results with the survey data from Zhagaitunuoergong area, Inner Mongolia, China, well reveals the shape of the interface basement.

This study proposes using Deep Reinforcement Learning (DRL) for stock trading decisions and prediction. DRL is a machine learning technique that enables agents to learn optimal strategies by interacting with their environment. The proposed model surpasses traditional models and can make informed trading decisions in real-time. The study highlights the feasibility of applying DRL in financial markets and its advantages in strategic decision- making. The model's ability to learn from market dynamics makes it a promising approach for stock market forecasting. Overall, this paper provides valuable insights into the use of DRL for stock trading decisions and prediction, establishing a strong case for its adoption in financial markets. Keywords: reinforcement learning, stock market, deep reinforcement learning.

Intelligent Transportation Systems (ITS) combine emerging communication, computer, and system technologies to deliver intelligent road traffic services and optimize decision making within the transportation infrastructure. The advancement of connected vehicles, which generate dynamic data through wireless communications, enables ITS to improve their efficiency, especially in Traffic Signal Control (TSC), which is the backbone of traffic flow scheduling. However, wireless communications channels are vulnerable to various types of cyberattacks and can pose serious threats to dynamic TSC systems. Attackers may attempt to manipulate normal traffic flows and cause severe traffic congestion. Deep Reinforcement Learning (DRL) is a powerful technique that has been used to improve TSC systems performance in real-time environments. However, it can be used by attackers to exploit the dynamics of the ITS and learn the optimal attack policy under the lack of deterministic information about system behavior. In this work, to highlight and exploit existing vulnerabilities in TSC systems, we leverage DRL to create an intelligent Sybil attack on a traffic intersection, wherein connected vehicles with fake identities are optimally placed to alter traffic signal timings by corrupting traffic data. The results show that this attack leads to substantial increase in the vehicles’ travel time and yields disastrous traffic congestion, especially if carried out for a prolonged period of time, which will give rise to serious problems such as higher fuel consumption and air pollution in heavily dense cities. In the presence of such intelligent attacks, the design assumptions of existing TSC systems become highly questionable.

Urban traffic congestion is a major challenge in cities like Dar es Salaam. Traditional fixed-time traffic light systems fail to respond to real-time traffic changes, causing long waits, fuel wastage, and pollution. This study proposes a hybrid traffic light control model for Mwenge intersection using Long Short-Term Memory (LSTM) and Deep Q-Network (DQN) models. The LSTM predicts short-term traffic volumes, which regulate vehicle flow in the SIMULATION OF URBAN MOBILITY (SUMO) simulation environment. At the same time, the Deep Q-Network (DQN) adaptively adjusts signal phases based on real-time traffic states. Traffic data collected over 240 hours was used for model training and simulation. The LSTM achieved an average Test Mean Absolute Error (MAE) of 3 vehicles/min across all directions, indicating accurate traffic prediction. The Deep Q-Network (DQN) improved intersection performance with Average Wait Time (21.6s), Queue Length (9.2 vehicles), and Throughput (888 vehicles), outperforming the traditional model (38.4s, 17.1 vehicles, and 712 vehicles, respectively). Overall, the hybrid LSTM-DQN model reduced wait times by 43.8%, queue lengths by 46.2%, and increased throughput by 25%. The proposed approach offers an adaptive, cost-effective solution for optimising traffic signals in resource-constrained cities like Dar es Salaam, supporting smarter, more sustainable urban mobility. Policymakers are encouraged to adopt this model for busy intersections in Tanzania.

Design automation is undergoing a new generation of changes caused by artificial intelligence technologies represented by deep learning and reinforcement learning. Notably, the advantages of deep reinforcement learning in addressing solution optimisation and decision-making tasks with cognitive automation functionality have garnered attention in design. In the context of surrogate model-driven engineering design optimisation, this paper addresses current research challenges such as reliance on domain knowledge for local development, shortcomings in the self-learning and adaptive capabilities of optimisation algorithms for global exploration, etc. Centred around the deep reinforcement learning model, Deep Q-learning, and complemented by self-organising maps and neural network technologies, we propose a methodology considering multi-fidelity simulation data for design space exploration. This approach effectively reduces sampling costs and enables the optimisation model to learn the optimal direction for high-precision predictions and achieve rapid, accurate optimisation. Finally, the effectiveness of the proposed method is comprehensively validated through four typical optimisation scenarios and a case study involving the optimisation of a wheeled robot's suspension swing arm structure. This work will be a crucial reference for applying deep reinforcement learning in simulation-driven engineering design optimisation.

Deep Reinforcement Learning (RL) suffer from catastrophic forgetting due to being agnostic to the timescale of changes in the distribution of experiences. Although, RL algorithms are guaranteed to converge to optimal policies in Markov decision processes, this only holds in the presence of static environments. However, this assumption is very restrictive. In many real world problems like ridesharing, traffic control, etc., we are dealing with highly dynamic environments, where RL methods yield only sub-optimal decisions. In this paper, we introduce an adaptive model-free deep reinforcement approach that can recognize diurnal patterns in the ridesharing environment. To achieve this, we (1) adopt a change point detection algorithm to detect the changes in the distribution of experiences, then (2) we develop a Deep Q Network (DQN) agent that is capable of recognizing diurnal patterns and making informed dispatching decisions according to the changes in the underlying environment. Based on the demand, our DQN approach re-balances idle vehicles by dispatching them to the areas of anticipated high demand using Deep Reinforcement Learning. This approach can be adopted in various domains through tuning the RL agent's objective function, where it will still capture the changes in the corresponding underlying environment. Our framework is validated using the New York City Taxi public dataset. Experimental results show the effectiveness of our approach in real-time and large scale settings.

This paper investigates stock portfolios by application of Deep Reinforcement Learning (DRL) Models to achieve an optimal tactical asset allocation. The research problem is described as an optimization scenario that seeks to maximize the portfolio risk adjusted returns for a given portfolio asset allocation. The problem is set up with an initial capital investment which is invested in a set of assets. The initial strategic allocation is determined, which in our case is the equal weight allocation, and all of the capital is invested in the set of assets. At each point in time, the assets are reallocated according to the allocation which will increase the portfolio value. Two DRL models are implemented. The performance of the DRL models is compared with the uniform weights portfolio. The results show that, generally, two DRL models have higher cumulative returns.