Traffic Signal Control with Deep Reinforcement Learning

Automated market makers (AMMs) have revolutionized decentralized finance (DeFi), enabling trustless asset exchange through algorithmic liquidity pools. One notable AMM is Uniswap, which improved capital efficiency by introducing concentrated liquidity, allowing Liquidity Providers (LPs) to allocate capital within specific price ranges. However, this flexibility requires active management to maximize fee generation while effectively addressing impermanent loss (IL) and gascosts. We formulate concentrated liquidity management (CLM) as a stochastic control problem and propose deep reinforcement learning (RL) strategies, specifically Proximal Policy Optimization (PPO) and Deep Q-Network (DQN), to optimize liquidity positioning. Unlike heuristics, RL agents adapt dynamically to market conditions. We adopt a two-stage synthetic-to-historical evaluation strategy, involving RL model training on synthetic data and evaluation on historical data. Results show that training on synthetic data and testing historic data allows RL-based strategies to perform well across regimes, capturing more fees while keeping net performance ahead of a passive buy-and-hold strategy. These findings highlight the robustness and practical viability of such strategies for CLM.

Learning ability is the basic characteristic of human intelligence. The July 1, 2005 issue of Science published a list of 125 important questions in science. Among them, the question 94 “What are the limits of learning by machines?”. The annotation “Computers can already beat the world’s best chess players, and they have a wealth of information on the Web to draw on. But abstract reasoning is still beyond any machine”. In recent artificial intelligence has made great progresses. In 1997, the rise of the man-machine war, IBM Supercomputer Deep Blue defeated the chess master Garry Kasparov. On February 14, 2011, IBM’s Watson supercomputer won a practice round against Jeopardy champions Ken Jennings and Brad Rutter. In March 2016, Google DeepMind’s AlphaGo sealed a 4-1 victory over a South Korean Go grandmaster Lee Se-dol. This paper focuses on the machine learning methods of AlphaGo, including reinforcement learning, deep learning, deep reinforcement learning, analysis of the existing problems and the latest research progress. Deep reinforcement learning is the combination of deep learning and reinforcement learning, which can realize the learning algorithm from the perception to action. Simply said, this is the same as human behavior, input sensing information such as vision, and then, direct output action through the deep neural network. Deep reinforcement learning has the potential to learn a variety of skills for the robot to achieve full autonomy. Even though reinforcement learning is practiced successfully, but feature states need to manually set, for complex scene is a difficult thing, especially easy to cause the dimension disaster, and expression is not good. In 2010, Sascha Lange and Martin Riedmiller proposed deep auto-encoder neural networks in reinforcement learning to extract feature, which is used to control the visual correlation. In 2013, DeepMind proposed deep Q-network (DQN) in NIPS 2013, using convolution neural network to extract features, and then applied in reinforcement learning. They continue to improve and published an improved version of DQN on Nature in 2015, which has aroused widespread concern. In order to break through the limits of learning by machines, cognitive machine learning is proposed, which is the combination of machine learning and brain cognition, so that the machine intelligence is constantly evolving, and gradually reaches the human level of artificial intelligence. A cognitive model entitled Consciousness And Memory (CAM) is proposed by author, which consists of memory, consciousness, high-level cognitive functions, perception and motor. High-level cognitive functions of the brain include learning, language, thinking, decision making, emotion, and so on. Learning is a course to accept the stimulus through the nervous system and obtain new behavior, habits and accumulation experience. According to the current research progress of brain science and cognitive science, cognitive machine learning may be interested in learning emergence, procedural memory knowledge learning, learning evolution and so on. For intelligence, so-called evolution is refers to the learning of learning and the structure also follows the change. It is important to record the learning result by structure changing and improve the learning method.

Traffic signal control is a complex problem and it is difficult to determine an optimal strategy to control multi-directional traffic at multiple intersections. Recent years have witnessed numerous successes of deep learning neural networks in the fields of artificial intelligence. Motivated by the dominant performance of neural networks, this study attempts to develop a novel adaptive signal control approach by fusing deep learning (DL) and reinforcement learning (RL), i.e., deep reinforcement learning (DRL), for arterial signal coordination. DRL can considerably improve the ability to deal with large amounts of data processing, systematic perception and expression, which is key to coordinated control of arterial intersections. The proposed algorithm is implemented by utilizing real-time traffic detection data and aims to optimize the hybrid global and local reward functions. The experimental results obtained by traffic simulation software SUMO demonstrate the advantage of the proposed approach, as well as its efficiency and effectiveness compared with fixed-time and actual signal control methods.

Reinforcement learning is a mathematical framework for agents to interact intelligently with their environment. Unlike supervised learning, where a system learns with the help of labeled data, reinforcement learning agents learn how to act by trial and error only receiving a reward signal from their environments. A field where reinforcement learning has been prominently successful is robotics [3]. However, real-world control problems are also particularly challenging because of the noise and high- dimensionality of input data (e.g., visual input). In recent years, in the field of supervised learning, deep neural networks have been successfully used to extract meaning from this kind of data. Building on these advances, deep reinforcement learning was used to solve complex problems like Atari games and Go. Mnih et al. [1] built a system with fixed hyper parameters able to learn to play 49 different Atari games only from raw pixel inputs. However, in order to apply the same methods to real-world control problems, deep reinforcement learning has to be able to deal with continuous action spaces. Discretizing continuous action spaces would scale poorly, since the number of discrete actions grows exponentially with the dimensionality of the action. Furthermore, having a parametrized policy can be advantageous because it can generalize in the action space. Therefore with this thesis we study state-of-the-art deep reinforcement learning algorithm, Deep Deterministic Policy Gradients. We provide a theoretical comparison to other popular methods, an evaluation of its performance, identify its limitations and investigate future directions of research. The remainder of the thesis is organized as follows. We start by introducing the field of interest, machine learning, focusing our attention of deep learning and reinforcement learning. We continue by describing in details the two main algorithms, core of this study, namely Deep Q-Network (DQN) and Deep Deterministic Policy Gradients (DDPG). We then provide implementatory details of DDPG and our test environment, followed by a description of benchmark test cases. Finally, we discuss the results of our evaluation, identifying limitations of the current approach and proposing future avenues of research.

Comparison of end-to-end and hybrid deep reinforcement learning strategies for controlling cable-driven parallel robots

To solve problems involving sequential decision-making, Deep Reinforcement Learning (DRL) combines the advantages of Reinforcement Learning (RL) and Deep Learning (DL).These fusions are skilled at training agents for video games because they use neural networks to approximate nonlinear functions.The Deep Q Network (DQN), a key algorithm in this field, tends to overestimate Q-values despite its effectiveness.The Double Deep Q Network (Double DQN) was presented to address this.The Kera's framework in PyCharm is used in this study to examine the practical application and comparative analysis of DQN and Double DQN in the Flappy Bird game.To speed up its training, improvements were also made to the DQN model.The enhanced DQN performed better than the traditional DQN but less well than the Double DQN, according to the results.This study delves into the practical application and comparative analysis of DQN and Double DQN in the Flappy Bird game, utilizing the Kera's framework in PyCharm.Additionally, enhancements were made to the DQN model to expedite its training.Results affirmed that the enhanced DQN outperformed the conventional DQN but lagged the Double DQN.The training loss trajectory further substantiated Double DQN's superiority in mitigating overestimation issues.

Adaptive traffic signal control (ATSC) based on deep reinforcement learning (DRL) has shown promising prospects to reduce traffic congestion. Most existing methods keeping traffic signal phases fixed adopt two agent actions to match a four-phase suffering unstable performance and undesirable operation in a four-phase signalized intersection. In this paper, a Double Deep Q-Network (DDQN) with a dual-agent algorithm is proposed to obtain a stable traffic signal control policy. Specifically, two agents are denoted by two different states and shift the control of green lights to make the phase sequence fixed and control process stable. State representations and reward functions are presented by improving the observability and reducing the leaning difficulty of two agents. To enhance the feasibility and reliability of two agents in the traffic control of the four-phase signalized intersection, a network structure incorporating DDQN is proposed to map states to rewards. Experiments under Simulation of Urban Mobility (SUMO) are carried out, and results show that the proposed traffic signal control algorithm is effective in improving traffic capacity.

In this paper, we apply the Proximal Policy Optimization (PPO) algorithm in intelligent traffic signal control at a single intersection with eight lanes and four signal phases. The optimization goal is to minimize the average waiting time of vehicles so as to improve the traffic efficiency of the intersection. Extensive experiments are conducted in Simulation of Urban MObility (SUMO) to evaluate the performance of the proposed algorithm, and compare it with other classic algorithms including Deep Q-network (DQN), Advantage Actor Critic (A2C) and Fixed Time. Simulation results show that the proposed PPO algorithm outperforms the others under various traffic scenarios to different extent. The performance gain is significant under unbalanced traffic where one direction is saturated while the other is not, and becomes marginal when all the directions are saturated or unsaturated. PPO also demonstrates good portability and robustness over time-varying traffic patterns, while implies it could be a preferable option for implementation in real world intelligent traffic signal control systems.

Deep reinforcement learning (DRL) is one of the emerging areas of machine learning which focuses on maximized rewards. DRL is a type of machine learning that combines reinforcement learning and deep learning. It uses a series of algorithms to enable an agent to learn how to make decisions in a complex environment. DRL is a subset of artificial intelligence that focuses on making decisions based on the environment and the rewards associated with each action.The goal of DRL is to maximize the long-term reward of an agent. In order to do this, the agent must use a combination of deep learning, reinforcement learning and other AI techniques to learn which actions will lead to the highest reward. DRL is used to solve a variety of problems, from playing video games to controlling robots. It is also used in autonomous driving and robotics, as well as for financial trading. DRL is a powerful tool for solving complex problems and has been used in a variety of research projects. DRL has the potential to revolutionize the way we interact with machines and the environment.

Due to limited infrastructure capacity and the increasing number of vehicles, traffic congestion remains a major problem in urban management. The scalability and adaptability issues with traditional traffic signal control (TSC) techniques call for innovative solutions. In order to optimize intelligent transportation systems (ITS), this paper investigates the use of deep reinforcement learning (DRL) models, specifically deep Q-networks (DQN) and their variations. In this study, we employed a Markov Decision Process (MDP) framework to assess how various RL models improve traffic flow efficiency and alleviate congestion. This allows adaptive real-time decision-making to improve traffic flow. We assess how well two RL agents—"Choose Next Phase" and "Next or Not"—manage traffic at a four-way intersection. The simulation results show that RL-based methods perform better than conventional methods, reducing emergency vehicle delays, average waiting time, and queue length. By obtaining higher and more consistent rewards over training iterations, the "Choose Next Phase" agent showed exceptional adaptability. Additionally, under abnormal traffic conditions, we compare RL-based TSC with traditional heuristic-based techniques, emphasizing DRL’s superior handling of complex scenarios. To further increase scalability and decision-making reliability in practical applications, future research is highlighted with an emphasis on improving RL models using multi-agent learning, transformer-based architectures, and decentralized computing.

With the rapid development of AI technology, personalized learning path planning is becoming an important research field. Dual system teaching models have great influence on the learning effect of students. Therefore, how to optimize the learning process of students through the personalized learning path is a great challenge. A Deep Reinforcement Learning (DRL) algorithm for personalized learning path planning is proposed. Based on deep Q-networks and reinforcement learning feedback mechanisms, the algorithm achieves adaptive optimization. The experimental results show that the algorithm based on the DRL is superior to the traditional method in learning performance, learning efficiency and satisfaction. Compared with traditional path planning algorithms based on rules and collaborative filtering, the proposed algorithm improves learning performance by 20%. Moreover, it has obvious advantages in learning motivation and efficiency. This paper provides a new way of thinking to intelligent education system, especially dual-system teaching model, and it is of great practical significance.

Design automation is undergoing a new generation of changes caused by artificial intelligence technologies represented by deep learning and reinforcement learning. Notably, the advantages of deep reinforcement learning in addressing solution optimisation and decision-making tasks with cognitive automation functionality have garnered attention in design. In the context of surrogate model-driven engineering design optimisation, this paper addresses current research challenges such as reliance on domain knowledge for local development, shortcomings in the self-learning and adaptive capabilities of optimisation algorithms for global exploration, etc. Centred around the deep reinforcement learning model, Deep Q-learning, and complemented by self-organising maps and neural network technologies, we propose a methodology considering multi-fidelity simulation data for design space exploration. This approach effectively reduces sampling costs and enables the optimisation model to learn the optimal direction for high-precision predictions and achieve rapid, accurate optimisation. Finally, the effectiveness of the proposed method is comprehensively validated through four typical optimisation scenarios and a case study involving the optimisation of a wheeled robot's suspension swing arm structure. This work will be a crucial reference for applying deep reinforcement learning in simulation-driven engineering design optimisation.

Path planning refers to that the mobile robot can obtain the surrounding environment information and its own state information through the sensor carried by itself, which can avoid obstacles and move towards the target point. Deep reinforcement learning consists of two parts: reinforcement learning and deep learning, mainly used to deal with perception and decision-making problems, has become an important research branch in the field of artificial intelligence. This paper first introduces the basic knowledge of deep learning and reinforcement learning. Then, the research status of deep reinforcement learning algorithm based on value function and strategy gradient in path planning is described, and the application research of deep reinforcement learning in computer game, video game and autonomous navigation is described. Finally, I made a brief summary and outlook on the algorithms and applications of deep reinforcement learning.

Infrastructure‐to‐infrastructure (I2I) communication enables the exchange of traffic data between intersections, which brings a new challenge to urban traffic control. This paper proposes a novel deep reinforcement learning (DRL) framework for urban traffic signal control within the vehicle‐to‐everything (V2X) ecosystem. The framework incorporates a joint‐state representation integrating traffic data from both I2I and vehicle‐to‐infrastructure (V2I) communications, while considering communication range effects. The reward function is designed to optimize both local intersection conditions and global network performance, facilitating adaptive signal coordination. A simulation case study in Huzhou, China, evaluates the proposed model against conventional pretimed, actuated, nonjoint‐state DRL, and joint‐state reinforcement learning (RL) models. Results demonstrate superior performance of the joint‐state DRL model in episode rewards, average speed, and average time loss, particularly during peak traffic periods. To address data limitations, Monte Carlo cross‐validation (MCCV) is employed, further validating the model’s robustness. Results show consistent performance advantages in average speed and time loss across peak and off‐peak periods, with slight variations compared to joint‐state RL models in certain intervals. The impacts of the communication range are also discussed with the proposed model. Pearson correlation analysis reveals a strong positive correlation between the communication range and convergence time across all traffic periods. Meanwhile, correlations between the communication range and reward, average speed, and average time loss vary by traffic period. Findings highlight the transformative potential of integrating DRL with V2X communication technologies for enhancing traffic signal control in complex urban environments. The proposed model offers a flexible, adaptive approach to traffic management, optimizing flow while maintaining safety standards, with implications for future smart city developments.

Deep Reinforcement Learning (RL) suffer from catastrophic forgetting due to being agnostic to the timescale of changes in the distribution of experiences. Although, RL algorithms are guaranteed to converge to optimal policies in Markov decision processes, this only holds in the presence of static environments. However, this assumption is very restrictive. In many real world problems like ridesharing, traffic control, etc., we are dealing with highly dynamic environments, where RL methods yield only sub-optimal decisions. In this paper, we introduce an adaptive model-free deep reinforcement approach that can recognize diurnal patterns in the ridesharing environment. To achieve this, we (1) adopt a change point detection algorithm to detect the changes in the distribution of experiences, then (2) we develop a Deep Q Network (DQN) agent that is capable of recognizing diurnal patterns and making informed dispatching decisions according to the changes in the underlying environment. Based on the demand, our DQN approach re-balances idle vehicles by dispatching them to the areas of anticipated high demand using Deep Reinforcement Learning. This approach can be adopted in various domains through tuning the RL agent's objective function, where it will still capture the changes in the corresponding underlying environment. Our framework is validated using the New York City Taxi public dataset. Experimental results show the effectiveness of our approach in real-time and large scale settings.