Continuous residual reinforcement learning for traffic signal control optimization

In continual learning, an agent is exposed to a changing environment, requiring it to adapt during execution time. While traditional reinforcement learning (RL) methods have shown impressive results in various domains, there has been less progress in addressing the challenge of continual learning. Current RL approaches do not allow the agent to adapt during execution but only during a dedicated training phase. Here we study the problem of continual learning in a 2D bipedal walker domain, in which the legs of the walker grow over its lifetime, requiring the agent to adapt. The introduced approach combines neuroevolution, to determine the starting weights of a deep neural network, and a version of deep reinforcement learning that is continually running during execution time. The proof-of-concept results show that the combined approach gives a better generalisation performance when compared to evolution or reinforcement learning alone. The hybridization of reinforcement learning and evolution opens up exciting new research directions for continually learning agents that can benefit from suitable priors determined by an evolutionary process.

The continual learning and development are significant for robots to learn multiple tasks sequentially. The difficulty lies in balancing the efficient learning of new tasks and overcoming catastrophic forgetting of old tasks. Although many continual learning methods have been proposed for pattern recognition, continual reinforcement learning methods for redundant musculoskeletal and robotic systems are few and have limitations. Therefore, inspired by the developmental mechanisms in motor cortex, this article proposes a neural manifold modulated continual reinforcement learning method for musculoskeletal and robotic systems. First, a recurrent neural network (RNN) with an expected neural manifold is designed and conditions of weights are derived. Second, the ability of projectors for characterizing the neural manifold within RNN is analyzed. Furthermore, a continual reinforcement learning method of RNN is proposed with the modulation of a neural manifold. The method is validated by redundant musculoskeletal and robotic systems in simulation. The results suggest that it can realize continual reinforcement learning of multiple tasks in different movements and environments. Furthermore, compared with related works, the proposed method achieves better performance.

Adaptive multi-objective reinforcement learning with hybrid exploration for traffic signal control based on cooperative multi-agent framework

Adaptive traffic signal control (ATSC) based on deep reinforcement learning (DRL) has shown promising prospects to reduce traffic congestion. Most existing methods keeping traffic signal phases fixed adopt two agent actions to match a four-phase suffering unstable performance and undesirable operation in a four-phase signalized intersection. In this paper, a Double Deep Q-Network (DDQN) with a dual-agent algorithm is proposed to obtain a stable traffic signal control policy. Specifically, two agents are denoted by two different states and shift the control of green lights to make the phase sequence fixed and control process stable. State representations and reward functions are presented by improving the observability and reducing the leaning difficulty of two agents. To enhance the feasibility and reliability of two agents in the traffic control of the four-phase signalized intersection, a network structure incorporating DDQN is proposed to map states to rewards. Experiments under Simulation of Urban Mobility (SUMO) are carried out, and results show that the proposed traffic signal control algorithm is effective in improving traffic capacity.

Research about air-craft autonomous vehicles has become a trend issue among researchers. Vertical Take-Off and Landing (VTOL) controlling for a hover-craft vehicle is one of benchmarking control problems. VTOL ability of hover-craft is significantly crucial for the vehicle to do surveillance and rescue, military, farming, etc. There are many papers concerned with this VTOL benchmarking control. The purpose of this paper is applying Deep Deterministic Policy Gradient (DDPG) as the brand new continuous deep reinforcement learning approach to solve a complex nonlinear system, such as air-craft VTOL problem. Furthermore, this paper analyses the effect of Ornstein-Uhlenbeck (OU) Noise injection to action as the exploration key issue of DDPG algorithm, in addition of the analysis, this paper also applies the gaussian noise to the plant as the disturbance of system. The test was performed in the VTOL simulation. The evaluation used the episode reward, average reward, total reward from the training process. Further analysis also uses IAE (Integral of Absolute Magnitude of Error), ISE (Integral of the Square Error) and MSE (Mean of the Square Error) as the parameter error evaluation from the system. The comparison results of all conditions this paper show that applying DDPG with the injection of OU noise to VTOL system with the disturbance of Gaussian noise achieves the best total reward of −14,355.37 with the shortest episode training of 243, thus it works best. The VTOL system without the disturbance has higher IAE and VTOL with Gaussian noise as the disturbance has higher ISE and MSE.

Cooperative traffic signal control using Multi-step return and Off-policy Asynchronous Advantage Actor-Critic Graph algorithm

In this article, we aim to provide a literature review of different formulations and approaches to continual reinforcement learning (RL), also known as lifelong or non-stationary RL. We begin by discussing our perspective on why RL is a natural fit for studying continual learning. We then provide a taxonomy of different continual RL formulations by mathematically characterizing two key properties of non-stationarity, namely, the scope and driver non-stationarity. This offers a unified view of various formulations. Next, we review and present a taxonomy of continual RL approaches. We go on to discuss evaluation of continual RL agents, providing an overview of benchmarks used in the literature and important metrics for understanding agent performance. Finally, we highlight open problems and challenges in bridging the gap between the current state of continual RL and findings in neuroscience. While still in its early days, the study of continual RL has the promise to develop better incremental reinforcement learners that can function in increasingly realistic applications where non-stationarity plays a vital role. These include applications such as those in the fields of healthcare, education, logistics, and robotics.

Reinforcement learning (RL) algorithms have been successfully applied to control tasks associated with unmanned aerial vehicles and robotics. In recent years, safe RL has been proposed to allow the safe execution of RL algorithms in industrial and mission-critical systems that operate in closed loops. However, if the system operating conditions change, such as when an unknown fault occurs, typical safe RL algorithms cannot adapt while retaining past knowledge. Continual RL algorithms have been proposed to address this issue. However, the impact of continual adaptation on the system’s safety is an understudied problem. In this paper, we study the intersection of safe and continual RL. First, we empirically demonstrate that a popular continual RL algorithm, elastic weight consolidation, does not satisfy safety constraints in nonlinear systems subject to varying operating conditions. Specifically, we study the MuJoCo HalfCheetah and Ant environments with velocity constraints and sudden joint loss non-stationarity. Then, we show that an agent trained using constrained policy optimization, a safe RL algorithm, experiences catastrophic forgetting in continual learning settings. With this in mind, we explore a simple reward-shaping method to ensure that elastic weight consolidation prioritizes remembering both safety and task performance for safety-constrained, nonlinear, and non-stationary systems.

This paper proposes a map-free navigation scheme based on continuous deep reinforcement learning to solve the problem that robots cannot flexibly avoid obstacles and navigate in a dynamic environment. The reinforcement learning algorithm used in this article is near-end strategy optimization (proximal strategy optimization, PPO), and the benchmark algorithm is the discrete deep reinforcement learning algorithm Deep Q network algorithm (Deep Q network, DQN).Experiments in the Gazebo simulation environment prove that the training efficiency and success rate of the PPO algorithm are much higher than that of the DQN algorithm. In this paper, the trained strategy model in the simulation environment is directly transplanted to the actual robot. The experimental results verify that the physical robot can have good navigation and obstacle avoidance capabilities without training again. The tested single-target navigation success rate is 80%, and the multi-target navigation success rate is 70%.

In this work, a closed-loop identification method based on a reinforcement learning algorithm is proposed for multiple-input multiple-output (MIMO) systems. This method could be an attractive alternative solution to the problem that the current frequency-domain identification algorithms are usually dependent on the attenuation factor. With this method, after continuously interacting with the environment, the optimal attenuation factor can be identified by the continuous action reinforcement learning automata (CARLA), and then the corresponding parameters could be estimated in the end. Moreover, the proposed method could be applied to time-varying systems online due to its online learning ability. The simulation results suggest that the presented approach can meet the requirement of identification accuracy in both square and non-square systems.

Traffic signal control using reinforcement learning based on the teacher-student framework

The acceleration of urbanization has left cities around the world facing significant challenges, including traffic congestion and inefficient management. Intelligent transportation systems provide new technological means to address these issues, especially in traffic signal control, where advanced communication technologies play an important role in facilitating the process. The potential of long range radio (LoRa) technology in intelligent traffic signal control systems is attracting growing interest, largely due to its capacity for long-range transmission and low power consumption. This paper explores the potential benefits of LoRa technology in traffic flow monitoring and signal control by reviewing existing studies, especially in the field of long-distance transmission and low-power communication. The study indicates that LoRa technology can effectively boost the efficacy of traffic management, reduce traffic congestion, and provide technical support for the development of future intelligent transportation systems. Nevertheless, the stability of LoRa technology in high-density urban environments and its integration with other technologies still present challenges. Further research can investigate the anti-jamming ability, communication rate optimization, and its potential for wider application of LoRa technology in Intelligent Transport Systems (ITS) to promote more efficient urban traffic management

Rapid urbanization and increasing ownership of private vehicles have led to severe traffic congestion, which has become a critical challenge in modern urban areas operated with signalized intersections. This study proposes and evaluates a novel AI-based composite framework for traffic signal control optimization, which integrates a Long Short-Term Memory (LSTM) model for traffic flow prediction, a Deep Neural Network (DNN) for vehicle speed estimation, and a Genetic Algorithm (GA) for signal phase optimization under open-loop and closed-loop objectives. It has been validated through high-fidelity microscopic simulations (i.e. VISSIM), calibrated with field data at the atypical six-leg intersection in Busan, Korea. This study found that while the overall performance of the framework for 24 hours exhibits clear trade-offs (i.e. slight total traffic throughput decrease of 2.2%), its primary value lies in its demonstrated ability to effectively mitigate severe congestion during peak hours (i.e. speed increase of up to 38.5% during the evening peak), delivering significant gains in traffic quality in need. This study also revealed that the system actively performs traffic flow redistribution as an intelligent traffic control strategy. The contributions of this study are threefold. First, it provides a methodological blueprint for an interpretable, modular control framework that decouples prediction processes from optimization, offering a transparent alternative to end-to-end Reinforcement Learning (RL) models. Second, this study offers crucial empirical insight for the evaluation of adaptive traffic control systems. Their true effectiveness may be best measured by peak hour performance rather than 24-hour averages. Finally, this study makes a conceptual contribution by identifying and quantifying the ‘reality gap’ between the predictions of an offline-trained optimizer and the outcomes of a high-fidelity simulation, highlighting a fundamental challenge for the field. HIGHLIGHTS A hierarchical AI framework is an alternative to reinforcement learning model Closed-loop outperforms open-loop in optimizing real time traffic signal control A speed stabilization objective is adopted over a speed maximization approach Leading to tangible gains in traffic quality during peak hour over 24-hour average

The traffic congestion problem in urban areas is worsening since traditional traffic signal control systems cannot provide efficient traffic control. Therefore, dynamic traffic signal control in intelligent transportation system (ITS) recently has received increasing attention. This study devised an adaptive and cooperative multi-agent fuzzy system for a decentralized traffic signal control. To achieve this goal we have worked on a model, which has three levels of control. Every intersection is controlled by its own traffic situation, its neighboring intersections recommendations and a knowledge base, which provides the traffic pattern of each intersection in any particular day of the week and hour of the day. The proposed architecture comprises a knowledge base, prediction module and a traffic observer that provide data to real traffic data preparation module, then a decision-making layer takes decision to how long should the intersection green light be extended. The proposed architecture can achieve dynamic traffic signal control. We have also developed a NetLogo-based traffic simulator to serve as the agents' world. Our approach is tested with traffic control of a large connected junction and the result obtained is promising; The average delay time can be reduced by 21.76% compared to the conventional fixed sequence traffic signal and 14.77% compared to the vehicle actuated traffic signal control strategy.

The assumption of IRL is that demonstrations are optimally acting in an environment. In the past, most of the work on IRL needed to calculate optimal policies for different reward functions. However, this requirement is difficult to satisfy in large or continuous state space tasks. Let alone continuous action space. We propose a continuous maximum entropy deep inverse reinforcement learning algorithm for continuous state space and continues action space, which realizes the depth cognition of the environment model by the way of reconstructing the reward function based on the demonstrations, and a hot start mechanism based on demonstrations to make the training process faster and better. We compare this new approach to well‐known IRL algorithms using Maximum Entropy IRL, DDPG, hot start DDPG, etc. Empirical results on classical control environments on OpenAI Gym: MountainCarContinues‐v0 show that our approach is able to learn policies faster and better.