Continuous time reinforcement learning: A random measure approach

Traffic signal control can be naturally regarded as a reinforcement learning problem. Unfortunately, it is one of the most difficult classes of reinforcement learning problems owing to its large state space. A straightforward approach to address this challenge is to control traffic signals based on continuous reinforcement learning. Although they have been successful in traffic signal control, they may become unstable and fail to converge to near-optimal solutions. We develop adaptive traffic signal controllers based on continuous residual reinforcement learning (CRL-TSC) that is more stable. The effect of three feature functions is empirically investigated in a microscopic traffic simulation. Furthermore, the effects of departing streets, more actions, and the use of the spatial distribution of the vehicles on the performance of CRL-TSCs are assessed. The results show that the best setup of the CRL-TSC leads to saving average travel time by 15% in comparison to an optimized fixed-time controller.

The continual learning and development are significant for robots to learn multiple tasks sequentially. The difficulty lies in balancing the efficient learning of new tasks and overcoming catastrophic forgetting of old tasks. Although many continual learning methods have been proposed for pattern recognition, continual reinforcement learning methods for redundant musculoskeletal and robotic systems are few and have limitations. Therefore, inspired by the developmental mechanisms in motor cortex, this article proposes a neural manifold modulated continual reinforcement learning method for musculoskeletal and robotic systems. First, a recurrent neural network (RNN) with an expected neural manifold is designed and conditions of weights are derived. Second, the ability of projectors for characterizing the neural manifold within RNN is analyzed. Furthermore, a continual reinforcement learning method of RNN is proposed with the modulation of a neural manifold. The method is validated by redundant musculoskeletal and robotic systems in simulation. The results suggest that it can realize continual reinforcement learning of multiple tasks in different movements and environments. Furthermore, compared with related works, the proposed method achieves better performance.

In continual learning, an agent is exposed to a changing environment, requiring it to adapt during execution time. While traditional reinforcement learning (RL) methods have shown impressive results in various domains, there has been less progress in addressing the challenge of continual learning. Current RL approaches do not allow the agent to adapt during execution but only during a dedicated training phase. Here we study the problem of continual learning in a 2D bipedal walker domain, in which the legs of the walker grow over its lifetime, requiring the agent to adapt. The introduced approach combines neuroevolution, to determine the starting weights of a deep neural network, and a version of deep reinforcement learning that is continually running during execution time. The proof-of-concept results show that the combined approach gives a better generalisation performance when compared to evolution or reinforcement learning alone. The hybridization of reinforcement learning and evolution opens up exciting new research directions for continually learning agents that can benefit from suitable priors determined by an evolutionary process.

Research about air-craft autonomous vehicles has become a trend issue among researchers. Vertical Take-Off and Landing (VTOL) controlling for a hover-craft vehicle is one of benchmarking control problems. VTOL ability of hover-craft is significantly crucial for the vehicle to do surveillance and rescue, military, farming, etc. There are many papers concerned with this VTOL benchmarking control. The purpose of this paper is applying Deep Deterministic Policy Gradient (DDPG) as the brand new continuous deep reinforcement learning approach to solve a complex nonlinear system, such as air-craft VTOL problem. Furthermore, this paper analyses the effect of Ornstein-Uhlenbeck (OU) Noise injection to action as the exploration key issue of DDPG algorithm, in addition of the analysis, this paper also applies the gaussian noise to the plant as the disturbance of system. The test was performed in the VTOL simulation. The evaluation used the episode reward, average reward, total reward from the training process. Further analysis also uses IAE (Integral of Absolute Magnitude of Error), ISE (Integral of the Square Error) and MSE (Mean of the Square Error) as the parameter error evaluation from the system. The comparison results of all conditions this paper show that applying DDPG with the injection of OU noise to VTOL system with the disturbance of Gaussian noise achieves the best total reward of −14,355.37 with the shortest episode training of 243, thus it works best. The VTOL system without the disturbance has higher IAE and VTOL with Gaussian noise as the disturbance has higher ISE and MSE.

This paper proposes a map-free navigation scheme based on continuous deep reinforcement learning to solve the problem that robots cannot flexibly avoid obstacles and navigate in a dynamic environment. The reinforcement learning algorithm used in this article is near-end strategy optimization (proximal strategy optimization, PPO), and the benchmark algorithm is the discrete deep reinforcement learning algorithm Deep Q network algorithm (Deep Q network, DQN).Experiments in the Gazebo simulation environment prove that the training efficiency and success rate of the PPO algorithm are much higher than that of the DQN algorithm. In this paper, the trained strategy model in the simulation environment is directly transplanted to the actual robot. The experimental results verify that the physical robot can have good navigation and obstacle avoidance capabilities without training again. The tested single-target navigation success rate is 80%, and the multi-target navigation success rate is 70%.

In this work, a closed-loop identification method based on a reinforcement learning algorithm is proposed for multiple-input multiple-output (MIMO) systems. This method could be an attractive alternative solution to the problem that the current frequency-domain identification algorithms are usually dependent on the attenuation factor. With this method, after continuously interacting with the environment, the optimal attenuation factor can be identified by the continuous action reinforcement learning automata (CARLA), and then the corresponding parameters could be estimated in the end. Moreover, the proposed method could be applied to time-varying systems online due to its online learning ability. The simulation results suggest that the presented approach can meet the requirement of identification accuracy in both square and non-square systems.

In this article, we aim to provide a literature review of different formulations and approaches to continual reinforcement learning (RL), also known as lifelong or non-stationary RL. We begin by discussing our perspective on why RL is a natural fit for studying continual learning. We then provide a taxonomy of different continual RL formulations by mathematically characterizing two key properties of non-stationarity, namely, the scope and driver non-stationarity. This offers a unified view of various formulations. Next, we review and present a taxonomy of continual RL approaches. We go on to discuss evaluation of continual RL agents, providing an overview of benchmarks used in the literature and important metrics for understanding agent performance. Finally, we highlight open problems and challenges in bridging the gap between the current state of continual RL and findings in neuroscience. While still in its early days, the study of continual RL has the promise to develop better incremental reinforcement learners that can function in increasingly realistic applications where non-stationarity plays a vital role. These include applications such as those in the fields of healthcare, education, logistics, and robotics.

Learning automata as a tool for machine learning, could search the optimal state adaptively in random environment. Function optimization is a fundamental issue and many practical models are ultimately the mathematical optimization problems. In this paper, we apply the basic continuous action-set reinforcement learning automata (CARLA) model to function optimization. An application model called equiCARLA is constructed by means of equidistant discretization and linear interpolation, and it presents a superiority over the existing algorithms not only in speed but also in precision. The experimental results demonstrate the effectiveness and efficiency of our model for function optimization.

Reinforcement learning (RL) algorithms have been successfully applied to control tasks associated with unmanned aerial vehicles and robotics. In recent years, safe RL has been proposed to allow the safe execution of RL algorithms in industrial and mission-critical systems that operate in closed loops. However, if the system operating conditions change, such as when an unknown fault occurs, typical safe RL algorithms cannot adapt while retaining past knowledge. Continual RL algorithms have been proposed to address this issue. However, the impact of continual adaptation on the system’s safety is an understudied problem. In this paper, we study the intersection of safe and continual RL. First, we empirically demonstrate that a popular continual RL algorithm, elastic weight consolidation, does not satisfy safety constraints in nonlinear systems subject to varying operating conditions. Specifically, we study the MuJoCo HalfCheetah and Ant environments with velocity constraints and sudden joint loss non-stationarity. Then, we show that an agent trained using constrained policy optimization, a safe RL algorithm, experiences catastrophic forgetting in continual learning settings. With this in mind, we explore a simple reward-shaping method to ensure that elastic weight consolidation prioritizes remembering both safety and task performance for safety-constrained, nonlinear, and non-stationary systems.

The assumption of IRL is that demonstrations are optimally acting in an environment. In the past, most of the work on IRL needed to calculate optimal policies for different reward functions. However, this requirement is difficult to satisfy in large or continuous state space tasks. Let alone continuous action space. We propose a continuous maximum entropy deep inverse reinforcement learning algorithm for continuous state space and continues action space, which realizes the depth cognition of the environment model by the way of reconstructing the reward function based on the demonstrations, and a hot start mechanism based on demonstrations to make the training process faster and better. We compare this new approach to well‐known IRL algorithms using Maximum Entropy IRL, DDPG, hot start DDPG, etc. Empirical results on classical control environments on OpenAI Gym: MountainCarContinues‐v0 show that our approach is able to learn policies faster and better.

Bullying is a complex social issue that requires adaptive and context-aware intervention strategies. This paper presents a multi-agent model that integrates Theory of Mind (ToM), Reinforcement Learning (RL), and Continual Learning (CL) to improve the detection and intervention of bullying behaviors. ToM allows agents to infer the beliefs, intentions, and emotions of others, enabling socially aware decision-making. RL equips the observer agent with the ability to learn from past experiences, refining its intervention strategies dynamically. CL ensures long-term adaptability by allowing the system to retain learned behaviors while continuously adjusting to new bullying patterns. The proposed approach incorporates abstraction mechanisms derived from Theory-Theory (TT) and Simulation Theory (ST), enabling agents to reason about bullying interactions either through predefined rules or simulated experiences. The system has been evaluated in a simulated school environment with varying levels of bullying severity, demonstrating its effectiveness in dynamically adapting intervention strategies. The results indicate that combining ToM, RL, and CL leads to superior performance compared to standard RL-based approaches, particularly in high-risk bullying scenarios. This work provides a foundation for the development of socially intelligent AI systems capable of proactive and context-sensitive intervention in educational settings.

More and more complex cyberattacks targeting America’s essential infrastructure endanger the nation’s safety, financial health and people’s safety. A lot of the time, rule-based cybersecurity does not notice new and growing dangers in real-time, leaving major systems exposed. Our research introduces an AI cyber threat detection framework based on using autoencoders and LSTM networks that improves both accuracy and speed in finding threats. Continual learning and reinforcement learning are part of the system so it can adapt to new threats in real time. Tests of our system on data from replay SCADA logs and NSL-KDD show very effective detection. The model’s dependability is confirmed by metrics such as precision, recall and F1-score and both its edge and cloud deployments allow for both speed and support for a growing number of devices. One solution to explain how AI reaches its decisions is to use SHAP and LIME. For now, we have applied our results to simulated situations, but our next step is to use the system in real places. The research introduced a resilient, flexible and easily explainable artificial intelligence method to make national critical infrastructure more secure.

Large scale production grids are an important case for autonomic computing. They follow a mutualization paradigm: decision-making (human or automatic) is distributed and largely independent, and, at the same time, it must implement the highlevel goals of the grid management. This paper deals with the scheduling problem with two partially conflicting goals: fairshare and Quality of Service (QoS). Fair sharing is a wellknown issue motivated by return on investment for participating institutions. Differentiated QoS has emerged as an important and unexpected requirement in the current usage of production grids. In the framework of the EGEE grid (one of the largest existing grids), applications from diverse scientific communities require a pseudo-interactive response time. More generally, seamless integration of the grid power into everyday use calls for unplanned and interactive access to grid resources, which defines reactive grids. The major result of this paper is that the combination of utility functions and reinforcement learning (RL) provides a general and efficient method for dynamically allocating grid resources in order to satisfy both end users with differentiated requirements and participating institutions. Combining RL methods and utility functions for resource allocation was pioneered by Tesauro and Vengerov. While the application contexts are different, the resource allocation issues are very similar. The main difference in our work is that we consider a multi-criteria optimization problem that includes a fair-share objective. A first contribution of our work is the definition of a set of variables describing states and actions that allows us to formulate the grid scheduling problem as a continuous action-state space reinforcement learning problem. To capture the immediate goals of end users and the long-term objectives of administrators, we propose automatically derived utility functions. Finally, our experimental results on a synthetic workload and a real EGEE trace show that RL clearly outperforms the classical schedulers, so it is a realistic alternative to empirical scheduler design.

Pursuit-evasion games have been studied for decades under perfect information and analytically-derived policies for static environments. Here, we incorporate uncertainty in a target’s position via simulated measurements and propose a continuous deep reinforcement learning (RL) approach to support pursuit of a speed-advantaged target. An OpenAI gym environment was created for simulating pursuit game play. A Kalman filter was implemented for simulating data fusion of uncertainties associated with imperfect range and bearing measurements from the pursuer to the target. An actor-critic based, model-free, deep discrete policy gradient (DDPG) method was implemented for incrementally training an agent to compete at a speed-overmatched pursuit game. The resulting RL policy was tested under many scenarios and performance exceeded that of a baseline course-aligned strategy. Emerging reinforcement learning success strategies were analyzed and manually abstracted into grouped behaviors. An online decision-making framework is discussed for leveraging conditional past performance to predict future performance.

Central to the development of universal learning systems is the ability to solve multiple tasks without retraining from scratch when new data arrives. This is crucial because each task requires significant training time. Addressing the problem of continual learning necessitates various methods due to the complexity of the problem space. This problem space includes: (1) addressing catastrophic forgetting to retain previously learned tasks, (2) demonstrating positive forward transfer for faster learning, (3) ensuring scalability across numerous tasks, and (4) facilitating learning without requiring task labels, even in the absence of clear task boundaries. In this paper, the Task-Agnostic Policy Distillation (TAPD) framework is introduced. This framework alleviates problems (1)–(4) by incorporating a task-agnostic phase, where an agent explores its environment without any external goal and maximizes only its intrinsic motivation. The knowledge gained during this phase is later distilled for further exploration. Therefore, the agent acts in a self-supervised manner by systematically seeking novel states. By utilizing task-agnostic distilled knowledge, the agent can solve downstream tasks more efficiently, leading to improved sample efficiency. Our code is available at the repository: https://github.com/wabbajack1/TAPD.