Safe Reinforcement Learning in Continuous State Spaces based on DQN

Self-Preserving Genetic Algorithms (SPGA) combine the evolutionary strategy of a genetic algorithm with safety assurance methods commonly implemented in safe reinforcement learning (SRL), a branch of reinforcement learning (RL) that accounts for safety in the exploration and decision-making process of the agent. Safe learning approaches are especially important in safety-critical environments, where failure to account for the safety of the controlled system could result in the loss of millions of dollars in hardware or bodily harm to people working nearby, as is true of many cyber-physical systems. While SRL is a viable approach to safe learning, there are many challenges that must be taken into consideration when training agents, such as sample efficiency, stability, and exploration---an issue that is easily addressed by the evolutionary strategy of a genetic algorithm. By combining GAs with the safety mechanisms used with SRL, SPGA offers a safe learning alternative that is able to explore large areas of the solution space, addressing SRL's challenge of exploration. This work implements SPGA with both action masking and run time assurance safety strategies to evolve safe controllers for three types of discrete action space environments applicable to cyber physical systems (control, routing, and operations) and under various safety conditions. Training and testing evaluation metrics are compared with results from SRL trained controllers to validate results. SPGA and SRL controllers are trained across 5 random seeds and evaluated on 500 episodes to calculate average wall time to train, average expected return, and percentage of safe action evaluation metrics. SPGA achieves comparable reward and safety performance results with significantly improved training efficiency (55x faster on average), demonstrating the effectiveness of this safe learning approach.

Reinforcement learning (RL) algorithms have been successfully applied to control tasks associated with unmanned aerial vehicles and robotics. In recent years, safe RL has been proposed to allow the safe execution of RL algorithms in industrial and mission-critical systems that operate in closed loops. However, if the system operating conditions change, such as when an unknown fault occurs, typical safe RL algorithms cannot adapt while retaining past knowledge. Continual RL algorithms have been proposed to address this issue. However, the impact of continual adaptation on the system’s safety is an understudied problem. In this paper, we study the intersection of safe and continual RL. First, we empirically demonstrate that a popular continual RL algorithm, elastic weight consolidation, does not satisfy safety constraints in nonlinear systems subject to varying operating conditions. Specifically, we study the MuJoCo HalfCheetah and Ant environments with velocity constraints and sudden joint loss non-stationarity. Then, we show that an agent trained using constrained policy optimization, a safe RL algorithm, experiences catastrophic forgetting in continual learning settings. With this in mind, we explore a simple reward-shaping method to ensure that elastic weight consolidation prioritizes remembering both safety and task performance for safety-constrained, nonlinear, and non-stationary systems.

We propose a computationally efficient approach to safe reinforcement learning (RL) for frequency regulation in power systems with high levels of variable renewable energy resources. The approach draws on set-theoretic control techniques to craft a neural network-based control policy that is guaranteed to satisfy safety-critical state constraints, without needing to solve a model predictive control or projection problem in real time. By exploiting the properties of robust controlled-invariant polytopes, we construct a novel, closed-form "safety-filter" that enables end-to-end safe learning using any policy gradient-based RL algorithm. We then apply the safety filter in conjunction with the deep deterministic policy gradient (DDPG) algorithm to regulate frequency in a modified 9-bus power system, and show that the learned policy is more cost-effective than robust linear feedback control techniques while maintaining the same safety guarantee. We also show that the proposed paradigm outperforms DDPG augmented with constraint violation penalties.

Safe navigation is essential for autonomous systems operating in hazardous environments. Traditional planning methods are effective for solving long-horizon tasks but depend on the availability of a graph representation with prede-fined distance metrics. In contrast, safe Reinforcement Learning (RL) is capable of learning complex behaviors without relying on manual heuristics but fails to solve long-horizon tasks, particularly in goal-conditioned and multi-agent scenarios. In this paper, we introduce a novel method that integrates the strengths of both planning and safe RL. Our method leverages goal-conditioned RL (GCRL) and safe RL to learn a goal-conditioned policy for navigation while concurrently estimating cumulative distance and safety levels using learned value functions via an automated self-training algorithm. By constructing a graph with states from the replay buffer, our method prunes unsafe edges and generates a waypoint-based plan that the agent then executes by following those waypoints sequentially until their goal locations are reached. This graph pruning and planning approach via the learned value functions allows our approach to flexibly balance the trade-off between faster and safer routes especially over extended horizons. Utilizing this unified high-level graph and a shared low-level safe GCRL policy, we extend this approach to address the multi-agent safe navigation problem. In particular, we leverage Conflict-Based Search (CBS) to create waypoint-based plans for multiple agents allowing for their safer navigation over extended horizons. This integration enhances the scalability of goal-conditioned safe RL in multi-agent scenarios, enabling efficient coordination among agents. Extensive benchmarking against state-of-the-art baselines demonstrates the effectiveness of our method in achieving distance goals safely for multiple agents in complex and hazardous environments. Our code and further details about or work is available at https://safe-visual-mapf-mers.csail.mit.edu/.

Teaching a humanoid robot to walk faster through Safe Reinforcement Learning

Despite the gained prominence made by reinforcement learning (RL) in various domains, ensuring safety in real-world applications remains a significant challenge. Offline safe RL, which learns safe policies from pre-collected data, has emerged to address these concerns. However, existing approaches assume a single constraint mode and lack adaptability to diverse safety constraints. In real-world scenarios, we often find ourselves working with datasets gathered from various tasks, with the aim of developing a generalized policy capable of handling unknown tasks during testing. To deal with such offline safe meta RL problem, we introduce a novel framework called COSTA, which is designed to facilitate the learning of a safe generalized policy that can adapt and be transferred to unknown tasks during testing. COSTA addresses two key challenges in offline safe meta RL: First, it develops a cost-aware task inference module using contrastive learning to distinguish tasks based on safety constraints, mitigating the MDP ambiguity problem. Second, COSTA introduces a novel metric, Safe In-Distribution Score (SIDS), to assess the in-distribution degree of trajectories, out of the consideration of both reward maximization and cost constraint satisfaction. Trajectories collected with a safe exploration policy are filtered using SIDS for robust online task adaptation. Experimental results in a tailored benchmark suite within the Mujoco environments demonstrate that COSTA consistently balances safety and reward maximization, outperforming multiple baselines.

Safe reinforcement learning is important for the safety critical applications especially network security, as the exploration of some dangerous actions can result in huge short-term losses such as network failure or large scale privacy leakage. In this paper, we propose a reinforcement learning algorithm with safe exploration and uses transfer learning to reduce the initial random exploration. A blacklist is maintained to record the most dangerous state-action pairs as a safety constraint. A safe deep reinforcement learning version uses a convolutional neural network to estimate the risk levels and thus further improves the safety of the exploration and accelerates the learning speed for the learning agent. As a case study, the proposed reinforcement learning with safe exploration is applied in the anti-jamming robot communications. Experimental results show that the proposed algorithms can improve the jamming resistance of the robot and reduce the outage rate to enter the most dangerous states compared with the benchmark algorithms.

Reinforcement learning has shown encouraging results in decision-making tasks such as those found in gaming and robotics. However, when applied in safety-critical applications, like navigation, reinforcement learning can cause serious safety problems due to its trial-and-error nature. In this paper, we propose a safe reinforcement learning framework for LiDAR-based navigation. The proposed framework incorporates control barrier function theory with reinforcement learning to mitigate safety risks while learning a navigation strategy. Furthermore, by designing an online learning control barrier function, the proposed method can adapt to different obstacle settings through learning from past experiences. As a map-less solution, our approach uses only LiDAR as the perception system and requires no additional localization information. We conduct simulated experiments in ROS Gazebo under different indoor environments. The results show that our method outperforms other approaches by achieving up to twice the reward and five times fewer collisions when compared to the runner-up.

Learning-based controllers may offer advantageous performance in systems with model uncertainty but often lack safety guarantees. This paper introduces a distributed safe Reinforcement Learning (RL) architecture, where a decentralised learning-based controller is guided by a distributed Model Predictive Control (diMPC)-based safety checker. Each subsystem in the network of interconnected systems is controlled by a local reinforcement learning (RL) agent using local data only. The proposed framework exploits a distributed tube-based model predictive control logic to design local safety checkers, ensuring recursive feasibility and safety of the global system, as well as providing robustness with respect to external disturbances and coupling effects. Moreover, the local safety checker actively guides the local RL agent via a reward shaping technique, taking into account of safe or unsafe behaviour. Preliminary simulation analysis shows the effectiveness of the proposed approach for the load frequency control in a network of nonlinear interconnected microgrid systems.

Spacecraft safe proximity, as a critical component of on-orbit servicing missions, primarily encounters the following two challenges: the partial observability of the environment surrounding the service spacecraft and the necessity to evade uncertain obstacles. A safe reinforcement learning algorithm based on a graph neural network is proposed to address the constrained Markov decision problem in partially observable scenarios for spacecraft safe proximity missions. A graph neural network mechanism is introduced to solve the problem of dynamic variations in the quantity and location of obstacles in the observation area of the service spacecraft. The graph attention network is used to facilitate the extraction of feature information from the graph structure, which is then utilized as input for the subsequent reinforcement learning algorithm. The Soft Actor–Critic–Lagrangian algorithm is adopted to deal with the problems of tuning reward function parameters and balancing safety and optimality. By introducing Lagrange multipliers, the constrained optimization problem is transformed into an unconstrained optimization problem. In order to verify the effectiveness of the algorithm proposed in this paper, a spacecraft safe proximity environment model with dynamic obstacles is constructed, and the GAT-SACL algorithm proposed in this paper is validated by the Monte Carlo shooting method. The results show that the GAT-SACL algorithm possess excellent exploratory characteristics and delivers significant advantages in balancing optimality and safety.

Optimal dispatch of unbalanced distribution networks with phase-changing soft open points based on safe reinforcement learning

This paper studies safe Reinforcement Learning (safe RL) with linear function approximation and under hard instantaneous constraints where unsafe actions must be avoided at each step. Existing studies have considered safe RL with hard instantaneous constraints, but their approaches rely on several key assumptions: (i) the RL agent knows a safe action set for every state or knows a safe graph in which all the state-action-state triples are safe, and (ii) the constraint/cost functions are linear. In this paper, we consider safe RL with instantaneous hard constraints without assumption (i) and generalize (ii) to Reproducing Kernel Hilbert Space (RKHS). Our proposed algorithm, LSVI-AE, achieves O(√{d³H⁴K}) regret and O(H √{dK}) hard constraint violation when the cost function is linear and O(H?ₖ √{K}) hard constraint violation when the cost function belongs to RKHS. Here K is the learning horizon, H is the length of each episode, and ?ₖ is the information gain w.r.t the kernel used to approximate cost functions. Our results achieve the optimal dependency on the learning horizon K, matching the lower bound we provide in this paper and demonstrating the efficiency of LSVI-AE. Notably, the design of our approach encourages aggressive policy exploration, providing a unique perspective on safe RL with general cost functions and no prior knowledge of safe actions, which may be of independent interest.

Conventional penalty function-based safe reinforcement learning (RL) algorithms often handle safety and policy difference constraints separately, lacking specialized mechanisms to address scenarios where both constraints are violated simultaneously. This limitation sometimes results in severe performance degradation, underscoring the need for further investigation. To address this issue, a simple yet efficient safe RL algorithm is proposed in this work called coupled penalty-based safe policy optimization (CPSPO). CPSPO operates within the first-order local policy search framework, ensuring ease of implementation in practice. To enhance behavior correction when both constraints are violated, CPSPO introduces coupled penalties, considering the cost and Kullback-Leibler (KL)-divergence constraint violation situation simultaneously, effectively improving both constraints handling. Moreover, instead of the conventional policy ratio clipping mechanism, CPSPO directly incorporates the KL-divergence constraint in the loss function design, providing a more intuitive and practical approach to proximal safe policy optimization. Extensive experiments demonstrate that CPSPO outperforms mainstream safe RL algorithms in reward optimization and safety constraint satisfaction.

Summary In the process of waterflood development in oil and gas reservoirs, the injection/production ratio is one of the key indicators affecting both the development effectiveness and economic benefits. A reasonable injection/production ratio directly impacts the development parameters, such as reservoir pressure maintenance levels and water-cut rise rates, and it helps maintain high production capacity in the oil field, ultimately achieving a higher recovery factor. However, existing constrained-waterflood reservoir production optimization methods often face challenges such as time-consuming calculations and the risk of getting trapped in local optima when dealing with such nonlinear constraint problems, making it difficult to effectively achieve a balance between constraints and economic benefits. To address this issue, we propose an injection/production ratio–constrained production optimization method based on the worst-case soft actor-critic (WCSAC) algorithm. This method effectively handles the constraint issue by introducing a safety critic, ensuring the injection/production ratio constraint is met, and maximizing economic benefits throughout the entire life cycle. Specifically, this method models the injection/production ratio–constrained production optimization problem as a constrained Markov decision process (CMDP). On the one hand, it uses conditional value at risk (CVaR) as a safety metric to assess the safety risks of the policy and achieves a balance between reward and safety by adjusting adaptive safety weights. On the other hand, it uses the maximum entropy mechanism to enhance policy exploration and improve the efficiency of global optimization. This results in the training of a stable and reliable safe reinforcement learning (RL) agent to achieve injection/production ratio–constrained production optimization. The agent can adjust and determine the production optimization plan that satisfies the injection/production ratio constraint in real time based on the current reservoir development status, without the need for retraining, thus reducing computational time. To validate the performance of the proposed method, tests were conducted in two reservoir models. The analysis results indicate that, compared with other optimization methods, the proposed method can achieve a higher net present value (NPV) while satisfying the injection/production ratio constraints, demonstrating good model applicability.

Safe Chance Constrained Reinforcement Learning for Batch Process Optimization and Control