Worst-Case Soft Actor-Critic-Based Safe Reinforcement Learning Method for Nonlinear Constrained Waterflood Reservoir Production Optimization

Self-Preserving Genetic Algorithms (SPGA) combine the evolutionary strategy of a genetic algorithm with safety assurance methods commonly implemented in safe reinforcement learning (SRL), a branch of reinforcement learning (RL) that accounts for safety in the exploration and decision-making process of the agent. Safe learning approaches are especially important in safety-critical environments, where failure to account for the safety of the controlled system could result in the loss of millions of dollars in hardware or bodily harm to people working nearby, as is true of many cyber-physical systems. While SRL is a viable approach to safe learning, there are many challenges that must be taken into consideration when training agents, such as sample efficiency, stability, and exploration---an issue that is easily addressed by the evolutionary strategy of a genetic algorithm. By combining GAs with the safety mechanisms used with SRL, SPGA offers a safe learning alternative that is able to explore large areas of the solution space, addressing SRL's challenge of exploration. This work implements SPGA with both action masking and run time assurance safety strategies to evolve safe controllers for three types of discrete action space environments applicable to cyber physical systems (control, routing, and operations) and under various safety conditions. Training and testing evaluation metrics are compared with results from SRL trained controllers to validate results. SPGA and SRL controllers are trained across 5 random seeds and evaluated on 500 episodes to calculate average wall time to train, average expected return, and percentage of safe action evaluation metrics. SPGA achieves comparable reward and safety performance results with significantly improved training efficiency (55x faster on average), demonstrating the effectiveness of this safe learning approach.

An add-on module for SINTEF's Matlab Reservoir Simulation Toolbox. Uses Proper Orthogonal Decomposition and Trajectory Piecewise Linearization to perform production optimization, history matching and closed-loop reservoir management for reservoir simulation. One simple OW example and Egg model examples are included in order to test the module. Needs to be used in conjunction with MRST (tested with version 2014b)

Optimal dispatch of unbalanced distribution networks with phase-changing soft open points based on safe reinforcement learning

SummaryIn this work, we investigate the efficient estimation of the optimal design variables that maximize net present value (NPV) for the life-cycle production optimization during a single-well carbon dioxide (CO2) huff-n-puff (HnP) process in unconventional oil reservoirs. A synthetic unconventional reservoir model based on Bakken Formation oil composition is used. The model accounts for the natural fracture and geomechanical effects. Both the deterministic (based on a single reservoir model) and robust (based on an ensemble of reservoir models) production optimization strategies are considered. The injection rate of CO2, the production bottomhole pressure (BHP), the duration of injection and the production periods in each cycle of the HnP process, and the cycle lengths for a predetermined life-cycle time can be included in the set of optimum design (or well control) variables. During optimization, the NPV is calculated by a machine learning (ML) proxy model trained to accurately approximate the NPV that would be calculated from a reservoir simulator run. Similar to the ML algorithms, we use both least-squares (LS) support vector regression (SVR) and Gaussian process regression (GPR). Given a set of forward simulation runs with a commercial compositional simulator that simulates the miscible CO2 HnP process, a proxy is built based on the ML method chosen. Having the proxy model, we use it in an iterative-sampling-refinement optimization algorithm directly to optimize the design variables. As an optimization tool, the sequential quadratic programming (SQP) method is used inside this iterative-sampling-refinement optimization algorithm. Computational efficiencies of the ML proxy-based optimization methods are compared with those of the conventional stochastic simplex approximate gradient (StoSAG)-based methods. Our results show that the LS-SVR- and GPR-based proxy models are accurate and useful in approximating NPV in the optimization of the CO2 HnP process. The results also indicate that both the GPR and LS-SVR methods exhibit very similar convergence rates, but GPR requires 10 times more computational time than LS-SVR. However, GPR provides flexibility over LS-SVR to access uncertainty in our NPV predictions because it considers the covariance information of the GPR model. Both ML-based methods prove to be quite efficient in production optimization, saving significant computational times (at least 4 times more efficient) over a stochastic gradient computed from a high-fidelity compositional simulator directly in a gradient ascent algorithm. To our knowledge, this is the first study presenting a comprehensive review and comparison of two different ML-proxy-based optimization methods with traditional StoSAG-based optimization methods for the production optimization problem of a miscible CO2 HnP.

A safe reinforcement learning-based charging strategy for electric vehicles in residential microgrid

We propose a computationally efficient approach to safe reinforcement learning (RL) for frequency regulation in power systems with high levels of variable renewable energy resources. The approach draws on set-theoretic control techniques to craft a neural network-based control policy that is guaranteed to satisfy safety-critical state constraints, without needing to solve a model predictive control or projection problem in real time. By exploiting the properties of robust controlled-invariant polytopes, we construct a novel, closed-form "safety-filter" that enables end-to-end safe learning using any policy gradient-based RL algorithm. We then apply the safety filter in conjunction with the deep deterministic policy gradient (DDPG) algorithm to regulate frequency in a modified 9-bus power system, and show that the learned policy is more cost-effective than robust linear feedback control techniques while maintaining the same safety guarantee. We also show that the proposed paradigm outperforms DDPG augmented with constraint violation penalties.

Comparison of SQP and AL algorithms for deterministic constrained production optimization of hydrocarbon reservoirs

The study examines the complex processes of intelligent management of energy resilience in modern healthcare facilities during critical situations, large-scale failures, and prolonged outages of the external centralized power supply. The aim is to develop a comprehensive mathematical model and a Safe Deep Reinforcement Learning (Safe DRL) method that ensures guaranteed compliance with the strict physical and operational constraints of hospital energy systems, even during the intensive training phase of a neural network agent. The objectives are: to formalize in detail the decision-making procedure in an energy system in detail by transitioning to the paradigm of Constrained Markov Decision Processes (CMDP); to develop an innovative mathematical model featuring the implementation of a specialized safety layer based on Lyapunov functions; and to ensure high resilience and autonomy of the system through the implementation of a decentralized Edge-Fog data processing architecture. The methods used include: the theory of Constrained Markov Decision Processes (CMDP), deep reinforcement learning methods based on the Actor-Critic architecture, the mathematical apparatus of Lyapunov stability theory for the analytical correction of actions, and methods of simulation modeling for complex dynamic energy systems. The following results were obtained. In the course of the study, a Safe DRL method was proposed and substantiated, which integrates a Lyapunov-based projection directly into the training loop for the immediate correction of the agent’s control actions. This makes it possible to ensure strict theoretical guarantees of maintaining the required State of Charge (SoC) of battery systems and to prevent critical violations of energy system parameters beyond established safety limits, which is crucial for patient life-support. The effectiveness of the proposed approach was confirmed by a series of numerical experiments in the specialized environment, HospitalEnergyEnv. Under a full blackout scenario, the agent demonstrated adaptability and high accuracy in resource management without any violation of the established physical limits during the entire process of autonomous operation. Conclusions. The scientific novelty of the obtained results lies in the following: the existing optimization model for building energy management systems (BEMS) has been improved by introducing an analytical safety projection mechanism, which minimizes the risks of emergency equipment shutdown during the adaptation of artificial intelligence algorithms; further development of decentralized control methods for critical infrastructure based on Edge-Fog computing has been achieved, which significantly increases system fault tolerance in the event of a loss of connection with the global network and ensures obtaining quasi-optimal solutions in high-dimensional problems. The practical value of this work lies in the potential to create highly reliable autonomous energy systems for critical infrastructure facilities.

We consider primal-dual-based reinforcement learning (RL) in episodic constrained Markov decision processes (CMDPs) with non-stationary objectives and constraints, which plays a central role in ensuring the safety of RL in time-varying environments. In this problem, the reward/utility functions and the state transition functions are both allowed to vary arbitrarily over time as long as their cumulative variations do not exceed certain known variation budgets. Designing safe RL algorithms in time-varying environments is particularly challenging because of the need to integrate the constraint violation reduction, safe exploration, and adaptation to the non-stationarity. To this end, we identify two alternative conditions on the time-varying constraints under which we can guarantee the safety in the long run. We also propose the Periodically Restarted Optimistic Primal-Dual Proximal Policy Optimization (PROPD-PPO) algorithm that can coordinate with both two conditions. Furthermore, a dynamic regret bound and a constraint violation bound are established for the proposed algorithm in both the linear kernel CMDP function approximation setting and the tabular CMDP setting under two alternative conditions. This paper provides the first provably efficient algorithm for non-stationary CMDPs with safe exploration.

Teaching a humanoid robot to walk faster through Safe Reinforcement Learning

Naturally fractured reservoirs present large development risks due to the uncertainty nature of complex subsurface fracture distributions and heterogeneous petrophysical properties. therefore, it is important to consider geological uncertainties when performing field development optimization for fractured reservoirs. The objective of this study is to propose a robust optimization method capable of considering multiple fractured model realizations while performing bi-objective optimizations focused on maximizing field Net Present Value (NPV) and minimizing specific risk metrics. In this study, the embedded discrete fracture model (EDFM) was used to perform forward simulations for fracture reservoirs, and the stochastic simplex approximate gradient (StoSAG) algorithm was adopted as the optimizer. With respect to the optimization target, we considered both the mean NPV of multiple realizations and a specified risk metric of NPV from the worst-case scenario, the standard deviation, or conditional value at risk (CvaR). The Lexicographic method was employed to handle the bi-objective optimization. This method was applied to the EGG base model, which is commonly used in the field of reservoir development optimization. Fractures with different distribution patterns were embedded in the model to represent fractured reservoirs, and 10 permeability fields were used to consider permeability distribution uncertainties. The optimization results for the three selected risk metrics showed that the range of decreasing in the mean NPV was from 0.04% to 0.31 %, with the worst scenario final NPV increasing by $1.26×106, the standard deviation of the set decreasing by $2.84×106, and the CVaR at 20% increasing by $3.33×105. Although the range of change in each metric for this case was not significant compared to their original magnitudes, the changes in each optimization curve showed a positive improvement trend. This study introduced a bi-objective robust production optimization algorithm, based on the StoSAG and Lexicographic methods, designed to address production optimization of complex fractured reservoirs considering geological uncertainties. Three different risk metrics were investigated to minimize development risk while optimizing the expected field NPV, which can be critical to minimize field development risks when performing production optimization tasks.

Summary In the process of oil and gas reservoir development, the natural heterogeneity of the reservoir makes the development performance extremely sensitive to the matching of the injection-production well control scheme. Reasonable production optimization is the key to achieving balanced displacement and efficient development. Due to the complexity of oil and gas reservoirs and the large number of variables, traditional production optimization methods often face problems, such as complex and time-consuming calculations, hindering their ability to obtain global optimal solutions. Reinforcement learning (RL), with its intrinsic strengths in sequential decision-making and high-dimensional problem solving, offers a promising alternative for production optimization. Nevertheless, existing RL approaches exhibit limited exploration capabilities in complex reservoir environments, often becoming trapped in local optima. Additionally, they typically fail to incorporate well pattern information effectively, overlooking the location information and the connectivity between injection and production wells. To address these challenges, we propose a dual-reward coupled RL production optimization algorithm based on a global perspective. Specifically, the production optimization problem will be modeled as a Markov decision process. Through the intrinsic reward mechanism, the agent is encouraged to conduct deeper exploration to overcome the optimization limitations in complex environments, and the well network production information is integrated into the model in combination with the graph neural network (GNN) to enhance the modeling ability of the dynamic characteristics of the reservoir. In this way, the agent can effectively avoid relying only on local objectives for optimization and then achieve a more balanced injection and production strategy. The proposed method adaptively learns by engaging in intrinsic-extrinsic directional interactions with the uncertain reservoir environment, effectively leveraging accumulated well control experience in a manner that is similar to real-world field operating mode. Simulation results derived from two reservoir models demonstrate that, in comparison with other optimization methods, the proposed approach achieves higher net present value (NPV) and exhibits outstanding performance in oil displacement.

Safe navigation is essential for autonomous systems operating in hazardous environments. Traditional planning methods are effective for solving long-horizon tasks but depend on the availability of a graph representation with prede-fined distance metrics. In contrast, safe Reinforcement Learning (RL) is capable of learning complex behaviors without relying on manual heuristics but fails to solve long-horizon tasks, particularly in goal-conditioned and multi-agent scenarios. In this paper, we introduce a novel method that integrates the strengths of both planning and safe RL. Our method leverages goal-conditioned RL (GCRL) and safe RL to learn a goal-conditioned policy for navigation while concurrently estimating cumulative distance and safety levels using learned value functions via an automated self-training algorithm. By constructing a graph with states from the replay buffer, our method prunes unsafe edges and generates a waypoint-based plan that the agent then executes by following those waypoints sequentially until their goal locations are reached. This graph pruning and planning approach via the learned value functions allows our approach to flexibly balance the trade-off between faster and safer routes especially over extended horizons. Utilizing this unified high-level graph and a shared low-level safe GCRL policy, we extend this approach to address the multi-agent safe navigation problem. In particular, we leverage Conflict-Based Search (CBS) to create waypoint-based plans for multiple agents allowing for their safer navigation over extended horizons. This integration enhances the scalability of goal-conditioned safe RL in multi-agent scenarios, enabling efficient coordination among agents. Extensive benchmarking against state-of-the-art baselines demonstrates the effectiveness of our method in achieving distance goals safely for multiple agents in complex and hazardous environments. Our code and further details about or work is available at https://safe-visual-mapf-mers.csail.mit.edu/.

Summary Normally only approximately 30% of the oil in a reservoir is extracted during primary production, but using secondary-production methods such as water or gas injection, it is often possible to increase that percentage significantly and maintain the production rate of a reservoir over a longer period of time. In reservoirs under water or gas injection, additional gains can be obtained through an efficient strategy for management of front movement and reservoir sweep. The objective of reservoir production optimization is to maximize an outcome such as sweep efficiency or net present value (NPV) through the control of completion rates or pressures. Using optimization methods, it is possible to compute control settings that result in increased oil production and decreased water production compared with production from standard practices. In this paper, we focus on optimization using sequential quadratic programming (SQP) with an ensemble-based approach to estimate the gradient for the optimization. Although uncertainty in reservoir properties is usually important for the computation of optimal controls, here we use a single realization of the reservoir to evaluate the efficiency of the optimization algorithm. The most expensive aspect of gradient-based optimization is usually the computation of gradients. Most practical production-optimization problems involve large-scale, highly complex reservoir models with thousands of constraints, which makes numerical calculation of the gradient time consuming. Here, we use an ensemble-based approach for finding gradients and use localization to improve estimation of the gradient from a small number of realizations. The Broyden-Fletcher-Goldfarb-Shanno (BFGS) method is used for maximizing the objective function, with the Hessian estimated from a sequence of estimates of the gradient. Improving the gradient approximation using localization results in improvement to the Hessian approximation. A second important aspect of the efficiency of the method is the identification of active constraints. In this paper, we use a method for eliminating nonnegativity constraints to decrease computation time and an updating procedure to solve each iteration of SQP much faster than the base case. Both the speed of the algorithm and the final NPV were increased significantly. We evaluate the method by applying it to optimization of control settings in the Brugge field. Brugge is a 3D synthetic model designed by TNO with 20 vertical producers and 10 vertical peripheral water injectors. All of the producers and injectors are smart wells whose downhole chokes must be adjusted to optimize NPV. The total number of completion flow rates to be controlled is 84 at each timestep, with 40 timesteps (every 6 months). There are 1,200 inequality constraints on total well liquid rates and 3,360 nonnegativity constraints on completion liquid rates. There are also inequality constraints on the bottomhole pressure (BHP) for wells at each time period.

Volt-VAR control is critical to keeping distribution network voltages within allowable range, minimizing losses, and reducing wear and tear of voltage regulating devices. To deal with incomplete and inaccurate distribution network models, we propose a safe off-policy deep reinforcement learning algorithm to solve Volt-VAR control problems in a model-free manner. The Volt-VAR control problem is formulated as a constrained Markov decision process with discrete action space, and solved by our proposed constrained soft actor-critic algorithm. Our proposed reinforcement learning algorithm achieves scalability, sample efficiency, and constraint satisfaction by synergistically combining the merits of the maximum-entropy framework, the method of multiplier, a device-decoupled neural network structure, and an ordinal encoding scheme. Comprehensive numerical studies with the IEEE distribution test feeders show that our proposed algorithm outperforms the existing reinforcement learning algorithms and conventional optimization-based approaches on a large feeder.