Sustainable Reservoir Operation and Control Using a Deep Reinforcement Learning Policy Gradient Method

This paper treats a class of finite-stage dynamic programming (DP) problems whose state space is an interval in the one-dimensional Euclidean space R1. By introducing a notion of “reward space” we can define, in this class, an inverse problem (called the inverse DP) to a given DP (called the main DP), because of the monotonicity and continuity of the reward functions and state transformations. Roughly speaking the inverse DP to the main DP is defined by replacing “max”, “reward functions” and “state transformations” with “min”, “inverse (in some sense) to the state transformations” and “inverse (in some sense) to the reward functions”, respectively, by replacing “terminal reward function” with “inverse to itself”, and by alternating “reward spaces” with “state spaces”, but by letting “action spaces” remain as they are. According to the monotonicity of the reward functions and state transformations we clarify the relation between the main DP and the inverse DP separately for four cases. Our main result is stated as follows. The pair of optimal reward functions and optimal policy for the main DP characterizes, in an inverse sense, the pair of optimal reward functions and optimal policy for the inverse DP, and vice versa (Section 2). Further we show that our DP represents a mathematical programming problem which has the property “recursiveness with monotonicity” in [5] and conversely such a problem can also be represented by our DP (Section 3). The last section is devoted to the illustration of several examples.

The natural flow regime can sustain the ecological integrity of riverine ecosystems. Different reservoir operation polices differ in their effects on the degree of alteration of natural flow regimes. Dynamic programming plays an important role in developing operation policies. When using dynamic programming models to develop operation policies, the discrete number of storage states (DNSS), which is a key factor affecting the reservoirs operation policies, is always been determined based on computational efficiency and economic benefits. Little consideration has been given to the ecological disturbance caused by different DNSS‐based operation policies. To analyze the impact of DNSS, we built a deterministic dynamic programming model to explore the relationship among DNSS, the flow regime alteration (ecological disturbance), and the cumulative annual power generation (economic benefits) by setting a range of DNSS scenarios. We used three reservoirs with different storage coefficients (ratios of usable storage to annual average runoff) as examples and used the range of variability approach to assess the ecological disturbance under these scenarios. We compared the results with those of a stochastic dynamic programming (SDP) model and a Bayesian SDP (BSDP) model. We found that when DNSS is low, increasing DNSS improves economic benefits but causes a more severe ecological disturbance; when DNSS is high, increasing DNSS improves the economic benefits only slightly, without exacerbating the ecological disturbance; for a given DNSS, the BSDP model provides higher economic benefits than the SDP model and a similar disturbance of the riverine ecosystem; and larger reservoirs more often cause more severe disturbance of riverine ecosystems because monthly mean flows and annual extreme flows change more drastically. Our results will help to protect the riverine ecosystems and improve economic benefits if reservoir operation managers consider DNSS using dynamic programming models.

Reservoir management is a highly complex problem, which includes the uncertainty in the inflows as well as in the objectives. Stochastic dynamic programming (SDP) has been widely used to develop the optimal operating policies of multipurpose reservoirs. In the recent years, fuzzy optimization has been used to incorporate the uncertainty caused due to the imprecise nature of objectives. In the present study, a comparison is made between a conventional SDP model and fuzzy dynamic programming models for optimal operation of a multipurpose reservoir. The two models are developed for Hirakud Reservoir on River Mahanadi in the State of Orissa in India. The performance of the reservoir is evaluated under the operating policies derived from these models through simulation. The results indicate that the fuzzy dynamic programming model performs better in achieving the flood control objective and thus in the overall performan1ce of reservoir.

The real world is full of random phenomena, complicating predictions of future changes in water resources, hence probabilistic distributions can assist in understanding their potential effects. Uncertainties could be unintended consequences of intentional changes in accidental phenomena which are part of decision making and policymaking in water resource management, and their disregard for decreasing the reliability and risk of water resource systems. Uncertainty analysis and sensitivity analysis are two processes that can be used to simulate unexpected changes and improve the usefulness of the results of modeling water resource systems. Also, because water resource systems are always associated with natural phenomena, they can be analyzed by the aid of the uncertainty and sensitivity analysis to examine different effects of the phenomena on these systems. In this chapter, the general components of water resources systems, and the concepts and terminology related to uncertainty in these systems are presented. furthermore, it has been proposed to introduce the uncertainty and necessity of its review in water resources systems, the basic principles of statistical methods and probability, and various methods. The analysis and evaluation of the failure to solve problems in modeling, simulation, and optimization of water resources systems have been addressed. In the end, due to the importance of the performance criteria of water resource systems in evaluating and analyzing these systems, it introduces common performance criteria in their performance evaluation.

The present study aims at the application of the hybrid model, which consists of artificial neural network and fuzzy logic in the reservoir operating policy during critical periods. The proposed hybrid model [fuzzy neural network (FNN)] combines the learning ability of artificial neural networks and the transparent nature of fuzzy logic. The FNN model is found to be highly adaptive and efficient in investigating nonlinear relationships among different variables. The FNN model has been developed to study the behavior of optimal release operating policy on the proposed reservoir in Pagladiya River of the Assam State in India. Here, reservoir operation policies were formulated through dynamic programming. The optimal release was related to storage, inflow, and demand. The advantages of using the FNN model in reservoir release are discussed using the case study.

Reservoir operating rules are often based on deterministic models using critical period (CP) analysis, which results in very conservative decisions. CP is defined as the historical hydrologic period that includes the lowest flow or the most severe drought, which represents functionally as the full-empty cycle of reservoir storage under a given firm yield. This paper compares CP with another time scale, the forecast horizon (FH) underlying hedging rule policies for reservoir operation. When the decisions in the initial few periods are not affected by forecast data beyond a certain period, the period is known as FH, and the number of the initial periods is known as the decision horizon. When a reservoir operation policy follows the concept of CP, CP is used as FH and the optimal yield generated within the FH is the firm yield. FH as long as the CP is not realistic because in the real world data for a such a long future period is unavailable; and using CP as FH is not effective either, because CP—the longest FH-involving the most severe drought leads to very conservative release decisions, i.e., the “near-zero-risk” decision and the time preference of utility diminishes the influence of hedging for over long periods.

Reservoir operation is a multi-objective optimization problem, and is traditionally solved with dynamic programming (DP) and stochastic dynamic programming (SDP) algorithms. The thesis presents novel algorithms for optimal reservoir operation, named nested DP (nDP), nested SDP (nSDP), nested reinforcement learning (nRL) and their multi-objective (MO) variants, correspondingly MOnDP, MOnSDP and MOnRL. The idea is to include a nested optimization algorithm into each state transition, which reduces the initial problem dimension and alleviates the curse of dimensionality. These algorithms can solve multi-objective optimization problems, without significantly increasing the algorithm complexity or the computational expenses. It can additionally handle dense and irregular variable discretization. All algorithms are coded in Java and were tested on the case study of the Knezevo reservoir in the Republic of Macedonia. Nested optimization algorithms are embedded in a cloud application platform for water resources modeling and optimization. The platform is available 24/7, accessible from everywhere, scalable, distributed, interoperable, and it creates a real-time multiuser collaboration platform.This thesis contributes with new and more powerful algorithms for an optimal reservoir operation and cloud application platform. All source codes are available for public use and can be used by researchers and practitioners to further advance the mentioned areas.

The simulation elevation-surface area-storage interrelationship of a reservoir is a crucial task in developing ideal water release policies for reservoir and dam operations. In this study, an inclusive (stochastic dynamic programming-artificial neural network (SDP-ANN)) model was established and applied to obtain an ideal reservoir operation strategy for Sg. Langat reservoir in Malaysia. The problems associated with the management of water resources mostly relate to uncertainty and the stochastic nature of the reservoir inflow, and the SDP-ANN model is meant to consider uncertainty in the input parameters such as reservoir inflow and reservoir evaporation losses. The performance of the SDP-ANN model was compared to that of the stochastic dynamic programming-autoregression (AR) model. The primary aim of the model is to decrease the squared deviation from the desired water release, which we determined by comparing the SDP-AR and SDP-ANN model performances. The results indicate that the SDP-ANN model demonstrated greater resilience and reliability with a lower supply deficit. Consequently, the case study results confirm that the SDP-ANN model performs better than the SDP-AR model in obtaining the best parameters for the reservoir operation. Specifically, a comparison of the models shows that the proposed Model 2 increased the reliability and resilience of the system by 7.5% and 6.3%, respectively.

Scalable reinforcement learning for plant-wide control of vinyl acetate monomer process

An approach to the solution of multidimensional dynamic programs is presented which differs from most decomposition methods currently used in the optimization of water resources systems. The basic concept is directly dependent on certain characteristics of incremental dynamic programing (IDP). The method, referred to as multilevel incremental dynamic programing, results in a marked reduction in computing time and thus increases the power of IDP as a tool in the optimization of multidimensional deterministic systems.

Thermal management systems (TMSs) are crucial for driving safety, mileage, and comfort in battery electric vehicles. To maximize the potential of the integrated TMSs in terms of temperature control and energy saving, the deep deterministic policy gradient (DDPG) is utilized to design a learning thermal management methodology (TMM) for it. Considering the key challenges faced, linear mapping trick and gated recurrent unit are employed to improve the original DDPG. The former empowers the DDPG agents with the ability to make decisions in the discrete-continuous hybrid action space while helping it to avoid the ’curse of dimensionality’. The latter provides the DDPG agents with beneficial historical information, which enhances its decision-making quality. Simulation results show that the proposed TMM decreases the convergence episode to 77 while increasing the convergence reward to 553.4. Further adaptive tests demonstrate that the suggested TMM promptly stabilizes the motor and battery temperatures at the desired values. Simultaneously, energy consumption decreases by 14.71% and 11.45% for two cases compared to the conventional rule-based TMM. In conclusion, the proposed method provides a theoretical foundation for addressing the hybrid action space optimization problem in integrated TMSs.

Water uses in the Indus River Basin (IRB) – agriculture, hydropower, etc. – primarily depend on meltwater from Himalayan glaciers, seasonal snow, and monsoon rains. This dependency makes the IRB vulnerable to climate change and associated deeply uncertain shifts in seasonal melt patterns, long-term glacier recession and erratic monsoon patterns. Therefore, there's a growing emphasis on creating robust adaptation plans suitable for various future scenarios, rather than focusing on a specific uncertain future. Our research aims to develop a robustness framework to capture the multi-reservoir system with multiple contributions to streamflow from various sources with unique characteristics for the IRB. Our methodology quantifies streamflow components across multiple reservoirs for the IRB into base flow, rainfall, snowmelt, and glacial melt. This breakdown enables us to model each individual component's impact under a changing climate. We use Fourier series transformations to represent shifts in magnitude and seasonality of each component. This then enables us to explore how these shifts impact the resilience of reservoir operating policies, typically based on static hydrological conditions. To determine operating policies in complex multi-reservoir systems, we use Stochastic Dual Dynamic Programming (SDDP), due to its ability to capture the stochastic nature of streamflow amidst dynamic and uncertain hydrological inputs. Running SDDP with historical flows yields policies whose robustness to streamflow change can be tested; alternatively, running SDDP with select climate change scenarios yields altered policies whose climate “niche” can then be determined. This approach enables the design of coordinated reservoir operation policy, optimal under different streamflow conditions, as well as their evaluation over a wide range of potential future conditions, enhancing the optimisation of water use and management policies for the IRB. Our findings are expected to offer valuable insights assisting policymakers, stakeholders, and water resource managers in developing more resilient and sustainable strategies to address the impacts of climate change on water resources.

Optimal control in microgrid using multi-agent reinforcement learning

Energy management and power allocation policy is considered for energy harvesting (EH) communications. In this letter, we propose a joint optimization problem with the continuous EH time and transmit power to maximize the long-term throughput based on deep deterministic policy gradient (DDPG). However, the joint optimization problem leads to a large continuous action space. In order to reduce the dimension of action space, we present a deep reinforcement learning (DRL) framework by combining DDPG and convex program. The original problem is decomposed into two-layer optimization subproblems by using the primal decomposition method. The primary problem can be solved by DDPG with a low-dimensional action space. The lower-layer subproblem can be solved by using the existing convex toolbox. Numerical simulation results show that, compared with the existing energy management or power allocation policies for EH communications, the proposed DRL framework can achieve higher long-term throughput.

The increased water demand and the increased capital construction costs of water resource development projects force engineers and water planners to utilize efficiently the existing and proposed water resources systems. Hence, it is considered necessary to find the optimum real time operating rules for systems of multiple reservoirs.Stochastic dynamic programming (SDP) has been used as the tool for deriving the real time operating rules for a system of multiple reservoirs, in preference to simulation analysis and linear programming. This is because the operation of the storages is considered as a sequential decision problem and dynamic programming is well suited to handle sequential, multi-period type decision problems. Further, the stochastic nature of streamflow is considered explicitly through a multivariate distribution for streamflow. The method theoretically produces the global optimum solution.The computational difficulties in terms of computer time and memory space have prohibited the realistic application of this method to systems of multiple reservoirs in the past. In this study, two methods have been introduced to reduce the computational difficulties of the method, thus making SDP more attractive to use in the derivation of the real time operating rules for a system of multiple reservoirs.The first method, the assumption of cross correlation of streamflow, reduces the number of states in the SDP method which reduces computer time and memory space. Further, it reduces the number of conditional probability estimates of streamflow which are used in recursive relation of the SDP algorithm. The second method, the “corridor” approach, reduces computer time involved in the SDP method by eliminating the computation of infeasible and/or inferior (suboptimal) solutions. These two methods have resulted in significant savings in the computer effort of the SDP method.The conditional probabilities of streamflow are computed by considering the hypervolume under the conditional probability density function between the integration limits using the computationally efficient Gaussian Legendre quadrature method. The conditional probability density function is derived from the historical streamflow data.Real time operating rules are expressed as a function of the storage volume(s) at the beginning of the current time interval and the forecast inflow into the storage(s) during the current time interval.The Melbourne water supply system is used as the example. The derived operating rules have been tested by operating a simulation model and the conclusion is made that the SDP method is a satisfactory technique for objectively deriving real time operating rules for a system of multiple reservoirs.Finally, a combined stochastic dynamic programming – statistical disaggregation approach is introduced to determine the real time operating rules for a system of multiple reservoirs where operational data are available. Releases from individual storages are a function of the total system storage volume and inflow. The method uses SDP to determine the release rules from the equivalent single reservoir of the multiple reservoir system; these releases are then disaggregated to individual releases using the statistical release disaggregation model whose parameters are derived from the historical operational data. The approach was used to derive real time operating rules for the Melbourne water supply system and the operating rules were validated using a simulation model of the system.