Sequential Learning for Adaptive Critic Design: An Industrial Control Application

This paper presents a method for real-time tuning of cost functions in coordinated wide area controller for improved power system transient stability. The algorithm is an optimal Wide Area System-Centric Controller and Observer (WASCCO) based on Adaptive Critic Design (ACD). ACD techniques using reinforcement learning (RL) are able to approximate dynamic programming based nonlinear optimal control but, the cost function for the optimal control requires online tuning. Here cost function defined in ACD are analyzed online with Lyapunov stability function. Further an online tuning method is designed based on eigenvalues extracted based on Prony analysis and linked to performance index generation. Results on a two area system shows better response compared to conventional schemes.

The wastewater treatment process (WWTP) is of great significance to environmental protection. To improve the efficiency of the WWTP, it is crucial to ensure that the dissolved oxygen (DO) concentration tracks the set value efficiently. Due to the nonlinear and time-varying dynamics of the WWTP, traditional control methods cannot accurately control the DO concentration. To overcome these challenges, this paper proposes an online transferred heuristic dynamic programming (TrHDP) control design by combining transfer learning with adaptive critic design. First, we use the historical sample data to construct a mathematical model of the WWTP and learn the prior knowledge from the model. Then, the online control process of the DO concentration is guided by utilizing the prior knowledge. In order to avoid negative transfer and save computing resources, we design a novel decay function with the truncation mechanism. In addition, we prove the stability of the TrHDP control scheme by constructing a Lyapunov function. Finally, the performance of the TrHDP scheme is verified by the Benchmark Simulation Model No. 1. Compared with other methods, the TrHDP method possesses higher control accuracy for the DO concentration and overcomes the disadvantage of low learning efficiency of general online methods.

The design of fight control is full of challenges because of the complex model and the complicated fight environment. The traditional control methods no longer ensure the demand in the many fight modes. Advanced control methods attract more and more researchers to make application research in this field. Adaptive critic design (ACD) is an effective control method to learn the optimal control online. The performance of fight control with atmospheric disturbance needs to be improved using active control or robust control. This paper presents adaptive critic based controller design of aircraft with atmospheric disturbance. The controller does not rely on the parameters of the aircraft, and it can learn online very well. When there are atmospheric disturbances, such as random turbulence, discrete gust and wind shear, adaptive critic based controller can resist the disturbances effectively, and shows the good robust performance. Finally, the simulation proves the validity of the adaptive critic based controller for aircraft with atmospheric disturbance.

Transformation invariant image recognition has been an active research area due to its widespread applications in a variety of fields such as military operations, robotics, medical practices, geographic scene analysis, and many others. One of the primary challenges is detection and recognition of objects in the presence of transformations such as resolution, rotation, translation, scale and occlusion. In this work, we investigate a biologically-inspired computational modeling approach that exploits reinforcement learning (RL) for transformation-invariant image recognition. The RL is implemented in an adaptive critic design (ACD) framework to approximate the neuro-dynamic programming. Two ACD algorithms such as Heuristic Dynamic Programming (HDP) and Dual Heuristic dynamic Programming (DHP) are investigated and compared for transformation invariant recognition. The two learning algorithms are evaluated statistically using simulated transformations in 2-D images as well as with a large-scale UMIST 2-D face database with pose variations. Our simulations show promising results for both HDP and DHP for transformation-invariant image recognition as well as face authentication. Comparing the two algorithms, DHP outperforms HDP in learning capability, as DHP takes fewer steps to perform a successful recognition task in general. On the other hand, HDP is more robust than DHP as far as success rate across the database is concerned when applied in a stochastic and uncertain environment, and the computational complexity involved in HDP is much less.

We consider the problem of aircraft motion control under uncertainties caused by incomplete and inaccurate knowledge of the aircraft characteristics, as well as by abnormal situations in flight that affect the properties of the aircraft as a control object. One of the effective tools for solving problems of this kind, providing the adjustment of aircraft control algorithms taking into account its changed dynamics, is reinforcement learning in the variant of Approximate Dynamic Programming (ADP), in combination with artificial neural networks. In the last decade, a family of methods known as Adaptive Critic Design (ACD) has been actively developed within the ADP approach to control the behavior of complex dynamical systems. The paper discusses the application of one variant of the ACD approach, namely SNAC (Single Network Adaptive Critic) and its development through combined use with the dynamic inversion method. This approach makes it possible to form an optimal adaptive control law for aircraft motion. Its effectiveness is demonstrated on the example of longitudinal motion control for supersonic transport airplane (SST).

A new fuzzy identification method for unknown nonlinear discrete systems is presented by introducing adaptive critic designs into the generalized fuzzy hyperbolic model (GFHM). This method minimizes the long-time error other than the immediate error to improve the identification effect. We first represent the GFHM with a neural network structure, and then utilize the adaptive critic designs (ACDs) to get the optimal parameters of the network so that the long-time identification error is minimized. Finally, we give a simulation example to verify the effectiveness of this identification method.

This paper proposes an adaptive critic design-based event-triggered optimal control method for input-constrained continuous-time nonlinear systems. Adaptive critic design is a special framework of adaptive dynamic programming that approximates the value function by a critic neural network and derives the approximate optimal control policy through analytical methods. The proposed adaptive critic design considers the control input constraints by introducing a non-quadratic cost function and employs an event-triggered mechanism to reduce the number of controller executions. Unlike the existing event-triggered adaptive critic design, this paper proposes a novel finite-time adaptive law based on regression filtering scheme. The adaptive law utilizes the error information of the network weights to ensure fast convergence to the optimal control law under the event-triggered mechanism, which improves the real-time performance of the system. Additionally, explicit bounds for each parameter in the compact set and the specific convergence time estimates are provided in the convergence analysis. Finally, the effectiveness and practicality of the proposed method for real-time online applications are validated through two simulation examples.

Tuning the seismic control systems in order to achieve optimal performance is a challenging area due to the system and disturbance uncertainties. Although, model uncertainties, process time delay, and actuator dynamics can be considered as typical uncertainties, the main source of uncertainty in a seismic control system comes from the aleatory nature of earthquake disturbances. In this case, tuning of the control system based on a given seismic record may not necessarily result in optimal performance for other earthquakes. In this paper, a methodological approach is proposed for online control of active structural control systems considering seismic uncertainties. For this purpose, the concept of reinforcement learning is utilized for online tuning of an active mass drive system. The controller comprises a gain-scheduling fuzzy proportional derivative controller whose gains are tuned via an online reinforcement learning algorithm. Moreover, in order to tackle the time delays, a dynamic state predictor is utilized in conjunction with the proposed controller. To evaluate the performance of proposed controller, according to an assumed site hazard, thousand ground motion records are generated and clustered based on their spectral features using a fuzzy clustering approach. Finally, the controller is implemented in a laboratory-scale structure, and its performance is examined in the presence of the cluster centers and some real seismic records simulated on a shake table. The test results reveal successful performance of the proposed controller in tackling a wide range of seismic disturbances in the presence of time delay.

Reinforcement Learning (RL) is a machine learning methodology that develops the capability to make sequential decisions in intricate issues using trial-and-error techniques. RL has become increasingly prevalent for decision-making and control tasks in diverse fields such as industrial processes, biochemical systems and energy management. This review paper presents a comprehensive examination of the development, models, algorithms and practical uses of RL, with a specific emphasis on its application in process control. The study examines the fundamental theories, methodology and applications of RL, classifying them into two categories: classical RL such as such as Markov decision processes (MDP) and deep RL viz., actor critic methods. RL is a topic of discussion in multiple process industries, such as industrial chemical process control, biochemical process control, energy systems, wastewater treatment and the oil and gas sector. Nevertheless, the paper also highlights challenges that hinder its larger acceptance, including the requirement for substantial computational resources, the complexity of simulating real-world settings and the challenge of guaranteeing the stability and resilience of RL algorithms in dynamic and unpredictable environments. RL has demonstrated significant promise, but more research is needed to fully integrate it into industrial and environmental systems in order to solve the current challenges. Abbreviations: AC: Actor critic; AI: Artificial intelligence; ANN: Artificial neural networks; A3C: Asynchronous advantage actor critic; CRL : Classical Reinforcement learning; CV : Controlled variable; DDPG : Deep deterministic policy gradient; DQN: Deep Q network; DRL: Deep reinforcement learning; DP: Dynamic programming; FOMDP: Fully observable Markov decision process; GRU: Gated recurrent unit; LQR: Linear quadratic regulator; LSTM: Long short-term memory; ML: Machine learning; MV : Manipulated variable; MC: Monte Carlo; MDP: Markov decision process; MPC: Model predictive controller; MIMO: Multi input multi output; PG: Policy gradient; PID: Proportional integral derivative; PPO: Proximal policy optimisation; RL: Reinforcement learning; PPO: Proximal policy optimisation; SAC: Soft actor critic; SISO: Single input single output; TD: Temporal difference; TRPO: Trust region policy optimisation; TD3: Twin delayed deep deterministic policy gradient.

This paper presents the design of a neuro-controller for a turbogenerator using a novel technique based on adaptive critic designs (ACD). This adaptive critic design based neuro-controller augments/replaces the traditional automatic voltage regulator (AVR) and the turbine governor of the generator. Simulation results are presented to show that neural network controllers with the ACD have the potential to control turbogenerators when system conditions and configuration changes.

Introduction Reinforcement learning formalizes the concept of learning from interactions.1 Broadly, reinforcement learning focuses on a setting in which an agent (decision maker) sequentially interacts with an environment that is partially unknown to them. At each stage, the agent takes an action and receives a reward. The objective of the agent is to maximize rewards accumulated in the long run. There are many situations in health care where decisions are made sequentially for which reinforcement learning approaches could prove useful for decision making. Throughout this article, we consider treatment prescription as an archetypical example to connect reinforcement learning concepts to a health care setting. In this setting, the care provider, the prescribed treatment, and the patients can be viewed as the agent, the action, and the environment, respectively, as depicted in Figure 1.Figure 1: Sequential treatment of AKI or CKD complications modeled as a reinforcement learning problem.Background In this section, with the objective of making reinforcement learning literature more accessible to a clinical audience, we briefly introduce related fundamental concepts and approaches. We refer the interested reader to Sutton and Barto1 for a comprehensive introduction to reinforcement learning. Markov Decision Processes Markov decision processes (MDPs) are a formalism of the sequential decision-making problem that has been central to the theoretical and practical advancements of reinforcement learning. In each stage of an MDP, the agent observes the state of the environment and takes an action, which, in turn, results in a change of the state. This change of state is assumed to be probabilistic with the next state being determined only by the preceding state, the chosen action, and the transition probability. The agent also receives a reward that is a function of the taken action, the preceding state, and the subsequent state. In an MDP, the objective of the agent is to maximize the return defined as the reward accumulated over a time horizon. In some applications, it is common to consider the horizon to be infinite, in which case the future rewards are discounted by a factor smaller than one. The selection of action by the agent on the basis of the observed state is known as the policy. More formally, a policy is a probabilistic mapping from states to each possible action. Because the policy and the reward are a function of the state, it is critical to estimate the utility of being in a certain state. More specifically, the value function is defined as the expected return starting from a given state under the chosen policy. Under this formalism, the objective of the agent is to find the optimal policy that maximizes the value function for all states. Reinforcement Learning Methods Action-value methods are a class of reinforcement learning methods in which the actions are chosen on the basis of the estimation of their long-term value. A prominent example of an action-value method is Q-learning in which the agent iteratively takes actions with the highest estimated values and updates the action-state value function on the basis of new observations. Policy gradient methods are another class of reinforcement learning methods that seek to optimize the policy directly instead of choosing actions on the basis of their respective estimated value. Such methods could be advantageous in health care applications that entail a large number of possible actions, e.g., when recommending a wide range of drug dosages or treatment options. Clinical Applications Reinforcement learning frameworks and methods are broadly applicable to clinical settings in which decisions are made sequentially. A prominent clinical application of reinforcement learning is for treatment recommendation, which has been studied across a variety of diseases and treatments including radiation and chemotherapy for cancer, brain stimulation for epilepsy, and treatment strategies for sepsis.2–5 In such treatment recommendation settings, a policy is commonly known as a dynamic treatment regime. There are various other clinical applications of reinforcement learning including diagnosis, medical imaging, and decision support tools (see refs. 2–5 and the references therein). Reinforcement Learning in Nephrology Although there have been recent applications of machine learning in nephrology,6,7 to the best of the authors' knowledge, the application of reinforcement learning to nephrology has been primarily limited to optimizing the erythropoietin dosage in hemodialysis patients.8,9 However, there are other settings where reinforcement learning has the potential to improve patient care in nephrology. For example, reinforcement learning methods can be adopted in the treatment of the complications of AKI or CKD (Figure 1). In this problem, the state models the conditions of the patient (e.g., vital signs, laboratory test results including urine and blood tests, and urine output measurements). The action refers to the treatment options (e.g., the dosage of medications such as sodium polystyrene sulfonate, and hemodialysis). The reward models the improvement in patient conditions. Similarly, reinforcement learning can help automate and optimize the dosage of immunosuppressive drugs in kidney transplants. Challenges and Opportunities Despite the success of reinforcement learning in several simplified clinical settings, their large-scale application to patient care faces several open challenges. The complexity of human biology complicates modeling clinical decision making as a reinforcement learning problem. The state space in such settings is often enormous, which could make a purely computational approach infeasible. Moreover, modeling all potential objectives a priori as a reward function may not be feasible. To overcome these challenges and realize the potential of reinforcement learning, clinical insight can play a pivotal role. More specifically, restricting the state space to only include highly relevant clinical variables could greatly reduce the computational complexity. Furthermore, using inverse reinforcement learning,2 relevant reward functions can be learned from retrospective studies assuming the optimality of clinical decisions. Another critical challenge is addressing moral and ethical concerns. It is imperative to ensure that reinforcement learning methods do not cause harm to the patient. To this end, there exists a need for a thorough validation of such methods before their use in patient care. Hence, there is a need to go beyond retrospective studies that have been used for the proof of concept of most existing reinforcement learning methods in health care applications.2,3 The lessons learned from the success of reinforcement learning in other application areas (e.g., self-driving cars) can help navigate the path to realizing its potential in health care. Accessible open-source simulation environments that enable researchers to compare various approaches are essential to the field of reinforcement learning. OpenAI Gym is currently the leading toolkit containing a wide range of simulated environments, e.g., surgical robotics.10 The development of high-quality and reliable simulation environments for nephrology and other health care applications can facilitate the development and validation of reinforcement learning methods beyond limited retrospective studies. The adoption of methods validated in such simulation environments in actual clinical settings will require clinicians' oversight. Similar to how self-driving cars require a human driver to ensure collision avoidance, clinicians' oversight is critical to ensure the safety of the patients, especially in the early stages of the adoption of reinforcement learning methods. The data from clinicians' decisions (e.g., overruling the automated treatment recommendation) can be used to improve the reliability of autonomous systems over time and reduce the burden of clinicians' oversight.

Development and Evaluation of a Smart Charging Strategy for an Electric Vehicle Fleet Based on Reinforcement Learning

Intrinsically, driving is a Markov Decision Process which suits well the reinforcement learning paradigm. In this paper, we propose a novel agent which learns to drive a vehicle without any human assistance. We use the concept of reinforcement learning and evolutionary strategies to train our agent in a 2D simulation environment. Our model's architecture goes beyond the World Model's by introducing difference images in the auto encoder. This novel involvement of difference images in the auto-encoder gives better representation of the latent space with respect to the motion of vehicle and helps an autonomous agent to learn more efficiently how to drive a vehicle. Results show that our method requires fewer (96% less) total agents, (87.5% less) agents per generations, (70% less) generations and (90% less) rollouts than the original architecture while achieving the same accuracy of the original.

The essence of designing a good paging strategy is to incorporate the user mobility characteristics in a predictive mechanism that reduces the average paging cost with as little computational effort as possible. In this scope, we introduce a novel paging scheme based on the concept of reinforcement learning. Learning endows the paging mechanism with the predictive power necessary to determine a mobile terminal's position, without having to extract a location probability distribution for each specific user. The proposed algorithm is compared against a heuristic randomized learning strategy akin to reinforcement learning, that we invented for this purpose, and performs better than the case where no learning is used at all. It is shown that if the user normally moves only among a fraction of cells in the location area, significant savings can be achieved over the randomized strategy, without excessive time to train the network. Copyright © 2003 John Wiley & Sons, Ltd.

The essence of designing a good paging strategy is to incorporate user mobility characteristics in a predictive mechanism that reduces the average paging cost with as little computational effort as possible. We introduce a novel paging scheme based on the concept of reinforcement learning. Learning endows the paging mechanism with the predictive power necessary to determine a mobile terminal's position, without having to extract a location probability distribution for each specific user. The proposed algorithm is compared against a heuristic randomized learning strategy akin to reinforcement learning, that we invented for this purpose and performs better than the case where no learning is used at all. It is shown that if the user normally moves only among a fraction of cells in the location area, significant savings can be achieved over the randomized strategy, without excessive time to train the network.