Kernel-Based Approximate Dynamic Programming for Real-Time Online Learning Control: An Experimental Study

Iterative learning control demands the same initial state in each iteration, which is equal to the desired state. But this condition is unattainable in practice. This paper addresses the problem of some fixed initial state in iterative learning control for high-order nonlinear system. It presents a new control algorithm. In the process of tracking, this algorithm can rectify the initial errors through a step-by-step rectifying controller. The controller rectifies the x n at first, then x n−1 after finishing the rectifying actions of x n , and so on. All of these rectifying actions are finished in a small interval. Furthermore, the algorithm has shown effective in the improvement of tracking performance through simulation.

This paper presents a decentralized learning controller for large-scale nonlinear systems. A new learning method which is based on a performance index of sliding mode control is used to derive an iterative learning algorithm to obtain control actions. The effectiveness and the performance of the proposed approach are illustrated by the simulation results of a two-inverted pendulum system and a two-link manipulator. The attractive features also include a smaller residual error and robustness against nonlinear interactions.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

An approximate dynamic programming (ADP) method has shown good performance in solving optimal control problems in many small‐scale process control applications. The offline computational procedure of ADP constructs an approximation of the optimal “cost‐to‐go” function, which parameterizes the optimal control policy with respect to the state variable. With the approximate “cost‐to‐go” function computed, a multistage optimization problem that needs to be solved online at every sample time can be reduced to a single‐stage optimization, thereby significantly lessening the real‐time computational load. Moreover, stochastic uncertainties can be addressed relatively easily within this framework. Nonetheless, the existing ADP method requires excessive offline computation when applied to a high‐dimensional system. A case study of a reactor and a distillation column with recycle was used to illustrate this issue. Then, several ways were proposed to reduce the computational load so that the ADP method can be applied to high‐dimensional integrated plants. The results showed that the approach is much more superior to NMPC in both deterministic and stochastic cases. © 2009 American Institute of Chemical Engineers AIChE J, 2009

Learning-based robust output tracking control for unknown discrete-time nonlinear systems with dynamic uncertainty

In the field of dynamic vehicle routing, the importance to integrate stochastic information about possible future events in current decision making increases. Integration is achieved by anticipatory solution approaches, often based on approximate dynamic programming (ADP). ADP methods estimate the expected mean values of future outcomes. In many cases, decision makers are risk-averse, meaning that they avoid “risky” decisions with highly volatile outcomes. Current ADP methods in the field of dynamic vehicle routing are not able to integrate risk-aversion. In this paper, we adapt a recently proposed ADP method explicitly considering risk-aversion to a dynamic vehicle routing problem with stochastic requests. We analyze how risk-aversion impacts solutions’ quality and variance. We show that a mild risk-aversion may even improve the risk-neutral objective.

Approximate dynamic programming (ADP) has been widely studied from several important perspectives: algorithm development, learning efficiency measured by success or failure statistics, convergence rate, and learning error bounds. Given that many learning benchmarks used in ADP or reinforcement learning studies are control problems, it is important and necessary to examine the learning controllers from a control-theoretic perspective. This paper makes use of direct heuristic dynamic programming (direct HDP) and three typical benchmark examples to introduce a unique analytical framework that can be applied to other learning control paradigms and other complex control problems. The sensitivity analysis and the linear quadratic regulator (LQR) design are used in the paper for two purposes: to quantify direct HDP performances and to provide guidance toward designing better learning controllers. The use of LQR however does not limit the direct HDP to be a learning controller that addresses nonlinear dynamic system control issues. Toward this end, applications of the direct HDP for nonlinear control problems beyond sensitivity analysis and the confines of LQR have been developed and compared whenever appropriate to an LQR design.

SummaryThis paper proposes robust iterative learning control schemes for continuous‐time nonlinear systems with various nonparametric uncertainties under nonuniform trial length circumstances. The nonuniform trial length is described by a random variable, which causes a random data missing problem while designing and analyzing algorithms for the precise tracking problem. Three common types of nonparametric uncertainties are taken into account: norm‐bounded uncertainty, variation‐norm‐bounded uncertainty, and norm‐bounded uncertainty with unknown coefficients. A novel composite energy function is introduced with the help of a newly defined virtual tracking error for the asymptotical convergence of the proposed schemes. Extensions to multiple‐input–multiple‐output cases are also elaborated. Illustrative simulations are provided to verify the theoretical results.

MTN optimal control of MIMO non-affine nonlinear time-varying discrete systems for tracking only by output feedback

As a powerful tool for solving nonlinear complex system control problems, the model-free reinforcement learning hardly guarantees system stability in the early stage of learning, especially with high complicity learning components applied. In this paper, a reinforcement learning framework imitating many cognitive mechanisms of brain such as attention, competition, and integration is proposed to realize sample-efficient self-stabilized online learning control. Inspired by the generation of consciousness in human brain, multiple actors that work either competitively for best interaction results or cooperatively for more accurate modeling and predictions were applied. A deep reinforcement learning implementation for challenging control tasks and a real-time control implementation of the proposed framework are respectively given to demonstrate the high sample efficiency and the capability of maintaining system stability in the online learning process without requiring an initial admissible control.

Chapter 22 Duality Theory and Approximate Dynamic Programming for Pricing American Options and Portfolio Optimization

This work presents an adaptive optimal control algorithm based integral sliding mode control law for a class of continuous-time systems with input disturbance or uncertain and unknown parameters. The main objective is to find a general form of integral sliding mode control law can assure that the system states are forced to reach a sliding surface in the initial finite time. Then, an adaptive optimal control based on the approximate dynamic programming method is responsible for the robust stability of the closed-loop system. Finally, the theoretical analysis and simulation results demonstrate the performance of the proposed algorithm for a wheel inverted pendulum system.

This article introduces a droop-free, approximate optimal feedback control strategy to optimally control distributed generators (DGs) in islanded dc microgrids (DCMGs). Each DG is modeled as a control affine dynamical system, and constrained input of each DG is designed to minimize the infinite horizon quadratic state cost and nonquadratic control effort. The approximate dynamic programming (ADP) method is employed to solve the infinite horizon optimal control problem by successive approximation of the value function via a linear in the parameter (LIP) neural network (NN). The NN weights are updated online by a reinforcement learning (RL)-based tuning algorithm, and the convergence of the unknown weights to a neighborhood of the optimal weights is guaranteed without the persistence of excitation (PE). Simulation and experimental results are presented to demonstrate the effectiveness and applicability of the proposed concept.

The problem of learning control is addressed for a class of nonlinear systems for which there is no available analytic model. Based on the ability of fuzzy systems to approximate any nonlinear function, a global architecture is proposed where the unknown input-output linearizing controller is replaced by a fuzzy system whose rule conclusions are learned on line. A direct learning algorithm is implemented in which the parameter evolution is guided simultaneously by the tracking error and a prediction error of the fuzzy system output. During the learning phase, additive components, such as bounding control and sliding mode control are introduced to guarantee system's stability and parameter convergence. Stability of the closed loop system is proved according to Lyapunov theory. Simulation results for the inverted pendulum are included to demonstrate the method applicability.

In this paper, we propose a model reference learning control (MRLC) for nonlinear SISO plants where trials can do repeatedly from a same initial condition. It is step by step accomplished through off-line learning trials, not learning control input, but estimating optimal values of adjustable functions in feedback controller using input and output data. The only a priori information needed is the relative degree between input and output. Viability of the proposed control system is shown by a simple numerical simulation. >

This paper addresses the problem of designing an output error feedback control for single-input, single-output nonlinear systems with uncertain, smooth, output-dependent nonlinearities whose local Lipschitz constants are known. The considered systems are required to be observable, minimum phase with known relative degree and known high frequency gain sign: linear systems are included. The reference output signal is assumed to be smooth and periodic with known period. By developing in Fourier series expansion a suitable periodic input reference signal, an output error feedback adaptive learning control is designed which ldquolearnsrdquo the input reference signal by identifying its Fourier coefficients: bounded closed loop signals and exponential tracking of both input and output reference signals are obtained when the Fourier series expansion is finite, while arbitrary small tracking errors are exponentially achieved otherwise. The resulting control is not model based, is independent of the system order and depends only on the relative degree, the reference signal period and the high frequency gain sign.