An iterative adaptive dynamic programming algorithm for optimal control of unknown discrete-time nonlinear systems with constrained inputs

This paper deals with optimal tracking control problems for a class of discrete-time nonlinear systems using a generalized policy iteration adaptive dynamic programming (ADP) algorithm. First, by system transformation, the optimal tracking control problem is transformed into an optimal regulation problem. Then the generalized policy iteration ADP algorithm is employed to obtain the optimal tracking controller with convergence and optimality analysis. The developed algorithm uses the idea of two iteration procedures to obtain the iterative tracking control laws and the iterative value functions. Three neural networks, including model network, critic network and action network, are used to implement the developed algorithm. At last, an simulation example is given to demonstrate the effectiveness of the developed method.

In this paper, a novel iterative adaptive dynamic programming (ADP) algorithm is developed to solve infinite horizon optimal control problems for discrete-time nonlinear systems. When the iterative control law and iterative performance index function in each iteration cannot be accurately obtained, it is shown that the iterative controls can make the performance index function converge to within a finite error bound of the optimal performance index function. Stability properties are presented to show that the system can be stabilized under the iterative control law which makes the present iterative ADP algorithm feasible for implementation both on-line and off-line. Neural networks are used to approximate the iterative performance index function and compute the iterative control policy, respectively, to implement the iterative ADP algorithm. Finally, two simulation examples are given to illustrate the performance of the present method.

In this paper, a new iterative optimal learning control scheme for discrete-time nonlinear systems using iterative adaptive dynamic programming (ADP) approach is developed to obtain the optimal control law for coal gasification control systems. For the unknown coal gasification process, neural networks (NNs) are introduced to reconstruct the dynamics of the gasification process, where the approximation errors of the reconstruction dynamics are considered. Via system transformation, the optimal tracking control problem with approximation errors is transformed into a two-person zero-sum optimal control problem. A new iterative ADP algorithm is then developed to obtain the optimal control law for the transformed system with convergence analysis. Finally, numerical results are given to illustrate the performance of the present method.

Finite-horizon neuro-optimal tracking control for a class of discrete-time nonlinear systems using adaptive dynamic programming approach

In this paper, a novel value iteration adaptive dynamic programming (ADP) algorithm, called "generalized value iteration ADP" algorithm, is developed to solve infinite horizon optimal tracking control problems for a class of discrete-time nonlinear systems. The developed generalized value iteration ADP algorithm permits an arbitrary positive semi-definite function to initialize it, which overcomes the disadvantage of traditional value iteration algorithms. Convergence property is developed to guarantee that the iterative performance index function will converge to the optimum. Neural networks are used to approximate the iterative performance index function and compute the iterative control policy, respectively, to implement the iterative ADP algorithm. Finally, a simulation example is given to illustrate the performance of the developed algorithm.

A neural-network-based finite-horizon optimal tracking control scheme for a class of unknown nonlinear discrete-time systems is developed. First, the tracking control problem is converted into designing a regulator for the tracking error dynamics under the framework of finite-horizon optimal control theory. Then, with convergence analysis in terms of cost function and control law, the iterative adaptive dynamic programming algorithm is introduced to obtain the finite-horizon optimal controller to make the cost function close to its optimal value within an ε-error bound. Furthermore, in order to implement the algorithm via dual heuristic dynamic programming technique, three neural networks are employed to approximate the error dynamics, the cost function, and the control law, respectively. In addition, a numerical example is given to demonstrate the validity of the present approach.

Adaptive dynamic programming for security of networked control systems with actuator saturation

This paper is concerned with a new iterative adaptive dynamic programming (ADP) algorithm to solve optimal control problems for infinite horizon discrete-time nonlinear systems using a numerical controller. The convergence conditions of the iterative ADP are developed considering the errors by the numerical controller which show that the iterative performance index functions can converge to the greatest lower bound of all performance indices within a finite error bound. Neural networks and digital computer are used to approximate the iterative performance index function and compute the numerically iterative control policy, respectively, for facilitating the implementation of the iterative ADP algorithm. Finally, a simulation example is given to illustrate the performance of the present method.

In this chapter, differential games are studied for continuous-time linear and nonlinear systems, including two-player zero-sum games, multi-player zero-sum games, and multi-player nonzero-sum games, via a series of adaptive dynamic programming (ADP) approaches. First, an integral policy iteration algorithm is developed to learn online the Nash equilibrium of two-player zero-sum differential games with completely unknown continuous-time linear dynamics using the state and control data. Second, multi-player zero-sum differential games for a class of continuous-time uncertain nonlinear systems are solved by using a novel iterative ADP algorithm. Via neural network modeling for the system dynamics, the ADP technique is employed to obtain the optimal control pair iteratively so that the iterative value function reaches the optimal solution of the zero-sum differential games. Finally, an online synchronous approximate optimal learning algorithm based on policy iteration is developed to solve multi-player nonzero-sum games of continuous-time nonlinear systems without the requirement of exact knowledge of system dynamics.

Adaptive Dynamic Programming (ADP) is a new way for the complex nonlinear system optimal controller design. But the iteration step requires infinite ensure to find the optimal control law. In this paper, the greed iterative ADP algorithm was improved by adopting-approximate method. Further, the strict mathematical analysis and proof is given. The improved algorithm based in initial state points iterative limited steps to find approximate optimal control law. So reduces the computational complexity, and beneficial to outcome of the project. Simulation illustrate to the algorithm is effective to overcome the difficulty of the infinite iteration.

We are honored to organize this special issue of Neural Computation and Applications, based on contributions to the Ninth International Symposium on Neural Networks (ISNN 2012) held on July 11–14, 2012, in Shenyang, China. Evaluated by the contributions and the recommendation of ISNN 2012 organizers, 22 papers were selected for the further review process with their extended form. Each paper has been reviewed thoroughly by three independent experts in related research area. We are very grateful for the hard work of the reviewers, which help to facilitate the outcome of this special issue with greatly improved quality of the contributions. Finally, 20 papers are accepted, to be introduced as follows. In brief, the contributions of this special issue can be roughly classified into three major categories: data-based control, data-based optimization, and data-based modeling. In each category, not only the theoretical foundation is analyzed, but also some prospective applications are demonstrated. In this introduction, we like to borrow some ideas from the contributions, such as hierarchical, and cluster as the subsidiary category idea. The first category consisting of 6 papers is devoted to the data-based control and applications area, by which an optimal or near-optimal control can be derived from the input and output data. The first 4 papers focus on the hot research direction, adaptive dynamic programming and reinforcement learning (ADPRL), and the remaining 2 papers relate to iterative learning control and optimal control. Derong Liu et al. develop an online algorithm based on policy iteration for continuous-time optimal control with infinite horizon cost for nonlinear systems. In the proposed method, a discount value function is employed, which is considered to be a more general case for optimal control problems. Meanwhile, without knowledge of the internal system dynamics, the algorithm can converge uniformly online to the optimal solution of the modified Hamilton– Jacobi–Bellman equation. By means of two neural networks, the algorithm is able to find suitable approximations of both the optimal control and the optimal cost. Dual iterative adaptive dynamic programming for a class of discrete-time nonlinear systems with time-delays is investigated by Qinglai Wei et al. Dual iterative ADP algorithm is introduced to obtain the optimal solutions of the optimal performance index function and control, where in each iteration, the performance index function and the system states are both updated. Convergence analysis is presented to prove the performance index function to reach D. Zhao (&) State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China e-mail: Dongbin.zhao@ia.ac.cn

In this paper, an iterative adaptive dynamic programming (ADP) algorithm is developed to solve the optimal cooperative control problems for residential multi-battery systems. To avoid solving high-dimensional optimal control problems, we first constrain all the batteries at their worst performance, which transforms the multi-input optimal control problem into a single-input one. Based on the worst-performance optimal control law, the optimal cooperative control law for the residential multi-battery systems is obtained, where in each iteration, only a single-input optimization problem is implemented. Finally, numerical results are given to illustrate the performance of the developed algorithm.

This paper focuses on a class of unknown discrete-time nonlinear systems with dead-zone control constraints based on adaptive dynamic programming (ADP). The discrete-time Hamilton-Jacobi-Bellman (DTHJB) equation corresponding to the dead-zone control input is formulated. Based on ADP technique, a new cost function is proposed that solves the optimal control with dead zone constraints effectively. It shows that this algorithm allows the implementation of the optimal control without knowing nonlinear affine system model and dead-zone dynamics. Finally, a simulation example is provided to verify the effectiveness of the proposed iterative algorithm.

SummaryThis paper considers optimal consensus control problem for unknown nonlinear multiagent systems (MASs) subjected to control constraints by utilizing event‐triggered adaptive dynamic programming (ETADP) technique. To deal with the control constraints, we introduce nonquadratic energy consumption functions into performance indices and formulate the Hamilton‐Jacobi‐Bellman (HJB) equations. Then, based on the Bellman's optimality principle, constrained optimal consensus control policies are designed from the HJB equations. In order to implement the ETADP algorithm, the critic networks and action networks are developed to approximate the value functions and consensus control policies respectively based on the measurable system data. Under the event‐triggered control framework, the weights of the critic networks and action networks are only updated at the triggering instants which are decided by the designed adaptive triggered conditions. The Lyapunov method is used to prove that the local neighbor consensus errors and the weight estimation errors of the critic networks and action networks are ultimately bounded. Finally, a numerical example is provided to show the effectiveness of the proposed ETADP method.

Optimal control of Markovian jump systems via a neural network-based ADP iterative algorithm