Continuous-Time Time-Varying Policy Iteration.

This article presents a nearly optimal solution to the cooperative formation control problem for large-scale multiagent system (MAS). First, multigroup technique is widely used for the decomposition of the large-scale problem, but there is no consensus between different subgroups. Inspired by the hierarchical structure applied in the MAS, a hierarchical leader-following formation control structure with multigroup technique is constructed, where two layers and three types of agents are designed. Second, adaptive dynamic programming technique is conformed to the optimal formation control problem by the establishment of performance index function. Based on the traditional generalized policy iteration (PI) algorithm, the multistep generalized policy iteration (MsGPI) is developed with the modification of policy evaluation. The novel algorithm not only inherits the advantages of high convergence speed and low computational complexity in the generalized PI algorithm but also further accelerates the convergence speed and reduces run time. Besides, the stability analysis, convergence analysis, and optimality analysis are given for the proposed multistep PI algorithm. Afterward, a neural network-based actor-critic structure is built for approximating the iterative control policies and value functions. Finally, a large-scale formation control problem is provided to demonstrate the performance of our developed hierarchical leader-following formation control structure and MsGPI algorithm.

This paper deals with optimal tracking control problems for a class of discrete-time nonlinear systems using a generalized policy iteration adaptive dynamic programming (ADP) algorithm. First, by system transformation, the optimal tracking control problem is transformed into an optimal regulation problem. Then the generalized policy iteration ADP algorithm is employed to obtain the optimal tracking controller with convergence and optimality analysis. The developed algorithm uses the idea of two iteration procedures to obtain the iterative tracking control laws and the iterative value functions. Three neural networks, including model network, critic network and action network, are used to implement the developed algorithm. At last, an simulation example is given to demonstrate the effectiveness of the developed method.

In this paper, an optimal self-learning control scheme for discrete-time nonlinear systems is developed using a new local value iteration based adaptive dynamic programming (ADP) algorithm. The developed local value iteration algorithm permits an arbitrary positive semi-definite function to initialize the algorithm. In the developed local value iteration algorithm, the iterative value function and iterative control law are updated by a subset of the state space. A new analysis method of the convergence property is presented to show that the iterative value functions will converge to the optimum. The convergence criterion for the local value iteration algorithm is presented. A simulation example is given to demonstrate the validity of the present optimal control scheme.

In this chapter, differential games are studied for continuous-time linear and nonlinear systems, including two-player zero-sum games, multi-player zero-sum games, and multi-player nonzero-sum games, via a series of adaptive dynamic programming (ADP) approaches. First, an integral policy iteration algorithm is developed to learn online the Nash equilibrium of two-player zero-sum differential games with completely unknown continuous-time linear dynamics using the state and control data. Second, multi-player zero-sum differential games for a class of continuous-time uncertain nonlinear systems are solved by using a novel iterative ADP algorithm. Via neural network modeling for the system dynamics, the ADP technique is employed to obtain the optimal control pair iteratively so that the iterative value function reaches the optimal solution of the zero-sum differential games. Finally, an online synchronous approximate optimal learning algorithm based on policy iteration is developed to solve multi-player nonzero-sum games of continuous-time nonlinear systems without the requirement of exact knowledge of system dynamics.

To intercept a maneuvering target with a predetermined impact angle, a computational intelligence guidance law was proposed in this paper. Based on the theory of two-player zero-sum differential games, this problem is resolved efficiently by solving the Hamilton–Jacobi–Isaacs (HJI) equation. The Nash equilibrium solution of HJI equation can be solved with a policy iteration (PI) algorithm. Instead of using the offline PI algorithm, an online PI algorithm is introduced, in which the disturbance and control policies can be updated simultaneously. It can be proved that the online PI algorithm is a replacement for Newton’s iterative algorithm, the convergence of which is ensured by Kantorovich’s theorem. In the scenario of missiles intercepting targets, an adaptive critic structure based on a neural network (NN) is proposed to implement the online PI algorithm. Only one critic NN approximator is used in the PI algorithm to calculate a value function and the approximate Nash equilibrium solution. It is not necessary to acquire the exact internal dynamics information of nonlinear systems on the basis of online data sampling. The effectiveness of the computational intelligence guidance law is proven by simulation results.

Optimal [formula omitted] tracking control of nonlinear systems with zero-equilibrium-free via novel adaptive critic designs

In this paper, a novel iterative adaptive dynamic programming (ADP) algorithm is developed to solve infinite horizon optimal control problems for discrete-time nonlinear systems. When the iterative control law and iterative performance index function in each iteration cannot be accurately obtained, it is shown that the iterative controls can make the performance index function converge to within a finite error bound of the optimal performance index function. Stability properties are presented to show that the system can be stabilized under the iterative control law which makes the present iterative ADP algorithm feasible for implementation both on-line and off-line. Neural networks are used to approximate the iterative performance index function and compute the iterative control policy, respectively, to implement the iterative ADP algorithm. Finally, two simulation examples are given to illustrate the performance of the present method.

Model-free optimal controller design for continuous-time nonlinear systems by adaptive dynamic programming based on a precompensator

The convergence and stability of a value-iteration-based adaptive dynamic programming (ADP) algorithm are considered for discrete-time nonlinear systems accompanied by a discounted quadric performance index. More importantly than sufficing to achieve a good approximate structure, the iterative feedback control law must guarantee the closed-loop stability. Specifically, it is firstly proved that the iterative value function sequence will precisely converge to the optimum. Secondly, the necessary and sufficient condition of the optimal value function serving as a Lyapunov function is investigated. We prove that for the case of infinite horizon, there exists a finite horizon length of which the iterative feedback control law will provide stability, and this increases the practicability of the proposed value iteration algorithm. Neural networks (NNs) are employed to approximate the value functions and the optimal feedback control laws, and the approach allows the implementation of the algorithm without knowing the internal dynamics of the system. Finally, a simulation example is employed to demonstrate the effectiveness of the developed optimal control method.

In this paper, a model-free integral sliding mode control method is presented via adaptive dynamic programming (ADP) to address the trajectory tracking problem of reconfigurable manipulators. The integral sliding mode control method is adopted to deal with the model uncertainties of joint subsystems and a local neural network identifier is designed to compensate the effect of the interconnection dynamic couplings (IDCs) between the subsystems. Based on the ADP and the policy iteration (PI) algorithm, the Hamilton-Jacobi-Bellman (HJB) equation can be solved by using critic neural network and then we can get the optimal control strategy. Based on the Lyapunov theory, the closed-loop manipulator system is proved to be asymptotic stability. Finally, simulation results are demonstrated the effectiveness of the proposed method.

For discrete-time nonlinear systems, this research is concerned with optimal control problems (OCPs) with constrained cost, and a novel value iteration with constrained cost (VICC) method is developed to solve the optimal control law with the constrained cost functions. The VICC method is initialized through a value function constructed by a feasible control law. It is proven that the iterative value function is nonincreasing and converges to the solution of the Bellman equation with constrained cost. The feasibility of the iterative control law is proven. The method to find the initial feasible control law is given. Implementation using neural networks (NNs) is introduced, and the convergence is proven by considering the approximation error. Finally, the property of the present VICC method is shown by two simulation examples.

Optimal control of Markovian jump systems via a neural network-based ADP iterative algorithm

In this paper, the distributed zero-sum differential game problem for multi-agent nonlinear systems is investigated by using adaptive dynamic programming (ADP) technique, which brings together the distributed control, zero-sum differential graphical game and ADP. To find the cooperative Nash-equilibrium solutions of the associated coupled Hamilton-Jacobi-Isaacs (HJI) equations, the ADP technique is employed, in which a critic network is constructed to approximate the cooperative cost function online with a novel updating law. By utilizing the Lyapunov direct method, the closed-loop system and weight estimation errors are guaranteed to be uniformly ultimately bounded (UUB). Finally, simulation results demonstrate the effectiveness of the proposed method.

This paper investigates a novel fault tolerant control (FTC) scheme for reconfigurable manipulators with actuator failures based on adaptive dynamic programming (ADP) with an improved performance index function. With the help of fault observer, the estimated actuator failure is utilized to construct the performance index function that reflects the failure, regulation and control simultaneously. The fault tolerant controller consists of the desired controller and the approximated optimal feedback one. The desired controller is obtained by the system model directly. By establishing a critic neural network, the policy iteration (PI) algorithm is utilized to solve the Hamilton-Jacobi-Bellman (HJB) equation, and then the approximated optimal feedback controller can be derived directly. It is proven that the closed-loop system is uniformly ultimately bounded (UUB) based on Lyapunov stability theorem. Finally, the simulation study is given to show the effectiveness of the proposed FTC scheme.

In this chapter, by using neural network (NN)-based online learning optimal control approach, a decentralized control strategy is developed to stabilize a class of continuous-time large-scale interconnected nonlinear systems . It is proven that the decentralized control strategy of the overall system can be established by adding appropriate feedback gains to the optimal control laws of the isolated subsystems . Then, an online policy iteration (PI) algorithm is developed to solve the Hamilton–Jacobi–Bellman equations related to the optimal control problem. By constructing a set of critic NNs, the cost functions can be obtained by NN approximation, followed by the control laws. Furthermore, as a generalization, an NN-based decentralized control law is developed to stabilize the large-scale interconnected nonlinear systems using an online model-free integral PI algorithm. The model-free PI approach can solve the decentralized control problem for large-scale interconnected nonlinear systems with unknown dynamics. Finally, two simulation examples are provided to illustrate the effectiveness of the present decentralized control scheme.