Zero-sum differential games for nonlinear systems using adaptive dynamic programming with input constraint

In this paper, the distributed zero-sum differential game problem for multi-agent nonlinear systems is investigated by using adaptive dynamic programming (ADP) technique, which brings together the distributed control, zero-sum differential graphical game and ADP. To find the cooperative Nash-equilibrium solutions of the associated coupled Hamilton-Jacobi-Isaacs (HJI) equations, the ADP technique is employed, in which a critic network is constructed to approximate the cooperative cost function online with a novel updating law. By utilizing the Lyapunov direct method, the closed-loop system and weight estimation errors are guaranteed to be uniformly ultimately bounded (UUB). Finally, simulation results demonstrate the effectiveness of the proposed method.

To intercept a maneuvering target with a predetermined impact angle, a computational intelligence guidance law was proposed in this paper. Based on the theory of two-player zero-sum differential games, this problem is resolved efficiently by solving the Hamilton–Jacobi–Isaacs (HJI) equation. The Nash equilibrium solution of HJI equation can be solved with a policy iteration (PI) algorithm. Instead of using the offline PI algorithm, an online PI algorithm is introduced, in which the disturbance and control policies can be updated simultaneously. It can be proved that the online PI algorithm is a replacement for Newton’s iterative algorithm, the convergence of which is ensured by Kantorovich’s theorem. In the scenario of missiles intercepting targets, an adaptive critic structure based on a neural network (NN) is proposed to implement the online PI algorithm. Only one critic NN approximator is used in the PI algorithm to calculate a value function and the approximate Nash equilibrium solution. It is not necessary to acquire the exact internal dynamics information of nonlinear systems on the basis of online data sampling. The effectiveness of the computational intelligence guidance law is proven by simulation results.

In this chapter, differential games are studied for continuous-time linear and nonlinear systems, including two-player zero-sum games, multi-player zero-sum games, and multi-player nonzero-sum games, via a series of adaptive dynamic programming (ADP) approaches. First, an integral policy iteration algorithm is developed to learn online the Nash equilibrium of two-player zero-sum differential games with completely unknown continuous-time linear dynamics using the state and control data. Second, multi-player zero-sum differential games for a class of continuous-time uncertain nonlinear systems are solved by using a novel iterative ADP algorithm. Via neural network modeling for the system dynamics, the ADP technique is employed to obtain the optimal control pair iteratively so that the iterative value function reaches the optimal solution of the zero-sum differential games. Finally, an online synchronous approximate optimal learning algorithm based on policy iteration is developed to solve multi-player nonzero-sum games of continuous-time nonlinear systems without the requirement of exact knowledge of system dynamics.

ABSTRACTThis paper investigates a nonlinear two‐player non‐zero‐sum differential game with state and input constraints. To solve this problem, this paper constructs a neural network (NN) framework to approximate the solution of the Hamilton‐Jacobi‐Isaacs (HJI) equation. The adaptive dynamic programming (ADP) method is utilized where each player only needs one critic NN. To solve the issue of state and input saturations, this paper develops a novel constrained system for the differential game, firstly to make the states within the predetermined constraint set. Then, the non‐quadratic expression is used to substitute the traditional quadratic expression for the two‐player non‐zero‐sum differential game, and both of the inputs of the two players are constrained. With these treatments, the control input and the system are more in line with real‐world applications. Moreover, the stability of the system is also analyzed using the Lyapunov theorem. Two numerical examples are presented to illustrate that the critic NN weights estimation errors and the system are uniformly ultimately bounded (UUB), and the state and input constraints can be achieved.

In this paper, an adaptive dynamic programming (ADP)-based self-learning algorithm is developed for solving the two-person zero-sum differential games for continuous-time nonlinear systems with saturating controllers. Optimal control pair is iteratively obtained by the proposed ADP algorithm that makes the performance index function reach the saddle point of the zero-sum differential games. It shows that the iterative control pairs stabilize the nonlinear systems and the iterative performance index functions converge to the saddle point. Finally, a simulation example is given to illustrate the performance of the proposed method.

Optimal [formula omitted] tracking control of nonlinear systems with zero-equilibrium-free via novel adaptive critic designs

In this study, the problem of intercepting a manoeuvring target is posed in a zero‐sum differential game problem for a class of strict‐feedback non‐linear systems with output and input constraints. By introducing a barrier Lyapunov function and an auxiliary system to deal with the output constraints and input constraints, respectively, a novel backstepping feedforward controller is designed to transform the tracking problem for strict‐feedback systems into an equivalence differential game problem for affine systems. Subsequently, a zero‐sum differential game strategy is developed by using the adaptive dynamic programming technique. A critic network is constructed to learn the Nash equilibrium of the Hamilton–Jacobi–Isaacs equation online. The convergence properties of the proposed backstepping‐based differential games are developed by utilising the Lyapunov method. Finally, the effectiveness of the proposed strategy is demonstrated by simulation using missile‐target interception system.

This paper focuses on a class of unknown discrete-time nonlinear systems with dead-zone control constraints based on adaptive dynamic programming (ADP). The discrete-time Hamilton-Jacobi-Bellman (DTHJB) equation corresponding to the dead-zone control input is formulated. Based on ADP technique, a new cost function is proposed that solves the optimal control with dead zone constraints effectively. It shows that this algorithm allows the implementation of the optimal control without knowing nonlinear affine system model and dead-zone dynamics. Finally, a simulation example is provided to verify the effectiveness of the proposed iterative algorithm.

Optimal control of aluminum electrolysis production process (AEPP) has long been a challenging industrial issue due to its inherent difficulty in establishing an accurate dynamic model. In this paper, a novel robust optimal control algorithm based on adaptive dynamic programming (ADP) is proposed for the AEPP, where the system subjects to input constraints. First, to establish an accurate dynamic model for the AEPP system, recursive neural network (RNN) is employed to reconstruct the system dynamic using the input-output production data. To ensure input constraints are not to exceed the bound of the actuator, the optimal control problem of the AEPP is formulated under a new nonquadratic form performance index function. Then, considering the perturbation of the AEPP, the robust control problem is effectively converted to the constrained optimal control problem via system transformation. Furthermore, a single critic network framework is developed to obtain the approximate solution of the Hamilton-Jacobi-Bellman (HJB) equation. Finally, the proposed ADP controller is applied to the AEPP system to validate the effectiveness and performance.

In this paper, an on-line adaptive dynamic programming (ADP) algorithm is presented to solve the optimal control problem for continuous-time nonlinear systems with control constraints. First, a suitable non-quadratic performance function is introduced to confront control constraints, and then we present an on-line ADP algorithm in which dynamic programming techniques and neural networks are used to solve the optimal control problem for nonlinear systems with control constraints, which is completely different with the traditional ADP algorithm using off-line training, and the corresponding convergence proof is given. Finally, a simulation examples is given to demonstrate the convergence and feasibility of the proposed algorithm.

In adaptive dynamic programming (ADP), traditional actor-critic methods make use of both policy gradient learning and value function approximation to search for near-optimal control policies in continuous spaces. Since neural networks are used to design the critic component, the generalization capability and learning efficiency of ADP need to be improved. In this paper, by integrating online kernel learning and approximately linear dependence (ALD) analysis to design critic module of ADP, a novel online sparse kernel-based ADP algorithm is presented. And then, the designing procedure of kernel-based heuristic dynamic programming (KHDP) is analyzed. Finally, in optimal control nonlinear discrete-time systems, an illustrated example is provided to demonstrate the effectiveness of the KHDP algorithm.

This paper proposes a novel cerebellar model articulation controller (CMAC) based controller design method for nonlinear mechanical systems. CMAC is employed to approximate and compensate the uncertainties induced by inaccurate modelling of the system to be controlled. A mathematical model of CMAC in which the adjustable parameters appear linearly is derived and the stability of the overall system is proved using the Lyapunov theory. The authors also extend the proposed scheme to obtain robustness under the bounded disturbances and the approximation error of the CMAC. Although the control problem of a simple nonlinear system is considered in this work, the proposed scheme can easily be extended to more general nonlinear systems.

This paper addresses the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$H_{\infty }$</tex-math></inline-formula> output regulation problem of affine nonlinear systems with unmeasurable states and completely unknown dynamics. The <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$H_{\infty }$</tex-math></inline-formula> output regulation problem is formulated into a two-player zero-sum differential game problem with the control input and disturbance being the two adversary players. A model-based global state- feedback optimal solution to the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$H_{\infty }$</tex-math></inline-formula> output regulation problem is presented by solving a Hamilton-Jacobi-Isaacs (HJI) equation, and a state- feedback reinforcement learning (RL) algorithm is proposed to solve the HJI equation without knowledge of system dynamics. A linear data-driven observer is designed to estimate the unmeasurable states and a novel output- feedback RL algorithm is developed based on the state- feedback RL framework by employing a critic-actor-disturbance neural network (NN) with a hidden layer. The salient found of this work is that, under the proposed output- feedback RL algorithm, the closed-loop system can reach globally stable. A numerical example is provided to demonstrate the effectiveness of the proposed solution.

In the past decade, adaptive dynamic programming (ADP) has been widely used to realize online learning tracking control of dynamical systems, where neural networks with manually designed features are commonly used. In order to improve the generalization capability and learning efficiency of ADP, this paper presents a novel framework of ADP with sparse kernel machines by integrating kernel methods and approximately linear dependence (ALD) analysis into the critic module of ADP for the optimal tracking controller design. An ADP algorithm based on sparse kernel learning and heuristic dynamic programming (HDP) is proposed, that is, kernel HDP (KHDP). Based on KHDP, an experiment is established. By simulation, the effectiveness of proposed algorithm is demonstrated.

This paper is concerned with a new data-driven zero-sum neuro-optimal control problem for continuous-time unknown nonlinear systems with disturbance. According to the input-output data of the nonlinear system, an effective recurrent neural network is introduced to reconstruct the dynamics of the nonlinear system. Considering the system disturbance as a control input, a two-player zero-sum optimal control problem is established. Adaptive dynamic programming (ADP) is developed to obtain the optimal control under the worst case of the disturbance. Three single-layer neural networks, including one critic and two action networks, are employed to approximate the performance index function, the optimal control law, and the disturbance, respectively, for facilitating the implementation of the ADP method. Convergence properties of the ADP method are developed to show that the system state will converge to a finite neighborhood of the equilibrium. The weight matrices of the critic and the two action networks are also convergent to finite neighborhoods of their optimal ones. Finally, the simulation results will show the effectiveness of the developed data-driven ADP methods.