Decentralized Zero-sum Games for Nonlinear Systems Based on Off-policy Learning Scheme

Optimal Tracking Control Based on Integral Reinforcement Learning for An Underactuated Drone

This paper aims to find the optimal control solution of an affine nonlinear continuous-time system with unknown input dynamic. Based on Critic-Actor neural network, an online integral reinforcement learning algorithm has been proposed. The algorithm solves the Bellman equation online, while Critic neural network is used to approximate the value function and Actor neural network is used for policy improvement. The policy evaluation and policy improvement of integral reinforcement learning are performed alternately until the performance of control systems no longer improves. By using Lyapunov function theory, all the weights of Critic-Actor neural network and the states of the system are guaranteed to be locally uniformly ultimately bounded. The simulation results show the effectiveness of the developed method.

Data-based decentralized learning scheme for nonlinear systems with mismatched interconnections

This paper focuses on the optimal containment control problem for the nonlinear multiagent systems with partially unknown dynamics via an integral reinforcement learning algorithm. By employing integral reinforcement learning, the requirement of the drift dynamics is relaxed. The integral reinforcement learning method is proved to be equivalent to the model-based policy iteration, which guarantees the convergence of the proposed control algorithm. For each follower, the Hamilton-Jacobi-Bellman equation is solved by a single critic neural network with a modified updating law which guarantees the weight error dynamic to be asymptotically stable. Through using input-output data, the approximate optimal containment control protocol of each follower is obtained by applying the critic neural network. The closed-loop containment error system is guaranteed to be stable under the proposed optimal containment control scheme. Simulation results demonstrate the effectiveness of the presented control scheme.

In this work, a novel event-based integral reinforcement learning (IRL) adaptive control method is developed to solve the multiplayer non-zero-sum (NZS) games of the nonlinear systems with unknown drift dynamics. By virtue of the IRL algorithm, the system drift dynamics is no more needed in the controller design. Moreover, different from the existing iteration computation methods, this method is online implemented, on which condition the event-triggered control framework can be combined with the IRL algorithm in solving the NZS game problems. In this method, a state-dependent triggering condition is proposed, thus the computation and communication loads are reduced in the control process. Moreover, the uniform ultimate boundedness (UUB) stability of the controlled system and the convergence of the critic weights have also been proved. Finally, a numerical example is provided to demonstrate the effectiveness of our method.

Benefitting from the technology of integral reinforcement learning, the nonzero sum (NZS) game for distributed parameter systems is effectively solved in this paper when the information of system dynamics are unavailable. The Karhunen-Loève decomposition (KLD) is employed to convert the partial differential equation (PDE) systems into high-order ordinary differential equation (ODE) systems. Moreover, the off-policy IRL technology is introduced to design the optimal strategies for the NZS game. To confirm that the presented algorithm will converge to the optimal value functions, the traditional adaptive dynamic programming (ADP) method is first discussed. Then, the equivalence between the traditional ADP method and the presented off-policy method is proved. For implementing the presented off-policy IRL method, actor and critic neural networks are utilized to approach the value functions and control strategies in the iteration process, individually. Finally, a numerical simulation is shown to illustrate the effectiveness of the proposal off-policy algorithm.

Dynamic compensator-based near-optimal control for unknown nonaffine systems via integral reinforcement learning

In this paper, we develop an integral reinforcement learning algorithm based on policy iteration to learn online the Nash equilibrium solution for a two-player zero-sum differential game with completely unknown linear continuous-time dynamics. This algorithm is a fully model-free method solving the game algebraic Riccati equation forward in time. The developed algorithm updates value function, control and disturbance policies simultaneously. The convergence of the algorithm is demonstrated to be equivalent to Newton's method. To implement this algorithm, one critic network and two action networks are used to approximate the game value function, control and disturbance policies, respectively, and the least squares method is used to estimate the unknown parameters. The effectiveness of the developed scheme is demonstrated in the simulation by designing an H <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> state feedback controller for a power system.

This article addresses the optimal tracking control problem with prescribed performance for uncertain nonlinear systems subject to input constraint and unknown disturbances. First, a fixed‐time monotonic convergence function is introduced to restrain tracking error, and a nonlinear mapping technique is employed to transform the constrained error into an unconstrained variable, then the fixed‐time output tracking issue is boiled down to the boundedness problem of the transformed variable. With the aid of a nonquadratic cost function, the input constraint is encoded into the optimization problem. To solve the unknown disturbances, an auxiliary system and an auxiliary disturbance policy are constructed, and the optimal control problem is formulated as a two‐player zero‐sum game. Moreover, a Hamilton–Jacobi–Isaacs (HJI) equation associated with this nonquadratic zero‐sum game is established to give the optimal control and the worst‐case disturbance policy solution. Subsequently, to avoid using knowledge of the system dynamics, three neural network approximators, namely, actor, critic, and disturbance, which are tuned online and simultaneously for approximating the solution of HJI, are constructed based on the integral reinforcement learning algorithm. Theoretical analysis shows that the reconstructed error system states and the weight estimation errors are semi‐globally uniformly ultimately bounded. Finally, the simulation study further tests the availability of the proposed control strategy.

In this paper, a comparison is addressed between two methods, that is, the optimal tracking control methods of unknown nonlinear systems with and without constrained input. Firstly, the optimal tracking problem for a class of affine nonlinear system is formulated. The tracking cost functions are also defined, both for the two methods. The two methods are proved to be equivalent as the actuator bound is large enough. Integral reinforcement learning (IRL) algorithm is employed to solve the optimal tracking problem by using only system data. To facilitate the implementation of the IRL algorithm, the actor-critic neural network technique and the least squares method are employed in approximating the unknown weights iteratively. In the simulation, a detailed comparison is given to demonstrate the relationship between the two methods in the aspects of control input and tracking cost value.

This paper presents a model-free optimal tracking control algorithm for an aircraft skin inspection robot with constrained-input and input time-delay. To tackle the input time-delay problem, the original system is transformed into a delay-free system with constrained-input and unknown input coupling term. In order to overcome the optimal control problem subject to constrained-input, a discounted value function is employed. In general, it is known that the HJB equation does not admit a classical smooth solution. Moreover, since the input coupling term of the delay-free system is unknown, a model-free integral reinforcement learning(IRL) algorithm which only requires the system sampling data generated by arbitrary different control inputs and external disturbances is proposed. The model-free IRL method is implemented on an actor-critic neural network (NN) structure. A system sampling data set is utilized to learn the value function and control policy. Finally, the simulation verifies the effectiveness of the proposed algorithm.

This paper focuses on the data-driven controller design for optimal control problems of nonlinear nonaffine discrete-time systems. A novel policy gradient and Q-learning (PGQL) adaptive algorithm which learns the optimal control policy from real empirical data is developed without requiring system dynamics. A policy iteration scheme is designed to iteratively update the approximate Q-function, and the control policy is improved via gradient method until they converge to the bounded regions of the optimal Q-function and the optimal control policy, respectively. Two neural networks (NNs) are employed to realize the developed algorithm. Moreover, the convergence analysis of approximate Q-function is established. Since the control policy is parameterized, it can be upgraded through updating the actor-NN parameters in the direction of the performance gradient. Finally, the simulation results are given to verify the performance of the developed PGQL adaptive algorithm.

The proposed event-triggered control design uses the adaptive dynamic programming (ADP) technique to solve the infinite-horizon optimal control of nonlinear continuous time system in affine form with complete unknown system dynamics in a forward time and online manner. The approximation property of the neural network (NN) is used to estimate the system dynamics and the value function with event-based sampling of state vector. Subsequently the estimated values are used to design the near optimal control policy. In addition, the NN weights are updated as a jump at every trigger instant, hence aperiodic in nature, to save computation when compared to the traditional NN-based approaches. Further, the closed-loop dynamics are formulated as a nonlinear impulsive dynamical system and the extension of the Lyapunov technique is utilized to prove the locally ultimate boundedness of all the closed-loop signals by deriving an adaptive event-trigger condition. Nonetheless, a positive lower bound on the inter-event time is guaranteed to avoid accumulation point. Finally, the analytical design is evaluated by using an example.

This paper presents a novel decentralized control strategy for a class of uncertain nonlinear large-scale systems with mismatched interconnections. First, it is shown that the decentralized controller for the overall system can be represented by an array of optimal control policies of auxiliary subsystems. Then, within the framework of adaptive dynamic programming, a simultaneous policy iteration (SPI) algorithm is developed to solve the Hamilton–Jacobi–Bellman equations associated with auxiliary subsystem optimal control policies. The convergence of the SPI algorithm is guaranteed by an equivalence relationship. To implement the present SPI algorithm, actor and critic neural networks are applied to approximate the optimal control policies and the optimal value functions, respectively. Meanwhile, both the least squares method and the Monte Carlo integration technique are employed to derive the unknown weight parameters. Furthermore, by using Lyapunov’s direct method, the overall system with the obtained decentralized controller is proved to be asymptotically stable. Finally, the effectiveness of the proposed decentralized control scheme is illustrated via simulations for nonlinear plants and unstable power systems.

This paper is concentrated on aperiodic sampling and adaptive learning in continuous-time nonlinear systems. The objective is to find the optimal control policy by minimizing an infinite-quadratic cost function in an online fashion. An algorithm based on integral reinforcement learning is proposed for problems with unknown model information, and a novel dynamic triggering method is combined for saving communication as much as possible. The identifying neural network is utilized to free the input dynamics and critic neural network is constructed to approximate the cost function. The weight convergence and asymptotic stability are theoretically elaborated, and afterwards, a numerical example is presented for illustrating the effectiveness and superiority of the algorithm by comparison.