Integral Reinforcement Learning for Linear Continuous-Time Zero-Sum Games With Completely Unknown Dynamics

This paper focuses on the optimal containment control problem for the nonlinear multiagent systems with partially unknown dynamics via an integral reinforcement learning algorithm. By employing integral reinforcement learning, the requirement of the drift dynamics is relaxed. The integral reinforcement learning method is proved to be equivalent to the model-based policy iteration, which guarantees the convergence of the proposed control algorithm. For each follower, the Hamilton-Jacobi-Bellman equation is solved by a single critic neural network with a modified updating law which guarantees the weight error dynamic to be asymptotically stable. Through using input-output data, the approximate optimal containment control protocol of each follower is obtained by applying the critic neural network. The closed-loop containment error system is guaranteed to be stable under the proposed optimal containment control scheme. Simulation results demonstrate the effectiveness of the presented control scheme.

This paper aims to find the optimal control solution of an affine nonlinear continuous-time system with unknown input dynamic. Based on Critic-Actor neural network, an online integral reinforcement learning algorithm has been proposed. The algorithm solves the Bellman equation online, while Critic neural network is used to approximate the value function and Actor neural network is used for policy improvement. The policy evaluation and policy improvement of integral reinforcement learning are performed alternately until the performance of control systems no longer improves. By using Lyapunov function theory, all the weights of Critic-Actor neural network and the states of the system are guaranteed to be locally uniformly ultimately bounded. The simulation results show the effectiveness of the developed method.

Benefitting from the technology of integral reinforcement learning, the nonzero sum (NZS) game for distributed parameter systems is effectively solved in this paper when the information of system dynamics are unavailable. The Karhunen-Loève decomposition (KLD) is employed to convert the partial differential equation (PDE) systems into high-order ordinary differential equation (ODE) systems. Moreover, the off-policy IRL technology is introduced to design the optimal strategies for the NZS game. To confirm that the presented algorithm will converge to the optimal value functions, the traditional adaptive dynamic programming (ADP) method is first discussed. Then, the equivalence between the traditional ADP method and the presented off-policy method is proved. For implementing the presented off-policy IRL method, actor and critic neural networks are utilized to approach the value functions and control strategies in the iteration process, individually. Finally, a numerical simulation is shown to illustrate the effectiveness of the proposal off-policy algorithm.

Optimal Tracking Control Based on Integral Reinforcement Learning for An Underactuated Drone

Dynamic compensator-based near-optimal control for unknown nonaffine systems via integral reinforcement learning

This paper presents a model-free optimal tracking control algorithm for an aircraft skin inspection robot with constrained-input and input time-delay. To tackle the input time-delay problem, the original system is transformed into a delay-free system with constrained-input and unknown input coupling term. In order to overcome the optimal control problem subject to constrained-input, a discounted value function is employed. In general, it is known that the HJB equation does not admit a classical smooth solution. Moreover, since the input coupling term of the delay-free system is unknown, a model-free integral reinforcement learning(IRL) algorithm which only requires the system sampling data generated by arbitrary different control inputs and external disturbances is proposed. The model-free IRL method is implemented on an actor-critic neural network (NN) structure. A system sampling data set is utilized to learn the value function and control policy. Finally, the simulation verifies the effectiveness of the proposed algorithm.

Finite-horizon robust formation-containment control of multi-agent networks with unknown dynamics

In this work, a novel event-based integral reinforcement learning (IRL) adaptive control method is developed to solve the multiplayer non-zero-sum (NZS) games of the nonlinear systems with unknown drift dynamics. By virtue of the IRL algorithm, the system drift dynamics is no more needed in the controller design. Moreover, different from the existing iteration computation methods, this method is online implemented, on which condition the event-triggered control framework can be combined with the IRL algorithm in solving the NZS game problems. In this method, a state-dependent triggering condition is proposed, thus the computation and communication loads are reduced in the control process. Moreover, the uniform ultimate boundedness (UUB) stability of the controlled system and the convergence of the critic weights have also been proved. Finally, a numerical example is provided to demonstrate the effectiveness of our method.

This paper presents the design of a novel <inline-formula> <tex-math notation="LaTeX">$H_{\infty }$ </tex-math></inline-formula>-based control framework for state regulation of continuous-time linear systems with completely unknown dynamics. The proposed method solves the regulation problem with the desired convergence rate and simultaneously seeks to attenuate the adverse effect of disturbance on the system. The <inline-formula> <tex-math notation="LaTeX">$H_{\infty }$ </tex-math></inline-formula> regulation problem assumes a cost function that considers regulation with a guaranteed rate of convergence as well as disturbance attenuation. The problem is then turned into a two-player zero-sum game optimization problem that can be solved by solving the associated algebraic Riccati equation (ARE), which provides a model-based solution. To solve this problem in a model-free way, a novel integral reinforcement learning (IRL) algorithm is designed to learn the solution online without requiring any prior knowledge of the system dynamics. It is shown that the model-free method (i.e., IRL-based method) provides the same solution as the model-based method (i.e., ARE). The effectiveness of the proposed method is ascertained through simulation examples; it is shown that the proposed method effectively addresses the problem for both stable and unstable systems.

Integral reinforcement learning and experience replay for adaptive optimal control of partially-unknown constrained-input continuous-time systems

In this paper, the decentralized zero-sum differential games for nonlinear interconnected systems are investigated based on off-policy integral reinforcement learning algorithm. First, the decentralized zero-sum differential games of the overall system can be transformed into seeking the Nash equilibrium solution of each isolated subsystem. In order to obtain the Nash equilibrium solution, the off-policy integral reinforcement learning algorithm is introduced to solve the related HamiltonJacobi-Isaacs equations. The system dynamics can be completely unknown when implementing this proposed algorithm. In addition, three neural networks are constructed to approximate the optimal cost function, the optimal control policy and the worst disturbance policy, respectively. Finally, the validity of the proposed algorithm is verified through simulation results.

Model-free policy iteration approach to NCE-based strategy design for linear quadratic Gaussian games

Robust Optimal Control Based on the Off-Policy Integral Reinforcement Learning Algorithm for Surface Vessel Systems with Unknown Dynamics

Model-free Adaptive Dynamic Programming for Optimal Control of Discrete-time Affine Nonlinear System

This paper is concentrated on aperiodic sampling and adaptive learning in continuous-time nonlinear systems. The objective is to find the optimal control policy by minimizing an infinite-quadratic cost function in an online fashion. An algorithm based on integral reinforcement learning is proposed for problems with unknown model information, and a novel dynamic triggering method is combined for saving communication as much as possible. The identifying neural network is utilized to free the input dynamics and critic neural network is constructed to approximate the cost function. The weight convergence and asymptotic stability are theoretically elaborated, and afterwards, a numerical example is presented for illustrating the effectiveness and superiority of the algorithm by comparison.