Integral Reinforcement-Learning-Based Optimal Containment Control for Partially Unknown Nonlinear Multiagent Systems.

Optimal Tracking Control Based on Integral Reinforcement Learning for An Underactuated Drone

This paper aims to find the optimal control solution of an affine nonlinear continuous-time system with unknown input dynamic. Based on Critic-Actor neural network, an online integral reinforcement learning algorithm has been proposed. The algorithm solves the Bellman equation online, while Critic neural network is used to approximate the value function and Actor neural network is used for policy improvement. The policy evaluation and policy improvement of integral reinforcement learning are performed alternately until the performance of control systems no longer improves. By using Lyapunov function theory, all the weights of Critic-Actor neural network and the states of the system are guaranteed to be locally uniformly ultimately bounded. The simulation results show the effectiveness of the developed method.

This paper presents a model-free optimal tracking control algorithm for an aircraft skin inspection robot with constrained-input and input time-delay. To tackle the input time-delay problem, the original system is transformed into a delay-free system with constrained-input and unknown input coupling term. In order to overcome the optimal control problem subject to constrained-input, a discounted value function is employed. In general, it is known that the HJB equation does not admit a classical smooth solution. Moreover, since the input coupling term of the delay-free system is unknown, a model-free integral reinforcement learning(IRL) algorithm which only requires the system sampling data generated by arbitrary different control inputs and external disturbances is proposed. The model-free IRL method is implemented on an actor-critic neural network (NN) structure. A system sampling data set is utilized to learn the value function and control policy. Finally, the simulation verifies the effectiveness of the proposed algorithm.

Dynamic compensator-based near-optimal control for unknown nonaffine systems via integral reinforcement learning

Benefitting from the technology of integral reinforcement learning, the nonzero sum (NZS) game for distributed parameter systems is effectively solved in this paper when the information of system dynamics are unavailable. The Karhunen-Loève decomposition (KLD) is employed to convert the partial differential equation (PDE) systems into high-order ordinary differential equation (ODE) systems. Moreover, the off-policy IRL technology is introduced to design the optimal strategies for the NZS game. To confirm that the presented algorithm will converge to the optimal value functions, the traditional adaptive dynamic programming (ADP) method is first discussed. Then, the equivalence between the traditional ADP method and the presented off-policy method is proved. For implementing the presented off-policy IRL method, actor and critic neural networks are utilized to approach the value functions and control strategies in the iteration process, individually. Finally, a numerical simulation is shown to illustrate the effectiveness of the proposal off-policy algorithm.

In this paper, a comparison is addressed between two methods, that is, the optimal tracking control methods of unknown nonlinear systems with and without constrained input. Firstly, the optimal tracking problem for a class of affine nonlinear system is formulated. The tracking cost functions are also defined, both for the two methods. The two methods are proved to be equivalent as the actuator bound is large enough. Integral reinforcement learning (IRL) algorithm is employed to solve the optimal tracking problem by using only system data. To facilitate the implementation of the IRL algorithm, the actor-critic neural network technique and the least squares method are employed in approximating the unknown weights iteratively. In the simulation, a detailed comparison is given to demonstrate the relationship between the two methods in the aspects of control input and tracking cost value.

In this work, a novel event-based integral reinforcement learning (IRL) adaptive control method is developed to solve the multiplayer non-zero-sum (NZS) games of the nonlinear systems with unknown drift dynamics. By virtue of the IRL algorithm, the system drift dynamics is no more needed in the controller design. Moreover, different from the existing iteration computation methods, this method is online implemented, on which condition the event-triggered control framework can be combined with the IRL algorithm in solving the NZS game problems. In this method, a state-dependent triggering condition is proposed, thus the computation and communication loads are reduced in the control process. Moreover, the uniform ultimate boundedness (UUB) stability of the controlled system and the convergence of the critic weights have also been proved. Finally, a numerical example is provided to demonstrate the effectiveness of our method.

This article addresses the optimal tracking control problem with prescribed performance for uncertain nonlinear systems subject to input constraint and unknown disturbances. First, a fixed‐time monotonic convergence function is introduced to restrain tracking error, and a nonlinear mapping technique is employed to transform the constrained error into an unconstrained variable, then the fixed‐time output tracking issue is boiled down to the boundedness problem of the transformed variable. With the aid of a nonquadratic cost function, the input constraint is encoded into the optimization problem. To solve the unknown disturbances, an auxiliary system and an auxiliary disturbance policy are constructed, and the optimal control problem is formulated as a two‐player zero‐sum game. Moreover, a Hamilton–Jacobi–Isaacs (HJI) equation associated with this nonquadratic zero‐sum game is established to give the optimal control and the worst‐case disturbance policy solution. Subsequently, to avoid using knowledge of the system dynamics, three neural network approximators, namely, actor, critic, and disturbance, which are tuned online and simultaneously for approximating the solution of HJI, are constructed based on the integral reinforcement learning algorithm. Theoretical analysis shows that the reconstructed error system states and the weight estimation errors are semi‐globally uniformly ultimately bounded. Finally, the simulation study further tests the availability of the proposed control strategy.

This paper develops an integral reinforcement learning (IRL) controller for a class of high-order multivariable nonlinear systems with unknown control coefficients (UCCs). A new long-term performance index is first presented, and then the critic neural network (NN) and the action NN are presented to estimate the unobtainable long-term performance index and the unknown drift of systems, respectively. By combining the critic and action NNs with Nussbaum-type functions, the IRL controllers for high-order, nonsquare multivariable systems are proposed to cope with the problem of UCCs. The analysis are given to illustrate that the stability of the closed-loop system can be obtained, and the signals of the closed-loop systems are semiglobally uniformly ultimately bounded (UUB). Finally, one simulation example is provided to show the effectiveness of the proposed IRL controllers.

In this paper, we develop an integral reinforcement learning algorithm based on policy iteration to learn online the Nash equilibrium solution for a two-player zero-sum differential game with completely unknown linear continuous-time dynamics. This algorithm is a fully model-free method solving the game algebraic Riccati equation forward in time. The developed algorithm updates value function, control and disturbance policies simultaneously. The convergence of the algorithm is demonstrated to be equivalent to Newton's method. To implement this algorithm, one critic network and two action networks are used to approximate the game value function, control and disturbance policies, respectively, and the least squares method is used to estimate the unknown parameters. The effectiveness of the developed scheme is demonstrated in the simulation by designing an H <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> state feedback controller for a power system.

Integral reinforcement learning and experience replay for adaptive optimal control of partially-unknown constrained-input continuous-time systems

This paper is concentrated on aperiodic sampling and adaptive learning in continuous-time nonlinear systems. The objective is to find the optimal control policy by minimizing an infinite-quadratic cost function in an online fashion. An algorithm based on integral reinforcement learning is proposed for problems with unknown model information, and a novel dynamic triggering method is combined for saving communication as much as possible. The identifying neural network is utilized to free the input dynamics and critic neural network is constructed to approximate the cost function. The weight convergence and asymptotic stability are theoretically elaborated, and afterwards, a numerical example is presented for illustrating the effectiveness and superiority of the algorithm by comparison.

This article investigates the adaptive neural network (NN) event-triggered containment control problem for a class of nonlinear multiagent systems (MASs). Since the considered nonlinear MASs contain unknown nonlinear dynamics, immeasurable states, and quantized input signals, the NNs are adopted to model unknown agents, and an NN state observer is established by using the intermittent output signal. Subsequently, a novel event-triggered mechanism consisting of both the sensor-to-controller and controller-to-actuator channels are established. By decomposing quantized input signals into the sum of two bounded nonlinear functions and based on the adaptive backstepping control and first-order filter design theories, an adaptive NN event-triggered output-feedback containment control scheme is formulated. It is proved that the controlled system is semi-globally uniformly ultimately bounded (SGUUB) and the followers are within a convex hull formed by the leaders. Finally, a simulation example is given to validate the effectiveness of the presented NN containment control scheme.

In order to obviate the requirement of drift dynamics in adaptive dynamic programming, integral reinforcement learning (IRL) has been proposed as an alternate formulation of Bellman equation. However control coupling dynamics is still needed to obtain closed-form expression of optimal control effort. In addition to this, initial stabilizing controller and two sets of neural networks (NN) (known as Actor-Critic) are required to implement IRL scheme. In order to remedy these, this paper presents a critic-only IRL controller coupled with an experience replay (ER)-based identifier to solve optimal tracking control problem for an unknown continuous-time systems under actuator constraints. The presented control architecture is shown to yield tighter residual sets for state tracking error and error in NN weights. The simulation results establish the efficacy of the presented control scheme on continuous-time systems.

Cooperative Robust H-∞ Output Consensus in Continuous-Time Heterogeneous Multi-Agent Systems Using Integral Reinforcement Learning Method