Dynamic compensator-based near-optimal control for unknown nonaffine systems via integral reinforcement learning

This paper focuses on the optimal containment control problem for the nonlinear multiagent systems with partially unknown dynamics via an integral reinforcement learning algorithm. By employing integral reinforcement learning, the requirement of the drift dynamics is relaxed. The integral reinforcement learning method is proved to be equivalent to the model-based policy iteration, which guarantees the convergence of the proposed control algorithm. For each follower, the Hamilton-Jacobi-Bellman equation is solved by a single critic neural network with a modified updating law which guarantees the weight error dynamic to be asymptotically stable. Through using input-output data, the approximate optimal containment control protocol of each follower is obtained by applying the critic neural network. The closed-loop containment error system is guaranteed to be stable under the proposed optimal containment control scheme. Simulation results demonstrate the effectiveness of the presented control scheme.

Optimal Tracking Control Based on Integral Reinforcement Learning for An Underactuated Drone

In order to obviate the requirement of drift dynamics in adaptive dynamic programming, integral reinforcement learning (IRL) has been proposed as an alternate formulation of Bellman equation. However control coupling dynamics is still needed to obtain closed-form expression of optimal control effort. In addition to this, initial stabilizing controller and two sets of neural networks (NN) (known as Actor-Critic) are required to implement IRL scheme. In order to remedy these, this paper presents a critic-only IRL controller coupled with an experience replay (ER)-based identifier to solve optimal tracking control problem for an unknown continuous-time systems under actuator constraints. The presented control architecture is shown to yield tighter residual sets for state tracking error and error in NN weights. The simulation results establish the efficacy of the presented control scheme on continuous-time systems.

This paper presents a model-free optimal tracking control algorithm for an aircraft skin inspection robot with constrained-input and input time-delay. To tackle the input time-delay problem, the original system is transformed into a delay-free system with constrained-input and unknown input coupling term. In order to overcome the optimal control problem subject to constrained-input, a discounted value function is employed. In general, it is known that the HJB equation does not admit a classical smooth solution. Moreover, since the input coupling term of the delay-free system is unknown, a model-free integral reinforcement learning(IRL) algorithm which only requires the system sampling data generated by arbitrary different control inputs and external disturbances is proposed. The model-free IRL method is implemented on an actor-critic neural network (NN) structure. A system sampling data set is utilized to learn the value function and control policy. Finally, the simulation verifies the effectiveness of the proposed algorithm.

This paper aims to find the optimal control solution of an affine nonlinear continuous-time system with unknown input dynamic. Based on Critic-Actor neural network, an online integral reinforcement learning algorithm has been proposed. The algorithm solves the Bellman equation online, while Critic neural network is used to approximate the value function and Actor neural network is used for policy improvement. The policy evaluation and policy improvement of integral reinforcement learning are performed alternately until the performance of control systems no longer improves. By using Lyapunov function theory, all the weights of Critic-Actor neural network and the states of the system are guaranteed to be locally uniformly ultimately bounded. The simulation results show the effectiveness of the developed method.

An ADDHP-based Q-learning algorithm for optimal tracking control of linear discrete-time systems with unknown dynamics

In this paper, a comparison is addressed between two methods, that is, the optimal tracking control methods of unknown nonlinear systems with and without constrained input. Firstly, the optimal tracking problem for a class of affine nonlinear system is formulated. The tracking cost functions are also defined, both for the two methods. The two methods are proved to be equivalent as the actuator bound is large enough. Integral reinforcement learning (IRL) algorithm is employed to solve the optimal tracking problem by using only system data. To facilitate the implementation of the IRL algorithm, the actor-critic neural network technique and the least squares method are employed in approximating the unknown weights iteratively. In the simulation, a detailed comparison is given to demonstrate the relationship between the two methods in the aspects of control input and tracking cost value.

Integral reinforcement learning and experience replay for adaptive optimal control of partially-unknown constrained-input continuous-time systems

Benefitting from the technology of integral reinforcement learning, the nonzero sum (NZS) game for distributed parameter systems is effectively solved in this paper when the information of system dynamics are unavailable. The Karhunen-Loève decomposition (KLD) is employed to convert the partial differential equation (PDE) systems into high-order ordinary differential equation (ODE) systems. Moreover, the off-policy IRL technology is introduced to design the optimal strategies for the NZS game. To confirm that the presented algorithm will converge to the optimal value functions, the traditional adaptive dynamic programming (ADP) method is first discussed. Then, the equivalence between the traditional ADP method and the presented off-policy method is proved. For implementing the presented off-policy IRL method, actor and critic neural networks are utilized to approach the value functions and control strategies in the iteration process, individually. Finally, a numerical simulation is shown to illustrate the effectiveness of the proposal off-policy algorithm.

In this paper, we develop an integral reinforcement learning algorithm based on policy iteration to learn online the Nash equilibrium solution for a two-player zero-sum differential game with completely unknown linear continuous-time dynamics. This algorithm is a fully model-free method solving the game algebraic Riccati equation forward in time. The developed algorithm updates value function, control and disturbance policies simultaneously. The convergence of the algorithm is demonstrated to be equivalent to Newton's method. To implement this algorithm, one critic network and two action networks are used to approximate the game value function, control and disturbance policies, respectively, and the least squares method is used to estimate the unknown parameters. The effectiveness of the developed scheme is demonstrated in the simulation by designing an H <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> state feedback controller for a power system.

This paper is a note on the stability and robustness of the Nash Game (Vamvoudakis et al. in Adaptive optimal control algorithm for zero-sum nash games with integral reinforcement learning, 2012; Ning et al. in Optim Control Appl Methods. doi:10.1002/oca.2042, 2012; Bouyer et al. in Concurrent games with ordered objectives, 2012) based mixed \({\mathcal {H}}_{2}/{\mathcal {H}}_{\infty }\) Model Predictive Controllers (Aadaleesan and Saha in Mixed \(\mathcal {H}_{2}/\mathcal {H}_{\infty }\) Model Predictive Control for Unstable and Non-Minimum Constrained Processes, 2008; Aadaleesan in Nash Game based Mixed \(\mathcal {H}_{2}/\mathcal {H}_{\infty }\) Model Predictive Control applied with Laguerre-Wavelet Network Model, 2011) for linear state feedback systems addressed in Part 1 (Aadaleesan and Saha in Int J Dyn Control, 2016) of this series. The mixed \({\mathcal {H}}_{2}/{\mathcal {H}}_{\infty }\) MPC proposed in Part 1 (Aadaleesan and Saha in Int J Dyn Control, 2016) and that developed by Orukpe et al. (Model predictive control based on mixed \(\mathcal {H}_2/\mathcal {H}_{\infty }\) control approach, 2007) are compared in this Part 2. The issues of stability and robustness of the multi-criterion optimal control are dealt in this paper using set theoretic concepts.

In this work, a novel event-based integral reinforcement learning (IRL) adaptive control method is developed to solve the multiplayer non-zero-sum (NZS) games of the nonlinear systems with unknown drift dynamics. By virtue of the IRL algorithm, the system drift dynamics is no more needed in the controller design. Moreover, different from the existing iteration computation methods, this method is online implemented, on which condition the event-triggered control framework can be combined with the IRL algorithm in solving the NZS game problems. In this method, a state-dependent triggering condition is proposed, thus the computation and communication loads are reduced in the control process. Moreover, the uniform ultimate boundedness (UUB) stability of the controlled system and the convergence of the critic weights have also been proved. Finally, a numerical example is provided to demonstrate the effectiveness of our method.

Aggregated electrical vehicles (EVs) can flexibly serve grid frequency regulation (FR) with adjustable FR capacity (FRC) via vehicle-to-grid (V2G) technology. Therefore, current research applies integral reinforcement learning (IRL) to FR as it can easily solve difficult modeling and optimization problems. However, communication between EVs is vulnerable to denial of service (DoS) attacks, which can significantly degrade FR performance and even destabilize V2G FR systems. This paper proposes an adaptive V2G FRC-based FR scheme with IRL to improve FR resilience and mitigate FR performance degradation against DoS attacks. The proposed scheme optimizes V2G control by IRL without analytical models, integrating an event-triggered mechanism to reduce communication burden. It adapts V2G FRC to attack intensity to minimize frequency deviation and mitigate the impact of DoS attacks. The analytical estimation of convergence rate establishes the quantitative relationships among control performance, V2G FRC, and attack intensity, thereby deriving an adaptive mechanism. The theoretical analysis of the proposed scheme yields stability conditions that guarantee the FR performance. The simulations were performed on the IEEE 39-bus test system to verify the effectiveness and advantages of the proposed scheme. The results indicate that V2G FRC should be scaled down to mitigate performance degradation when intensive attacks occur.

This paper is concentrated on aperiodic sampling and adaptive learning in continuous-time nonlinear systems. The objective is to find the optimal control policy by minimizing an infinite-quadratic cost function in an online fashion. An algorithm based on integral reinforcement learning is proposed for problems with unknown model information, and a novel dynamic triggering method is combined for saving communication as much as possible. The identifying neural network is utilized to free the input dynamics and critic neural network is constructed to approximate the cost function. The weight convergence and asymptotic stability are theoretically elaborated, and afterwards, a numerical example is presented for illustrating the effectiveness and superiority of the algorithm by comparison.

Online lifelong optimal tracking control of uncertain nonlinear continuous-time strict-feedback systems using deep neural networks.