Adaptive Dynamic Programming for a Nonlinear Two‐Player Non‐Zero‐Sum Differential Game With State and Input Constraints

In this paper, the zero-sum differential game problem for a class of nonlinear system with input constraints is investigated via adaptive dynamic programming (ADP). A suitable non-quadratic functional is utilized to embed the control constraints into the differential game problem. Then, the Nash equilibrium solution is found by solving the constrained Hamilton-Jacobi-Isaacs (HJI) equation. The single critic network is constructed to approximate the solution of associated HJI equation online. A robustifying control term is added to the controller to eliminate the effect of residual error, leading to the asymptotically stability of the closed-loop system. Simulation results verify the effectiveness of proposed method by using a simple nonlinear system.

In this paper, the distributed zero-sum differential game problem for multi-agent nonlinear systems is investigated by using adaptive dynamic programming (ADP) technique, which brings together the distributed control, zero-sum differential graphical game and ADP. To find the cooperative Nash-equilibrium solutions of the associated coupled Hamilton-Jacobi-Isaacs (HJI) equations, the ADP technique is employed, in which a critic network is constructed to approximate the cooperative cost function online with a novel updating law. By utilizing the Lyapunov direct method, the closed-loop system and weight estimation errors are guaranteed to be uniformly ultimately bounded (UUB). Finally, simulation results demonstrate the effectiveness of the proposed method.

This paper studies the problem of optimal parallel tracking control for continuous-time general nonlinear systems. Unlike existing optimal state feedback control, the control input of the optimal parallel control is introduced into the feedback system. However, due to the introduction of control input into the feedback system, the optimal state feedback control methods can not be applied directly. To address this problem, an augmented system and an augmented performance index function are proposed firstly. Thus, the general nonlinear system is transformed into an affine nonlinear system. The difference between the optimal parallel control and the optimal state feedback control is analyzed theoretically. It is proven that the optimal parallel control with the augmented performance index function can be seen as the suboptimal state feedback control with the traditional performance index function. Moreover, an adaptive dynamic programming ( ADP ) technique is utilized to implement the optimal parallel tracking control using a critic neural network ( NN ) to approximate the value function online. The stability analysis of the closed-loop system is performed using the Lyapunov theory, and the tracking error and NN weights errors are uniformly ultimately bounded ( UUB ) . Also, the optimal parallel controller guarantees the continuity of the control input under the circumstance that there are finite jump discontinuities in the reference signals. Finally, the effectiveness of the developed optimal parallel control method is verified in two cases.

This paper studies the problem of event-triggered optimal control (ETOC) for continuous-time nonlinear systems and proposes a novel event-triggering condition that enables designing ETOC methods directly based on the solution of the Hamilton-Jacobi-Bellman (HJB) equation. We provide formal performance guarantees by proving a predetermined upper bound. Moreover, we also prove the existence of a lower bound for interexecution time. For implementation purposes, an adaptive dynamic programming (ADP) method is developed to realize the ETOC using a critic neural network (NN) to approximate the value function of the HJB equation. Subsequently, we prove that semiglobal uniform ultimate boundedness can be guaranteed for states and NN weight errors with the ADP-based ETOC. Simulation results demonstrate the effectiveness of the developed ADP-based ETOC method.

Benefitting from the technology of integral reinforcement learning, the nonzero sum (NZS) game for distributed parameter systems is effectively solved in this paper when the information of system dynamics are unavailable. The Karhunen-Loève decomposition (KLD) is employed to convert the partial differential equation (PDE) systems into high-order ordinary differential equation (ODE) systems. Moreover, the off-policy IRL technology is introduced to design the optimal strategies for the NZS game. To confirm that the presented algorithm will converge to the optimal value functions, the traditional adaptive dynamic programming (ADP) method is first discussed. Then, the equivalence between the traditional ADP method and the presented off-policy method is proved. For implementing the presented off-policy IRL method, actor and critic neural networks are utilized to approach the value functions and control strategies in the iteration process, individually. Finally, a numerical simulation is shown to illustrate the effectiveness of the proposal off-policy algorithm.

In this study, the problem of intercepting a manoeuvring target is posed in a zero‐sum differential game problem for a class of strict‐feedback non‐linear systems with output and input constraints. By introducing a barrier Lyapunov function and an auxiliary system to deal with the output constraints and input constraints, respectively, a novel backstepping feedforward controller is designed to transform the tracking problem for strict‐feedback systems into an equivalence differential game problem for affine systems. Subsequently, a zero‐sum differential game strategy is developed by using the adaptive dynamic programming technique. A critic network is constructed to learn the Nash equilibrium of the Hamilton–Jacobi–Isaacs equation online. The convergence properties of the proposed backstepping‐based differential games are developed by utilising the Lyapunov method. Finally, the effectiveness of the proposed strategy is demonstrated by simulation using missile‐target interception system.

ABSTRACTThis article has investigated the problem of data‐driven distributed optimal consensus control for discrete‐time multiagent systems (MASs) with unknown dynamics. A novel event‐triggered goal representation heuristic dynamic programming (GrHDP)‐based control approach has been proposed. Note that traditional adaptive dynamic programming (ADP) methods usually establish two neural networks (NNs), the critic‐actor NNs, to approximate the performance index function and control inputs, respectively. In comparison, in this work, a goal NN has been used to generate a new local internal reinforcement signal (IRS) that contains more information from the environment, and then, a three neural‐networks structure, namely the goal‐critic‐actor NNs, has been established for each agent. Furthermore, to alleviate the communication load and computation cost, the event‐triggered mechanism has been constructed to update the control input aperiodically. Using the Lyapunov stability theory, detailed stability analysis results for the closed‐loop MASs with event‐triggered controller have been shown for both cases of the triggering and non‐triggering instants. Meanwhile, the weight estimation errors of the goal‐critic‐actor NNs and the local neighborhood consensus tracking errors have been guaranteed to be all uniformly ultimately bounded (UUB). Lastly, simulation studies have shown the effectiveness and superiority of our proposed method over existing works.

This article studies the multi-H∞ controls for the input-interference nonlinear systems via adaptive dynamic programming (ADP) method, which allows for multiple inputs to have the individual selfish component of the strategy to resist weighted interference. In this line, the ADP scheme is used to learn the Nash-optimization solutions of the input-interference nonlinear system such that multiple H∞ performance indices can reach the defined Nash equilibrium. First, the input-interference nonlinear system is given and the Nash equilibrium is defined. An adaptive neural network (NN) observer is introduced to identify the input-interference nonlinear dynamics. Then, the critic NNs are used to learn the multiple H∞ performance indices. A novel adaptive law is designed to update the critic NN weights by minimizing the Hamiltonian-Jacobi-Isaacs (HJI) equation, which can be used to directly calculate the multi-H∞ controls effectively by using input-output data such that the actor structure is avoided. Moreover, the control system stability and updated parameter convergence are proved. Finally, two numerical examples are simulated to verify the proposed ADP scheme for the input-interference nonlinear system.

In this article, the optimal control scheme for ice-storage air conditioning (IAC) system is solved via a data-based adaptive dynamic programming (ADP) method. It is the first time that ADP is employed to design a self-learning scheme, which obtains the optimal control policy of IAC system. First, based on the data of the temperature, irradiance, and cooling load in an actual project, a prediction model of cooling load is built by a three-layer neural network with the performance verification. Second, the operation of the IAC system is analyzed. Third, a data-based ADP method is designed to realize a self-learning optimal control for the IAC system. Then, numerical results show that using the data-based optimal control method can reduce the operation costs. Finally, the comparison results show that the developed ADP method improves the system efficiency, minimizing the overall cost. Thus, the superiority of the developed algorithm is verified.

This paper presents an optimal trajectory tracking control approach for wheeled mobile robots (WMRs) using the adaptive dynamic programming (ADP) method integrated with radial basis function neural network (RBFNN). Firstly, the kine-matic model and error dynamics of the wheeled mobile robots are derived, with the tracking error defined in a transformed coordinate system. Then, the control input is decomposed into feedforward and feedback components to track the reference trajectory. To circumvent the issue of solving the Hamilton-Jacobi-Bellman (HJB) equation, the critic learning approach based on RBFNN is employed to approximate the optimal value function. The modified critic network weight update rule is introduced, which combines stability indicators and concurrent learning terms. The feedback control strategy is formulated by combining the output of critic network with the error dynamics, ensuring convergence of tracking errors to the neighborhood of zero. Finally, the simulation experiments validate the effectiveness of the method.

Major challenges of controlling human-robot collaboration (HRC)-oriented modular robot manipulators (MRMs) include the estimation of human motion intention while cooperating with a robot and performance optimization. This article proposes a cooperative game-based approximate optimal control method of MRMs for HRC tasks. A harmonic drive compliance model-based human motion intention estimation method is developed using robot position measurements only, which forms the basis of the MRM dynamic model. Based on the cooperative differential game strategy, the optimal control problem of HRC-oriented MRM systems is transformed into a cooperative game problem of multiple subsystems. By taking advantage of the adaptive dynamic programming (ADP) algorithm, a joint cost function identifier is developed via the critic neural networks, which is implemented for solving the parametric Hamilton-Jacobi-Bellman (HJB) equation and Pareto optimal solutions. The trajectory tracking error under the HRC task of the closed-loop MRM system is proved to be ultimately uniformly bounded (UUB) by the Lyapunov theory. Finally, experiment results are presented, which reveal the advantage of the proposed method.

An event-based intelligent cooperative control policy is developed in this paper for a class of second-order multiagent systems (MASs). The followers are subject to external disturbances and output constraints, in which the constraint ranges are asymmetric and time-varying. By using one-to-one nonlinear mapping technique to obtain equivalent unconstrained systems and designing a distributed observer to estimate the leader's states, a simplified unconstrained tracking control task is established for each follower. After that, event-driven distributed optimal <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$H_\infty$</tex-math></inline-formula> controllers are designed to mitigate the effects of external disturbances, which are obtained from a set of decoupled Hamilton-Jacobi-Isaacs (HJI) equations, where the proposed event-triggered strategy is dynamic. Aiming at solving the HJI equations more efficiently, an event-based adaptive dynamic programming (ADP) learning algorithm applying only critic neural networks (NNs) is developed. And the critic NN weights are convergent by using stored data under finite excitation condition. Finally, the validity of the developed control scheme is demonstrated by multiple single-link robot arm systems.

This paper is concerned with the problem of active complementary control design for affine nonlinear control systems with actuator faults. The outage and loss of effectiveness fault cases are considered. In order to achieve the performance enhancement of the faulty control system, the complementary control scheme is designed in two steps. Firstly, a novel fault estimation scheme is developed. Then, by using the fault estimations to reconstruct the faulty system dynamics and introducing a cost function as the optimization objective, a nearly optimal complementary control is obtained online based on the adaptive dynamic programming (ADP) method. Unlike most of the previous ADP methods with the addition of a probing signal, new adaptive weight update laws are derived to guarantee the convergence of neural network weights and the stability of the closed-loop system, which strongly supports the online implementation of the ADP method. Finally, two simulation examples are given to illustrate the performance and effectiveness of the proposed method.

This article investigates the problem of path following for the underactuated unmanned surface vehicles (USVs) subject to state constraints. A useful control algorithm is proposed by combining the backstepping technique, adaptive dynamic programming (ADP), and the event-triggered mechanism. The presented approach consists of three modules: guidance law, dynamic controller, and event triggering. First, to deal with the "singularity" problem, the guidance-based path-following (GBPF) principle is introduced in the guidance law loop. In contrast to the traditional barrier Lyapunov function (BLF) method, this article converts the USV's constraint model to a class of nonlinear systems without state constraints by introducing a nonlinear mapping. The control signal generated by the dynamic controller module consists of a backstepping-based feedforward control signal and an ADP-based approximate optimal feedback control signal. Therefore, the presented scheme can guarantee the approximate optimal performance. To approximate the cost function and its partial derivative, a critic neural network (NN) is constructed. By considering the event-triggered condition, the dynamic controller is further improved. Compared with traditional time-triggered control methods, the proposed approach can greatly reduce communication and computational burdens. This article proves that the closed-loop system is stable, and the simulation results and experimental validation are given to illustrate the effectiveness of the proposed approach.

Online event-triggered adaptive critic design for multi-player zero-sum games of partially unknown nonlinear systems with input constraints