Model-free optimal controller design for continuous-time nonlinear systems by adaptive dynamic programming based on a precompensator

In this paper, an on-line adaptive dynamic programming (ADP) algorithm is presented to solve the optimal control problem for continuous-time nonlinear systems with control constraints. First, a suitable non-quadratic performance function is introduced to confront control constraints, and then we present an on-line ADP algorithm in which dynamic programming techniques and neural networks are used to solve the optimal control problem for nonlinear systems with control constraints, which is completely different with the traditional ADP algorithm using off-line training, and the corresponding convergence proof is given. Finally, a simulation examples is given to demonstrate the convergence and feasibility of the proposed algorithm.

This paper presents a novel method of global adaptive dynamic programming (ADP) for the adaptive optimal control of nonlinear polynomial systems. The strategy consists of relaxing the problem of solving the Hamilton-Jacobi-Bellman (HJB) equation to an optimization problem, which is solved via a new policy iteration method. The proposed method distinguishes from previously known nonlinear ADP methods in that the neural network approximation is avoided, giving rise to significant computational improvement. Instead of semiglobally or locally stabilizing, the resultant control policy is globally stabilizing for a general class of nonlinear polynomial systems. Furthermore, in the absence of the a priori knowledge of the system dynamics, an online learning method is devised to implement the proposed policy iteration technique by generalizing the current ADP theory. Finally, three numerical examples are provided to validate the effectiveness of the proposed method.

To intercept a maneuvering target with a predetermined impact angle, a computational intelligence guidance law was proposed in this paper. Based on the theory of two-player zero-sum differential games, this problem is resolved efficiently by solving the Hamilton–Jacobi–Isaacs (HJI) equation. The Nash equilibrium solution of HJI equation can be solved with a policy iteration (PI) algorithm. Instead of using the offline PI algorithm, an online PI algorithm is introduced, in which the disturbance and control policies can be updated simultaneously. It can be proved that the online PI algorithm is a replacement for Newton’s iterative algorithm, the convergence of which is ensured by Kantorovich’s theorem. In the scenario of missiles intercepting targets, an adaptive critic structure based on a neural network (NN) is proposed to implement the online PI algorithm. Only one critic NN approximator is used in the PI algorithm to calculate a value function and the approximate Nash equilibrium solution. It is not necessary to acquire the exact internal dynamics information of nonlinear systems on the basis of online data sampling. The effectiveness of the computational intelligence guidance law is proven by simulation results.

A novel policy iteration algorithm, called the continuous-time time-varying (CTTV) policy iteration algorithm, is presented in this paper to obtain the optimal control laws for infinite horizon CTTV nonlinear systems. The adaptive dynamic programming (ADP) technique is utilized to obtain the iterative control laws for the optimization of the performance index function. The properties of the CTTV policy iteration algorithm are analyzed. Monotonicity, convergence, and optimality of the iterative value function have been analyzed, and the iterative value function can be proven to monotonically converge to the optimal solution of the Hamilton-Jacobi-Bellman (HJB) equation. Furthermore, the iterative control law is guaranteed to be admissible to stabilize the nonlinear systems. In the implementation of the presented CTTV policy algorithm, the approximate iterative control laws and iterative value function are obtained by neural networks. Finally, the numerical results are given to verify the effectiveness of the presented method.

This paper investigates a novel fault tolerant control (FTC) scheme for reconfigurable manipulators with actuator failures based on adaptive dynamic programming (ADP) with an improved performance index function. With the help of fault observer, the estimated actuator failure is utilized to construct the performance index function that reflects the failure, regulation and control simultaneously. The fault tolerant controller consists of the desired controller and the approximated optimal feedback one. The desired controller is obtained by the system model directly. By establishing a critic neural network, the policy iteration (PI) algorithm is utilized to solve the Hamilton-Jacobi-Bellman (HJB) equation, and then the approximated optimal feedback controller can be derived directly. It is proven that the closed-loop system is uniformly ultimately bounded (UUB) based on Lyapunov stability theorem. Finally, the simulation study is given to show the effectiveness of the proposed FTC scheme.

In this paper, a model-free integral sliding mode control method is presented via adaptive dynamic programming (ADP) to address the trajectory tracking problem of reconfigurable manipulators. The integral sliding mode control method is adopted to deal with the model uncertainties of joint subsystems and a local neural network identifier is designed to compensate the effect of the interconnection dynamic couplings (IDCs) between the subsystems. Based on the ADP and the policy iteration (PI) algorithm, the Hamilton-Jacobi-Bellman (HJB) equation can be solved by using critic neural network and then we can get the optimal control strategy. Based on the Lyapunov theory, the closed-loop manipulator system is proved to be asymptotic stability. Finally, simulation results are demonstrated the effectiveness of the proposed method.

Robust tracking control of uncertain nonlinear systems with adaptive dynamic programming

In this chapter, by using neural network (NN)-based online learning optimal control approach, a decentralized control strategy is developed to stabilize a class of continuous-time large-scale interconnected nonlinear systems . It is proven that the decentralized control strategy of the overall system can be established by adding appropriate feedback gains to the optimal control laws of the isolated subsystems . Then, an online policy iteration (PI) algorithm is developed to solve the Hamilton–Jacobi–Bellman equations related to the optimal control problem. By constructing a set of critic NNs, the cost functions can be obtained by NN approximation, followed by the control laws. Furthermore, as a generalization, an NN-based decentralized control law is developed to stabilize the large-scale interconnected nonlinear systems using an online model-free integral PI algorithm. The model-free PI approach can solve the decentralized control problem for large-scale interconnected nonlinear systems with unknown dynamics. Finally, two simulation examples are provided to illustrate the effectiveness of the present decentralized control scheme.

Optimal [formula omitted] tracking control of nonlinear systems with zero-equilibrium-free via novel adaptive critic designs

The proposed event-triggered control design uses the adaptive dynamic programming (ADP) technique to solve the infinite-horizon optimal control of nonlinear continuous time system in affine form with complete unknown system dynamics in a forward time and online manner. The approximation property of the neural network (NN) is used to estimate the system dynamics and the value function with event-based sampling of state vector. Subsequently the estimated values are used to design the near optimal control policy. In addition, the NN weights are updated as a jump at every trigger instant, hence aperiodic in nature, to save computation when compared to the traditional NN-based approaches. Further, the closed-loop dynamics are formulated as a nonlinear impulsive dynamical system and the extension of the Lyapunov technique is utilized to prove the locally ultimate boundedness of all the closed-loop signals by deriving an adaptive event-trigger condition. Nonetheless, a positive lower bound on the inter-event time is guaranteed to avoid accumulation point. Finally, the analytical design is evaluated by using an example.

In this chapter, differential games are studied for continuous-time linear and nonlinear systems, including two-player zero-sum games, multi-player zero-sum games, and multi-player nonzero-sum games, via a series of adaptive dynamic programming (ADP) approaches. First, an integral policy iteration algorithm is developed to learn online the Nash equilibrium of two-player zero-sum differential games with completely unknown continuous-time linear dynamics using the state and control data. Second, multi-player zero-sum differential games for a class of continuous-time uncertain nonlinear systems are solved by using a novel iterative ADP algorithm. Via neural network modeling for the system dynamics, the ADP technique is employed to obtain the optimal control pair iteratively so that the iterative value function reaches the optimal solution of the zero-sum differential games. Finally, an online synchronous approximate optimal learning algorithm based on policy iteration is developed to solve multi-player nonzero-sum games of continuous-time nonlinear systems without the requirement of exact knowledge of system dynamics.

An adaptive dynamic programming (ADP)‐based fault‐tolerant control (FTC) scheme integrating a sliding mode surface (SMS) and an event‐triggered mechanism is developed for a class of nonlinear systems with unknown actuator faults. A fault estimation observer is constructed, and its outputs are embedded into a formulated performance index, thereby recasting the FTC problem as a tractable optimal control problem. Within the resulting ADP framework, a controller is synthesized that automatically delivers active and optimal fault compensation. In place of conventional actor‐critic neural network (ACNN) architectures, a simplified critic‐only neural network (NN) is adopted, together with policy iteration, to solve the associated Hamilton–Jacobi–Bellman (HJB) equation. This structure substantially reduces computational complexity and enhances online applicability. CNN weights are updated via gradient descent combined with experience replay, which obviates the need for the persistent excitation condition that is often difficult to maintain in real‐time operation. Moreover, the proposed SMS‐based event‐triggering mechanism achieves better communication efficiency and improved closed‐loop performance relative to conventional state‐dependent triggering schemes. Theoretical analysis rigorously establishes the stability and convergence properties of the overall control system, while simulation experiments validate its practical effectiveness.

This paper studies the problem of event-triggered optimal control (ETOC) for continuous-time nonlinear systems and proposes a novel event-triggering condition that enables designing ETOC methods directly based on the solution of the Hamilton-Jacobi-Bellman (HJB) equation. We provide formal performance guarantees by proving a predetermined upper bound. Moreover, we also prove the existence of a lower bound for interexecution time. For implementation purposes, an adaptive dynamic programming (ADP) method is developed to realize the ETOC using a critic neural network (NN) to approximate the value function of the HJB equation. Subsequently, we prove that semiglobal uniform ultimate boundedness can be guaranteed for states and NN weight errors with the ADP-based ETOC. Simulation results demonstrate the effectiveness of the developed ADP-based ETOC method.

In this paper, a novel active optimal fault-tolerant control (FTC) scheme is designed based on adaptive dynamic programming (ADP) for modular manipulator when sensor and actuator faults are concurrency. Firstly, the sensor fault is transformed into the pseudo-actuator fault by constructing a nonlinear transformation with diffeomorphism theory. Secondly, the faults estimated by the adaptive fault observer are applied to establish an improved performance index function. Next, the online policy iteration (PI) algorithm is used to solve the Hamilton-Jacobi-Bellman (HJB) equation via establishing a critic neural network. The optimal fault-tolerant controller is proved to be uniformly ultimately bounded (UUB) based on Lyapunov stable theory. Finally, the effectiveness of the proposed multi-fault-tolerant control algorithm is verified by simulation results.

An online adaptive optimal control is proposed for continuous-time nonlinear systems with completely unknown dynamics, which is achieved by developing a novel identifier-critic-based approximate dynamic programming algorithm with a dual neural network (NN) approximation structure. First, an adaptive NN identifier is designed to obviate the requirement of complete knowledge of system dynamics, and a critic NN is employed to approximate the optimal value function. Then, the optimal control law is computed based on the information from the identifier NN and the critic NN, so that the actor NN is not needed. In particular, a novel adaptive law design method with the parameter estimation error is proposed to online update the weights of both identifier NN and critic NN simultaneously, which converge to small neighbourhoods around their ideal values. The closed-loop system stability and the convergence to small vicinity around the optimal solution are all proved by means of the Lyapunov theory. The proposed adaptation algorithm is also improved to achieve finite-time convergence of the NN weights. Finally, simulation results are provided to exemplify the efficacy of the proposed methods.