Optimal control of Markovian jump systems via a neural network-based ADP iterative algorithm

SummaryThis paper considers optimal consensus control problem for unknown nonlinear multiagent systems (MASs) subjected to control constraints by utilizing event‐triggered adaptive dynamic programming (ETADP) technique. To deal with the control constraints, we introduce nonquadratic energy consumption functions into performance indices and formulate the Hamilton‐Jacobi‐Bellman (HJB) equations. Then, based on the Bellman's optimality principle, constrained optimal consensus control policies are designed from the HJB equations. In order to implement the ETADP algorithm, the critic networks and action networks are developed to approximate the value functions and consensus control policies respectively based on the measurable system data. Under the event‐triggered control framework, the weights of the critic networks and action networks are only updated at the triggering instants which are decided by the designed adaptive triggered conditions. The Lyapunov method is used to prove that the local neighbor consensus errors and the weight estimation errors of the critic networks and action networks are ultimately bounded. Finally, a numerical example is provided to show the effectiveness of the proposed ETADP method.

In this paper, an on-line adaptive dynamic programming (ADP) algorithm is presented to solve the optimal control problem for continuous-time nonlinear systems with control constraints. First, a suitable non-quadratic performance function is introduced to confront control constraints, and then we present an on-line ADP algorithm in which dynamic programming techniques and neural networks are used to solve the optimal control problem for nonlinear systems with control constraints, which is completely different with the traditional ADP algorithm using off-line training, and the corresponding convergence proof is given. Finally, a simulation examples is given to demonstrate the convergence and feasibility of the proposed algorithm.

This paper studies the problem of optimal parallel tracking control for continuous-time general nonlinear systems. Unlike existing optimal state feedback control, the control input of the optimal parallel control is introduced into the feedback system. However, due to the introduction of control input into the feedback system, the optimal state feedback control methods can not be applied directly. To address this problem, an augmented system and an augmented performance index function are proposed firstly. Thus, the general nonlinear system is transformed into an affine nonlinear system. The difference between the optimal parallel control and the optimal state feedback control is analyzed theoretically. It is proven that the optimal parallel control with the augmented performance index function can be seen as the suboptimal state feedback control with the traditional performance index function. Moreover, an adaptive dynamic programming ( ADP ) technique is utilized to implement the optimal parallel tracking control using a critic neural network ( NN ) to approximate the value function online. The stability analysis of the closed-loop system is performed using the Lyapunov theory, and the tracking error and NN weights errors are uniformly ultimately bounded ( UUB ) . Also, the optimal parallel controller guarantees the continuity of the control input under the circumstance that there are finite jump discontinuities in the reference signals. Finally, the effectiveness of the developed optimal parallel control method is verified in two cases.

In this paper, an optimal tracking control approach based on adaptive dynamic programming (ADP) algorithm is proposed to solve the linear quadratic regulation (LQR) problems for unknown discrete-time systems in an online fashion. First, we convert the optimal tracking problem into designing infinite-horizon optimal regulator for the tracking error dynamics based on the system transformation. Then we expand the error state equation by the history data of control and state. The iterative ADP algorithm of policy iteration (PI) and value iteration (VI) are introduced to solve the value function of the controlled system. It is shown that the proposed ADP algorithm solves the LQR without requiring any knowledge of the system dynamics. The simulation results show the convergence and effectiveness of the proposed control scheme.

In this paper, a finite-horizon ε-optimal control for a class of nonlinear systems with state delay is proposed by Adaptive Dynamic Programming (ADP) algorithm. First of all, the performance index function is defined and the Hamilton-Jacobi-Bellman (HJB) equation is obtained for the problem, the convergence of the iterative algorithm is also presented. Then, ADP algorithm for finite-horizon optimal control is introduced with an ε-error bound so as to get the ε-optimal control, and BP neural network is used to implement ADP algorithm. At last, an example is given to demonstrate the effectiveness of the proposed algorithm.

In this paper, an innovative adaptive dynamic programming (ADP) algorithm with fast convergence speed is designed for the optimal containment control problem of discrete‐time linear multi‐agent systems. Precisely, a quadratic input energy cost function, including local containment error information and actuator information in the neighborhood, is designed for each follower. Solving the stationary condition of the cost function, the optimal containment controllers are obtained. Traditional ADP methods use actor–critic neural networks to approximate optimal costs and control strategies, it is time‐consuming to solve large‐scale multi‐agent problems due to the computational complexity of neural networks. In order to seek faster convergence speed of optimal containment control without knowing the model information, the fast ADP algorithm framework is designed, it is proved theoretically that the convergence speed is determined by some configurable parameters, and the whale optimization algorithm is employed to globally optimize the parameters of given spaces to derive the optimal configuration. Finally, numerical simulation results are given to verify the effectiveness of the designed algorithm.

This paper deals with optimal tracking control problems for a class of discrete-time nonlinear systems using a generalized policy iteration adaptive dynamic programming (ADP) algorithm. First, by system transformation, the optimal tracking control problem is transformed into an optimal regulation problem. Then the generalized policy iteration ADP algorithm is employed to obtain the optimal tracking controller with convergence and optimality analysis. The developed algorithm uses the idea of two iteration procedures to obtain the iterative tracking control laws and the iterative value functions. Three neural networks, including model network, critic network and action network, are used to implement the developed algorithm. At last, an simulation example is given to demonstrate the effectiveness of the developed method.

Nearly finite-horizon optimal control for a class of nonaffine time-delay nonlinear systems based on adaptive dynamic programming

SummaryThis paper proposes an optimal control method of an unmanned aerial helicopter (UAH) with unknown disturbances. Solving the Hamilton‐Jacobi‐Bellman (HJB) equation is considered as the common approach to design an optimal controller under a meaningful cost function when facing the nonlinear optimal control problem. However, the HJB equation is hard to solve even for a simple problem. The inverse optimal control method that avoids the difficulties of solving the HJB equation has been adopted. In this inverse optimal control approach, a stabilizing optimal control law and a particular cost function that are obtained by a control Lyapunov function are required. An integrator backstepping method is used in designing the optimal control law of the UAH. Furthermore, a disturbance‐observer–based control (DOBC) approach has been adopted in the optimal control law for dealing with the unknown disturbances of the UAH system. Simulation results have been given to certify the stability of the nonlinear UAH system and the validity of this developed control method.

In this paper, we propose a new approach to address robust control problem of nonlinear systems with uncertainties based on an adaptive dynamic programming (ADP) algorithm. After reformulating the robust control problem as an optimal control problem, we propose a modified ADP method to solve the derived Hamilton-Jacobi-Bellman (HJB) equation, where the optimal cost function is approximated by online training a critic neural network (NN). Then the approximated optimal control action can be derived to guarantee the stability of the controlled system with uncertainties. The closed-loop system stability and convergence have been proved. A simulation example is provided to illustrate the effectiveness of the method.

In this paper, the distributed zero-sum differential game problem for multi-agent nonlinear systems is investigated by using adaptive dynamic programming (ADP) technique, which brings together the distributed control, zero-sum differential graphical game and ADP. To find the cooperative Nash-equilibrium solutions of the associated coupled Hamilton-Jacobi-Isaacs (HJI) equations, the ADP technique is employed, in which a critic network is constructed to approximate the cooperative cost function online with a novel updating law. By utilizing the Lyapunov direct method, the closed-loop system and weight estimation errors are guaranteed to be uniformly ultimately bounded (UUB). Finally, simulation results demonstrate the effectiveness of the proposed method.

In this paper, the direct neural dynamic programming technique is utilized to solve the Hamilton Jacobi-Bellman (HJB) equation online and forward-in-time for the decentralized nearly optimal control of nonlinear interconnected discrete-time systems in affine form with unknown internal subsystem and interconnection dynamics. Only the state vector of the local subsystem is considered measurable. The decentralized optimal controller design for each subsystem consists of an action neural network (NN) that is aimed to provide a nearly optimal control signal, and a critic NN which approximates the cost function. The NN weights are tuned online for both the NNs. It is shown that all subsystems signals are uniformly ultimately bounded (UUB) and that the subsystem inputs approach their corresponding nearly optimal control inputs with bounded error.

This paper proposes an optimal control method based on adaptive dynamic programming (ADP) for manipulators with joint limitation under input saturation. The asymmetric input saturation problem is solved by introducing a smooth function, by which the saturation and nonsmooth issues of the input caused by the actuators can be handled. Joint limitation of the manipulator is transformed through a transfer function, which is incorporated into the cost function of the whole robotic system, to guarantee the joint limitation will not be violated. Meanwhile, due to the difficulty in solving Hamilton-Jacobi-Bellman (HJB) equation, the critic network is constructed utilising radial basis function neural networks (RBFNNs) to approximate the cost function. This approach effectively overcomes the curse of dimensionality problem associated with dynamic programming. Subsequently, an ADP-based optimal controller is designed for the robotic system. Lyapunov synthesis is used in the stability analysis to prove all the signals are convergent and ultimately uniformly bounded (UUB). Simulations are conducted on a planar 3-DOF manipulator to validate the effectiveness of the proposed method. The results demonstrate the improved performance and feasibility of the developed ADP-based optimal control strategy in handling joint limitations and input saturation challenges in robotic systems.

In this paper, a novel value iteration adaptive dynamic programming (ADP) algorithm, called "generalized value iteration ADP" algorithm, is developed to solve infinite horizon optimal tracking control problems for a class of discrete-time nonlinear systems. The developed generalized value iteration ADP algorithm permits an arbitrary positive semi-definite function to initialize it, which overcomes the disadvantage of traditional value iteration algorithms. Convergence property is developed to guarantee that the iterative performance index function will converge to the optimum. Neural networks are used to approximate the iterative performance index function and compute the iterative control policy, respectively, to implement the iterative ADP algorithm. Finally, a simulation example is given to illustrate the performance of the developed algorithm.

A novel policy iteration algorithm, called the continuous-time time-varying (CTTV) policy iteration algorithm, is presented in this paper to obtain the optimal control laws for infinite horizon CTTV nonlinear systems. The adaptive dynamic programming (ADP) technique is utilized to obtain the iterative control laws for the optimization of the performance index function. The properties of the CTTV policy iteration algorithm are analyzed. Monotonicity, convergence, and optimality of the iterative value function have been analyzed, and the iterative value function can be proven to monotonically converge to the optimal solution of the Hamilton-Jacobi-Bellman (HJB) equation. Furthermore, the iterative control law is guaranteed to be admissible to stabilize the nonlinear systems. In the implementation of the presented CTTV policy algorithm, the approximate iterative control laws and iterative value function are obtained by neural networks. Finally, the numerical results are given to verify the effectiveness of the presented method.