Data-Driven Zero-Sum Neuro-Optimal Control for a Class of Continuous-Time Unknown Nonlinear Systems With Disturbance Using ADP.

SummaryThis paper considers optimal consensus control problem for unknown nonlinear multiagent systems (MASs) subjected to control constraints by utilizing event‐triggered adaptive dynamic programming (ETADP) technique. To deal with the control constraints, we introduce nonquadratic energy consumption functions into performance indices and formulate the Hamilton‐Jacobi‐Bellman (HJB) equations. Then, based on the Bellman's optimality principle, constrained optimal consensus control policies are designed from the HJB equations. In order to implement the ETADP algorithm, the critic networks and action networks are developed to approximate the value functions and consensus control policies respectively based on the measurable system data. Under the event‐triggered control framework, the weights of the critic networks and action networks are only updated at the triggering instants which are decided by the designed adaptive triggered conditions. The Lyapunov method is used to prove that the local neighbor consensus errors and the weight estimation errors of the critic networks and action networks are ultimately bounded. Finally, a numerical example is provided to show the effectiveness of the proposed ETADP method.

In this article, the optimal control scheme for ice-storage air conditioning (IAC) system is solved via a data-based adaptive dynamic programming (ADP) method. It is the first time that ADP is employed to design a self-learning scheme, which obtains the optimal control policy of IAC system. First, based on the data of the temperature, irradiance, and cooling load in an actual project, a prediction model of cooling load is built by a three-layer neural network with the performance verification. Second, the operation of the IAC system is analyzed. Third, a data-based ADP method is designed to realize a self-learning optimal control for the IAC system. Then, numerical results show that using the data-based optimal control method can reduce the operation costs. Finally, the comparison results show that the developed ADP method improves the system efficiency, minimizing the overall cost. Thus, the superiority of the developed algorithm is verified.

In the present chapter, intelligent dynamic optimization methods based on adaptive dynamic programming (ADP) are applied to deal with the challenges of intelligent price-responsive management of residential energy, with an emphasis on home battery connected to the power grid. First, an action-dependent heuristic dynamic programming method is developed to obtain the optimal residential energy control law. Second, a dual iterative Q-learning algorithm is developed to solve the optimal battery management and control problem in smart residential environments where two iterations, internal and external iterations, are employed. Based on the dual iterative Q-learning algorithm, the convergence property of iterative Q-learning method for the optimal battery management and control problem is proven. Finally, a distributed iterative ADP method is developed to solve the multi-battery optimal coordination control problems for home energy management systems.

In this paper, an on-line adaptive dynamic programming (ADP) algorithm is presented to solve the optimal control problem for continuous-time nonlinear systems with control constraints. First, a suitable non-quadratic performance function is introduced to confront control constraints, and then we present an on-line ADP algorithm in which dynamic programming techniques and neural networks are used to solve the optimal control problem for nonlinear systems with control constraints, which is completely different with the traditional ADP algorithm using off-line training, and the corresponding convergence proof is given. Finally, a simulation examples is given to demonstrate the convergence and feasibility of the proposed algorithm.

In this paper, a learning-based predictive control (LPC) scheme is proposed for adaptive optimal control of discrete-time nonlinear systems under stochastic disturbances. The proposed LPC scheme is different from conventional model predictive control (MPC), which uses open-loop optimization or simplified closed-loop optimal control techniques in each horizon. In LPC, the control task in each horizon is formulated as a closed-loop nonlinear optimal control problem and a finite-horizon iterative reinforcement learning (RL) algorithm is developed to obtain the closed-loop optimal/suboptimal solutions. Therefore, in LPC, RL and adaptive dynamic programming (ADP) are used as a new class of closed-loop learning-based optimization techniques for nonlinear predictive control with stochastic disturbances. Moreover, LPC also decomposes the infinite-horizon optimal control problem in previous RL and ADP methods into a series of finite horizon problems, so that the computational costs are reduced and the learning efficiency can be improved. Convergence of the finite-horizon iterative RL algorithm in each prediction horizon and the Lyapunov stability of the closed-loop control system are proved. Moreover, by using successive policy updates between adjoint time horizons, LPC also has lower computational costs than conventional MPC which has independent optimization procedures between two different prediction horizons. Simulation results illustrate that compared with conventional nonlinear MPC as well as ADP, the proposed LPC scheme can obtain a better performance both in terms of policy optimality and computational efficiency.

This paper is concerned with the problem of active complementary control design for affine nonlinear control systems with actuator faults. The outage and loss of effectiveness fault cases are considered. In order to achieve the performance enhancement of the faulty control system, the complementary control scheme is designed in two steps. Firstly, a novel fault estimation scheme is developed. Then, by using the fault estimations to reconstruct the faulty system dynamics and introducing a cost function as the optimization objective, a nearly optimal complementary control is obtained online based on the adaptive dynamic programming (ADP) method. Unlike most of the previous ADP methods with the addition of a probing signal, new adaptive weight update laws are derived to guarantee the convergence of neural network weights and the stability of the closed-loop system, which strongly supports the online implementation of the ADP method. Finally, two simulation examples are given to illustrate the performance and effectiveness of the proposed method.

In this paper, an iterative adaptive dynamic programming (ADP) algorithm is developed to solve the optimal cooperative control problems for residential multi-battery systems. To avoid solving high-dimensional optimal control problems, we first constrain all the batteries at their worst performance, which transforms the multi-input optimal control problem into a single-input one. Based on the worst-performance optimal control law, the optimal cooperative control law for the residential multi-battery systems is obtained, where in each iteration, only a single-input optimization problem is implemented. Finally, numerical results are given to illustrate the performance of the developed algorithm.

Model-free Adaptive Dynamic Programming for Optimal Control of Discrete-time Affine Nonlinear System

This paper focuses on a class of unknown discrete-time nonlinear systems with dead-zone control constraints based on adaptive dynamic programming (ADP). The discrete-time Hamilton-Jacobi-Bellman (DTHJB) equation corresponding to the dead-zone control input is formulated. Based on ADP technique, a new cost function is proposed that solves the optimal control with dead zone constraints effectively. It shows that this algorithm allows the implementation of the optimal control without knowing nonlinear affine system model and dead-zone dynamics. Finally, a simulation example is provided to verify the effectiveness of the proposed iterative algorithm.

This paper deals with optimal tracking control problems for a class of discrete-time nonlinear systems using a generalized policy iteration adaptive dynamic programming (ADP) algorithm. First, by system transformation, the optimal tracking control problem is transformed into an optimal regulation problem. Then the generalized policy iteration ADP algorithm is employed to obtain the optimal tracking controller with convergence and optimality analysis. The developed algorithm uses the idea of two iteration procedures to obtain the iterative tracking control laws and the iterative value functions. Three neural networks, including model network, critic network and action network, are used to implement the developed algorithm. At last, an simulation example is given to demonstrate the effectiveness of the developed method.

Optimal [formula omitted] tracking control of nonlinear systems with zero-equilibrium-free via novel adaptive critic designs

Fuzzy approximation-based optimal consensus control for nonlinear multiagent systems via adaptive dynamic programming

In this paper, we propose a new approach to address robust control problem of nonlinear systems with uncertainties based on an adaptive dynamic programming (ADP) algorithm. After reformulating the robust control problem as an optimal control problem, we propose a modified ADP method to solve the derived Hamilton-Jacobi-Bellman (HJB) equation, where the optimal cost function is approximated by online training a critic neural network (NN). Then the approximated optimal control action can be derived to guarantee the stability of the controlled system with uncertainties. The closed-loop system stability and convergence have been proved. A simulation example is provided to illustrate the effectiveness of the method.

With the increasing utilization of unmanned aerial vehicle (UAV) swarms in civilian and military field applications, it is crucial to eliminate the potential impacts of UAV faults on task completion and efficiency. To address this issue, the present paper investigates a fault-tolerant control scheme based on the adaptive dynamic programming (ADP) method and fault observer. The whole closed-loop system is composed of a position loop and an attitude loop. An ADP-based position-tracking controller is established with an improved cost function that can simultaneously represent actuator faults, control inputs, and tracking errors. In addition, a single-layer critic neural network is constructed to estimate the cost function and approximate control inputs. In the process of adjusting the weights of the neural network, a unique adjustment method combining policy gradient and prioritized experience replay is adopted to improve the data usage efficiency and speed up the weight convergence. Furthermore, a sliding mode–based attitude controller is utilized for the inner-loop attitude subsystem. The significant feature of this scheme is that it has the ability to self-learn and self-adapt. A Lyapunov stability analysis is performed to verify whether the signals are uniformly ultimately bounded. Finally, simulation experiments are conducted to demonstrate the effectiveness of the proposed scheme.

In this paper, a new iterative optimal learning control scheme for discrete-time nonlinear systems using iterative adaptive dynamic programming (ADP) approach is developed to obtain the optimal control law for coal gasification control systems. For the unknown coal gasification process, neural networks (NNs) are introduced to reconstruct the dynamics of the gasification process, where the approximation errors of the reconstruction dynamics are considered. Via system transformation, the optimal tracking control problem with approximation errors is transformed into a two-person zero-sum optimal control problem. A new iterative ADP algorithm is then developed to obtain the optimal control law for the transformed system with convergence analysis. Finally, numerical results are given to illustrate the performance of the present method.