A Stackelberg Differential Game-based Optimal Fault-tolerant Control Method for Over-actuated Systems

In this paper, a novel active optimal fault-tolerant control (FTC) scheme is designed based on adaptive dynamic programming (ADP) for modular manipulator when sensor and actuator faults are concurrency. Firstly, the sensor fault is transformed into the pseudo-actuator fault by constructing a nonlinear transformation with diffeomorphism theory. Secondly, the faults estimated by the adaptive fault observer are applied to establish an improved performance index function. Next, the online policy iteration (PI) algorithm is used to solve the Hamilton-Jacobi-Bellman (HJB) equation via establishing a critic neural network. The optimal fault-tolerant controller is proved to be uniformly ultimately bounded (UUB) based on Lyapunov stable theory. Finally, the effectiveness of the proposed multi-fault-tolerant control algorithm is verified by simulation results.

This paper studies the problem of optimal parallel tracking control for continuous-time general nonlinear systems. Unlike existing optimal state feedback control, the control input of the optimal parallel control is introduced into the feedback system. However, due to the introduction of control input into the feedback system, the optimal state feedback control methods can not be applied directly. To address this problem, an augmented system and an augmented performance index function are proposed firstly. Thus, the general nonlinear system is transformed into an affine nonlinear system. The difference between the optimal parallel control and the optimal state feedback control is analyzed theoretically. It is proven that the optimal parallel control with the augmented performance index function can be seen as the suboptimal state feedback control with the traditional performance index function. Moreover, an adaptive dynamic programming ( ADP ) technique is utilized to implement the optimal parallel tracking control using a critic neural network ( NN ) to approximate the value function online. The stability analysis of the closed-loop system is performed using the Lyapunov theory, and the tracking error and NN weights errors are uniformly ultimately bounded ( UUB ) . Also, the optimal parallel controller guarantees the continuity of the control input under the circumstance that there are finite jump discontinuities in the reference signals. Finally, the effectiveness of the developed optimal parallel control method is verified in two cases.

In this paper, an iterative adaptive dynamic programming (ADP) algorithm is developed to solve the optimal cooperative control problems for residential multi-battery systems. To avoid solving high-dimensional optimal control problems, we first constrain all the batteries at their worst performance, which transforms the multi-input optimal control problem into a single-input one. Based on the worst-performance optimal control law, the optimal cooperative control law for the residential multi-battery systems is obtained, where in each iteration, only a single-input optimization problem is implemented. Finally, numerical results are given to illustrate the performance of the developed algorithm.

In this paper, a learning-based predictive control (LPC) scheme is proposed for adaptive optimal control of discrete-time nonlinear systems under stochastic disturbances. The proposed LPC scheme is different from conventional model predictive control (MPC), which uses open-loop optimization or simplified closed-loop optimal control techniques in each horizon. In LPC, the control task in each horizon is formulated as a closed-loop nonlinear optimal control problem and a finite-horizon iterative reinforcement learning (RL) algorithm is developed to obtain the closed-loop optimal/suboptimal solutions. Therefore, in LPC, RL and adaptive dynamic programming (ADP) are used as a new class of closed-loop learning-based optimization techniques for nonlinear predictive control with stochastic disturbances. Moreover, LPC also decomposes the infinite-horizon optimal control problem in previous RL and ADP methods into a series of finite horizon problems, so that the computational costs are reduced and the learning efficiency can be improved. Convergence of the finite-horizon iterative RL algorithm in each prediction horizon and the Lyapunov stability of the closed-loop control system are proved. Moreover, by using successive policy updates between adjoint time horizons, LPC also has lower computational costs than conventional MPC which has independent optimization procedures between two different prediction horizons. Simulation results illustrate that compared with conventional nonlinear MPC as well as ADP, the proposed LPC scheme can obtain a better performance both in terms of policy optimality and computational efficiency.

In this paper, a new iterative optimal learning control scheme for discrete-time nonlinear systems using iterative adaptive dynamic programming (ADP) approach is developed to obtain the optimal control law for coal gasification control systems. For the unknown coal gasification process, neural networks (NNs) are introduced to reconstruct the dynamics of the gasification process, where the approximation errors of the reconstruction dynamics are considered. Via system transformation, the optimal tracking control problem with approximation errors is transformed into a two-person zero-sum optimal control problem. A new iterative ADP algorithm is then developed to obtain the optimal control law for the transformed system with convergence analysis. Finally, numerical results are given to illustrate the performance of the present method.

Nearly finite-horizon optimal control for a class of nonaffine time-delay nonlinear systems based on adaptive dynamic programming

This article investigates a novel reinforcement learning-based fault-tolerant control approach for steel-jacket offshore platforms. In the first step, the dynamic of the steal-jacket offshore platform with an active mass damper is considered, and the equivalent linear time-invariant model is obtained with the actuator fault. In fault-free conditions, an optimal controller is designed to keep the system stable under external wave force. Subsequently, in faulty conditions, the actuator fault is estimated by the fault observer. Next, by inserting the actuator fault estimation into the cost function, the fault-tolerant control problem transforms into the optimal control problem. The online policy iteration is used to minimize the new cost function. Finally, the final control law, which is a mixture of the nominal and the modified control law, stabilizes the offshore platform and improves its performance in the presence of the actuator fault without needing the complete knowledge of the offshore platform. The simulation results show the effectiveness of the proposed method.

In this paper, an infinite horizon optimal robust guaranteed cost control scheme of a class of continuous-time uncertain nonlinear systems is established based on adaptive dynamic programming. The main idea lies in that the optimal robust guaranteed cost control problem can be transformed into an optimal control problem. Actually, the optimal cost function of the nominal system is nothing but the optimal guaranteed cost of the original uncertain system. A critic neural network is constructed to help solving the modified Hamilton-Jacobi-Bellman equation corresponding to the nominal system. Then, an additional stabilizing term is introduced to reinforce the updating process of the weight vector and reduce the requirement of an initial stabilizing control. An example is provided to illustrate the effectiveness of the present control approach.

This paper presents a functional model predictive control (MPC) approach based on an adaptive dynamic programming (ADP) algorithm with the abilities of handling control constraints and disturbances for the optimal control of nonlinear discrete-time systems. In the proposed ADP-based nonlinear MPC (NMPC) structure, a neural-network-based identification is established first to reconstruct the unknown system dynamics. Then, the actor-critic scheme is adopted with a critic network to estimate the index performance function and an action network to approximate the optimal control input. Meanwhile, as the MPC strategy can effectively determine the current control by solving a finite horizon open-loop optimal control problem, in the proposed algorithm, the infinite horizon is decomposed into a series of finite horizons to obtain the optimal control. In each finite horizon, the finite ADP algorithm solves the optimal control problem subject to the terminal constraint, the control constraint, and the disturbance. The uniform ultimate boundedness of the closed-loop system is verified by the Lyapunov approach. Finally, the ADP-based NMPC is conducted on two different cases and the simulation results demonstrate the quick response and strong robustness of the proposed method.

Optimal control of aluminum electrolysis production process (AEPP) has long been a challenging industrial issue due to its inherent difficulty in establishing an accurate dynamic model. In this paper, a novel robust optimal control algorithm based on adaptive dynamic programming (ADP) is proposed for the AEPP, where the system subjects to input constraints. First, to establish an accurate dynamic model for the AEPP system, recursive neural network (RNN) is employed to reconstruct the system dynamic using the input-output production data. To ensure input constraints are not to exceed the bound of the actuator, the optimal control problem of the AEPP is formulated under a new nonquadratic form performance index function. Then, considering the perturbation of the AEPP, the robust control problem is effectively converted to the constrained optimal control problem via system transformation. Furthermore, a single critic network framework is developed to obtain the approximate solution of the Hamilton-Jacobi-Bellman (HJB) equation. Finally, the proposed ADP controller is applied to the AEPP system to validate the effectiveness and performance.

Optimal control of Markovian jump systems via a neural network-based ADP iterative algorithm

Pseudospectral approximation techniques have been shown to provide effective and flexible methods for solving optimal control problems in a variety of applications. In this paper, we provide the conditions for the convergence of the pseudospectral method for general nonlinear optimal control problems. Further, we show that this proof is directly extendible to the multidimensional pseudospectral method for optimal ensemble control of a class of parameterized dynamical systems. Examples from quantum control and neuroscience are included to demonstrate the method.

Dynamic compensator-based near-optimal control for unknown nonaffine systems via integral reinforcement learning

This paper aims at developing near optimal traffic signal control for multi-intersections in city. As a new optimization technique, adaptive dynamic programming (ADP) combines concepts of reinforcement learning and dynamic programming. ADP could learn continually from experience to achieve a near optimal control policy under varying conditions. However, without the cooperation among adjacent intersections, the near optimal control for each individual intersection can not guarantee a larger traffic area composing several intersections to be near optimal. This paper presents a new signal control method based on a model-free action-dependent ADP (ADHDP). This method can be used for cooperative control of multiple intersections. In every intersection, an ADHDP signal controller is adopted to adjust signal time according to an integrated unity parameter. The unity parameter is designed to consider not only the control performance in local intersection but also those in the neighbor intersections. Thus the designed controllers could achieve a set of near optimal control police for multi-intersections in a long run. Simulation results show that the trained controller achieves shorter average vehicular delay.

In the present chapter, intelligent dynamic optimization methods based on adaptive dynamic programming (ADP) are applied to deal with the challenges of intelligent price-responsive management of residential energy, with an emphasis on home battery connected to the power grid. First, an action-dependent heuristic dynamic programming method is developed to obtain the optimal residential energy control law. Second, a dual iterative Q-learning algorithm is developed to solve the optimal battery management and control problem in smart residential environments where two iterations, internal and external iterations, are employed. Based on the dual iterative Q-learning algorithm, the convergence property of iterative Q-learning method for the optimal battery management and control problem is proven. Finally, a distributed iterative ADP method is developed to solve the multi-battery optimal coordination control problems for home energy management systems.