Adaptive optimal control for a class of nonlinear partially uncertain dynamic systems via policy iteration

Approaching the problem of optimal adaptive control as ldquooptimal control made adaptive,rdquo namely, as a certainty equivalence combination of linear quadratic optimal control and standard parameter estimation, fails on two counts: numerical (as it requires a solution to a Riccati equation at each time step) and conceptual (as the combination actually does not possess any optimality property). In this note, we present a particular form of optimality achievable in Lyapunov-based adaptive control. State and control are subject to positive definite penalties, whereas the parameter estimation error is penalized through an exponential of its square, which means that no attempt is made to enforce the parameter convergence, but the estimation transients are penalized simultaneously with the state and control transients. The form of optimality we reveal here is different from our work in [Z. H. Li and M. Krstic, ldquoOptimal design of adaptive tracking controllers for nonlinear systems,rdquo <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Automatica</i> , vol. 33, pp. 1459-1473, 1997] where only the terminal value of the parameter error was penalized. We present our optimality concept on a partial differential equation (PDE) example-boundary control of a particular parabolic PDE with an unknown reaction coefficient. Two technical ideas are central to the developments in the note: a nonquadratic Lyapunov function and a normalization in the Lyapunov-based update law. The optimal adaptive control problem is fundamentally nonlinear and we explore this aspect through several examples that highlight the interplay between the non-quadratic cost and value functions.

This article addresses the event-based adaptive optimal fault-tolerant consensus control problem for a class of nonlinear multiagent systems (MASs) under a directed graph. It can be solved that both control gains and actuator failure parameters are unknown in considered MASs, which enhances the practicability of optimal consensus control. First, a reinforcement learning-based distributed optimal control law is designed by constructing the identifier-critic-actor learning networks. Furthermore, by utilizing the distributed optimal control law as an auxiliary variable, an adaptive fault-tolerant controller is proposed to effectively compensate for actuator failures. Meanwhile, a co-design scheme is proposed for the construction of an event-triggered control input with the fault-tolerant property. It is proven that the designed control input ensures the boundedness of closed-loop systems through rigorous stability analysis. Finally, the effectiveness of the developed approach can be illustrated via simulations of numerical nonlinear MASs and a group of autonomous underwater vehicles.

In this article, an observer-based fuzzy adaptive inverse optimal output feedback control problem is studied for a class of nonlinear systems in strict-feedback form. The considered nonlinear systems contain unknown nonlinear dynamics and their states are not measured directly. Fuzzy logic systems are applied to identify the unknown nonlinear dynamics and an auxiliary nonlinear system is constructed. Based on this auxiliary system, a fuzzy state observer is first designed to estimate the immeasurable states. By using the inverse optimal principle and adaptive backstepping design theory, an observer-based fuzzy adaptive inverse optimal output feedback control scheme is then developed. The proposed inverse optimal control scheme need not assume that the states are measurable. It also guarantees that the closed-loop system is semiglobally uniformly ultimately bounded, and achieves the optimal control objective as well. Finally, two simulation examples are provided to check the validity of the presented control method.

1 A Survey on Computational Optimal Control.- Issues in the Direct Transcription of Optimal Control Problems to Sparse Nonlinear Programs.- Optimization in Control of Robots.- Large-scale SQP Methods and their Application in Trajectory Optimization.- Solving Optimal Control and Pursuit-Evasion Game Problems of High Complexity.- 2 Theoretical Aspects of Optimal Control and Nonlinear Programming.- Continuation Methods In Boundary Value Problems.- Second Order Optimality Conditions for Singular Extremals.- Synthesis of Adaptive Optimal Controls for Linear Dynamic Systems.- Control Applications of Reduced SQP Methods.- Time Optimal Control of Mechanical Systems.- 3 Algorithms for Optimal Control Calculations.- Second Order Algorithm for Time Optimal Control of a Linear System.- An SQP-type Solution Method for Constrained Discrete-Time Optimal Control Problems.- Numerical Methods for Solving Differential Games, Prospective Applications to Technical Problems.- Construction of the Optimal Feedback Controller for Constrained Optimal Control Problems with Unknown Disturbances.- Repetitive Optimization for Predictive Control of Dynamic Systems under Uncertainty.- Optimal Control of Multistage Systems Described by High-Index Differential-Algebraic Equations.- A New Class of a High Order Interior Point Method for the Solution of Convex Semiinfinite Optimization Problems.- A Structured Interior Point SQP Method for Nonlinear Optimal Control Problems.- 4 Software for Optimal Control Calculations.- Automated Approach for Optimizing Dynamic Systems.- ANDECS: A Computation Environment for Control Applications of Optimization.- Application of Automatic Differentiation to Optimal Control Problems.- OCCAL: A mixed symbolic-numeric Optimal Control CALculator.- 5 Applications of Optimal Control.- A Robotic Satellite with Simplified Design.- Nonlinear Control under Constraints of a Biological System.- An Object-Oriented Approach to Optimally Describe and Specify a SCADA System Applied to a Power Network.- Near-Optimal Flight Trajectories Generated by Neural Networks.- Performance of a Feedback Method with Respect to Changes in the Air-Density during the Ascent of a Two-Stage-To-Orbit Vehicle.- Linear Optimal Control for Reentry Flight.- Steady-State Modelling of Turbine Engine with Controllers.- Shortest Paths for Satellite Mounted Robot Manipulators.- Optimal Control of the Industrial Robot Manutec r3.

In this chapter, by using neural network (NN)-based online learning optimal control approach, a decentralized control strategy is developed to stabilize a class of continuous-time large-scale interconnected nonlinear systems . It is proven that the decentralized control strategy of the overall system can be established by adding appropriate feedback gains to the optimal control laws of the isolated subsystems . Then, an online policy iteration (PI) algorithm is developed to solve the Hamilton–Jacobi–Bellman equations related to the optimal control problem. By constructing a set of critic NNs, the cost functions can be obtained by NN approximation, followed by the control laws. Furthermore, as a generalization, an NN-based decentralized control law is developed to stabilize the large-scale interconnected nonlinear systems using an online model-free integral PI algorithm. The model-free PI approach can solve the decentralized control problem for large-scale interconnected nonlinear systems with unknown dynamics. Finally, two simulation examples are provided to illustrate the effectiveness of the present decentralized control scheme.

In this study, an optimal adaptive control approach is established to solve the robust output tracking problem of a class of continuous time uncertain linear systems based on the policy iteration (PI) in actor-critic algorithm. First, by augmenting the integral variables of the tracking error into state variables, the robust tracking problem is transformed into a robust control problem of an augmented uncertain linear system. It is proven that the robust control law of the augmented system enables the output of the considered system to track a polynomial time signal asymptotically. Second, an optimal control method in the corresponding auxiliary nominal system is established, and based on the Bellman optimality principle, PI algorithms are proposed to solve online tracking controllers for the matched and the mismatched uncertain systems. Finally, for testing the availability of the proposed approach and theoretical results, two numerical experiments are provided.

Data-based adaptive neural network optimal output feedback control for nonlinear systems with actuator saturation

This article addresses the optimal tracking control problem with prescribed performance for uncertain nonlinear systems subject to input constraint and unknown disturbances. First, a fixed‐time monotonic convergence function is introduced to restrain tracking error, and a nonlinear mapping technique is employed to transform the constrained error into an unconstrained variable, then the fixed‐time output tracking issue is boiled down to the boundedness problem of the transformed variable. With the aid of a nonquadratic cost function, the input constraint is encoded into the optimization problem. To solve the unknown disturbances, an auxiliary system and an auxiliary disturbance policy are constructed, and the optimal control problem is formulated as a two‐player zero‐sum game. Moreover, a Hamilton–Jacobi–Isaacs (HJI) equation associated with this nonquadratic zero‐sum game is established to give the optimal control and the worst‐case disturbance policy solution. Subsequently, to avoid using knowledge of the system dynamics, three neural network approximators, namely, actor, critic, and disturbance, which are tuned online and simultaneously for approximating the solution of HJI, are constructed based on the integral reinforcement learning algorithm. Theoretical analysis shows that the reconstructed error system states and the weight estimation errors are semi‐globally uniformly ultimately bounded. Finally, the simulation study further tests the availability of the proposed control strategy.

This paper investigates fuzzy adaptive output feedback optimal control problem for a class of strict-feedback nonlinear systems. With the help of fuzzy logic systems approximating the unknown nonlinear functions and cost function, the unmeasured states are estimated by designing fuzzy adaptive state observer. Combining state observer with backstepping design technique, a feedforward controller is designed. Based on the designed feedforward control strategy, the controlled nonlinear system can be converted to an equivalence nonlinear system in affine-form. Finally, a fuzzy adaptive optimal controller with parameters adaptive laws is developed. The whole control scheme consists of a feedforward controller and a feedback optimal controller. It is shown that the proposed output feedback optimal control approach can guarantee that all signals in the closed-loop system are bounded, and the system output can track the reference signal. In addition, the proposed control approach can guarantee cost function is the smallest. Simulation results are given to demonstrate the effectiveness of the proposed control approach.

This article addresses an adaptive backstepping finite-time optimal formation control problem for second-order multiagent systems (MASs) with unknown nonlinear dynamics. Neural networks (NNs) are used to identify the unknown uncertain terms in the controlled system. Then, the finite-time optimal formation control is designed by constructing a novel optimal performance index function containing exponential power terms based on the framework of identifier–actor–critic. It is proved that all signals in the system are bounded in finite time, and the formation control is simultaneously achieved at minimum cost. The effectiveness and superiority of the proposed control algorithm are verified by simulation comparisons and data analyses.

Most optimal control method aim at finding a control law to minimize the corresponding cost function, and the key problem is to solve the Hamilton–Jacobi–Bellman equation (HJBE). However, the solution of HJBE is hard and even unavailable to obtain for nonlinear system and, thus, the optimal control method can not be developed. To address this problem, an adaptive neural inverse optimal control method is proposed for nonlinear systems with external disturbances and actuator failures in this article. The cost function in this article is defined based on the controller and, therefore, once the controller is obtained, the corresponding cost function is minimized. Accordingly, the directly solution of HJBE is avoided. In more detail, the auxiliary system is constructed to develop the auxiliary controller, based on which the actual optimal fault-tolerant controller can be further obtained. The uncertainties in system dynamics are compensated by neural networks, and the unknown actuator failures are handled by adaptive control gain. It is proved that nonlinear system is input-to-state stabilizable (ISS) and the proposed controller can minimize the cost function that puts penalty on tracking error and system input. Two simulation examples are provided to demonstrate the effectiveness of the proposed control method.

For the optimal tracking control problem of affine nonlinear systems, a general value iteration algorithm based on adaptive dynamic programming is proposed in this paper. By system transformation, the optimal tracking problem is converted into the optimal regulating problem for the tracking error dynamics. Then, general value iteration algorithm is developed to obtain the optimal control with convergence analysis. Considering the advantages of echo state network, we use three echo state networks with levenberg-Marquardt (LM) adjusting algorithm to approximate the system, the cost function and the control law. A simulation example is given to demonstrate the effectiveness of the presented scheme.

This paper investigates the robust optimal control problem of a class of continuous-time, partially linear, interconnected systems. In addition to the dynamic uncertainties resulted from the interconnected dynamic system, unknown bounded disturbances are taken into account throughout the learning process, wherein the system’s dynamics and the disturbances are assumed unknown. These challenges lead the collected online data to be imperfect. In this scenario, traditional data-driven control techniques, such as adaptive dynamic programming (ADP) and robust ADP, encounter a challenge in approximating the optimal control policy precisely due to imperfect data and computational errors. In this paper, a novel data-driven robust policy iteration method is proposed to simultaneously solve the robust optimal control problems. Without relying on the knowledge of the system’s dynamics, the external disturbances or the complete state, the implementation of the proposed method only needs to access the input and partial state information. Based on the small-gain theorem, the notions of strong unboundedness observability and input-to-output stability, it is guaranteed that the learned robust optimal control gain is stabilizing and that the solution of the closed-loop system is uniformly ultimately bounded despite the existence of dynamic uncertainties and unknown external disturbances. The simulation results reveal the efficiency and practicality of the proposed data-driven control method. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This work is motivated by the use of reinforcement learning to improve the quality of designing adaptive optimal controllers for engineering applications. Adaptive dynamic programming methods, in particular policy iteration (PI), are widely used in solving optimal control problems. However, due to the iterative nature of PI, the approximated optimal control policy may be inaccurate and imprecise. Especially, when using imperfect system’s measurements instead of the modelling information. This can result in causing the learned control policy to deviate from the actual optimal policy. This becomes more challenging in the existence of dynamic uncertainties and unknown external disturbances which corrupt the measurements and result in imperfect data. This work investigates the conditions on the uncertainties such that the proposed novel data-driven PI algorithm is robust to system’s uncertainties, unknown external disturbances and imperfect measurements. The approximated robust optimal control policy performs robustly in the existences of imperfect data and uncertainties, and at the same time is close enough to the optimal control policy.

Adaptive nearly optimal control for a class of continuous-time nonaffine nonlinear systems with inequality constraints

SummaryThe problem of adaptive optimal semiactive control of a structure subjected to a moving load is studied. The control is realised by a change of damping of the structure's supports. The results presented in the previous works of the authors demonstrate that switched optimal controls can be very efficient at reducing the vibration levels of the structure. On the other hand, these controls exhibit a high sensitivity to changes of the speed of the travelling load. The aim of this paper is to develop an algorithm that enables real‐time adaptation of the optimal controls according to both the measured speed of the travelling load and the estimated state of the structure. The control objective is to provide smooth passage for the vehicles and reduce the material stresses on the carrying structures. The designed adaptive algorithm uses reference optimal controls computed for constant speeds and a set of functions describing the sensitivity of the system dynamics to the measured parameters. The convergence of the algorithm, as well as aspects of its implementation, is studied. The performance of the proposed method is validated by means of numerical simulations conducted for different travelling speed scenarios. In the assumed objective functional, the proposed adaptive controller can outperform the reference optimal solutions by over 50%. The practicality of the proposed method should attract the attention of practising engineers.