Off-policy Q-learning-based Tracking Control for Stochastic Linear Discrete-Time Systems

This article investigates the optimal tracking control problem for data-based stochastic discrete-time linear systems. An average off-policy Q-learning algorithm is proposed to solve the optimal control problem with random disturbances. Compared with the existing off-policy reinforcement learning (RL) algorithm, the proposed average off-policy Q-learning algorithm avoids the assumption of an initial stability control. First, a pole placement strategy is used to design an initial stable control for systems with unknown dynamics. Second, the initial stable control is used to design a data-based average off-policy Q-learning algorithm. Then, this algorithm is used to solve the stochastic linear quadratic tracking (LQT) problem, and a convergence proof of the algorithm is provided. Finally, numerical examples show that this algorithm outperforms other algorithms in a simulation.

Adaptive Optimal Control of Nonlinear Inverted Pendulum System Using Policy Iteration Technique

Approaching the problem of optimal adaptive control as ldquooptimal control made adaptive,rdquo namely, as a certainty equivalence combination of linear quadratic optimal control and standard parameter estimation, fails on two counts: numerical (as it requires a solution to a Riccati equation at each time step) and conceptual (as the combination actually does not possess any optimality property). In this note, we present a particular form of optimality achievable in Lyapunov-based adaptive control. State and control are subject to positive definite penalties, whereas the parameter estimation error is penalized through an exponential of its square, which means that no attempt is made to enforce the parameter convergence, but the estimation transients are penalized simultaneously with the state and control transients. The form of optimality we reveal here is different from our work in [Z. H. Li and M. Krstic, ldquoOptimal design of adaptive tracking controllers for nonlinear systems,rdquo <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Automatica</i> , vol. 33, pp. 1459-1473, 1997] where only the terminal value of the parameter error was penalized. We present our optimality concept on a partial differential equation (PDE) example-boundary control of a particular parabolic PDE with an unknown reaction coefficient. Two technical ideas are central to the developments in the note: a nonquadratic Lyapunov function and a normalization in the Lyapunov-based update law. The optimal adaptive control problem is fundamentally nonlinear and we explore this aspect through several examples that highlight the interplay between the non-quadratic cost and value functions.

Data-based adaptive neural network optimal output feedback control for nonlinear systems with actuator saturation

An observer-based adaptive inverse optimal control (AIOC) scheme is developed for unknown nonlinear systems. First, a dynamic neural network (DNN) observer is designed to estimate system states. Then, based on the observed states, a neuro-controller is constructed based on control Lyapunov function (CLF) via inverse optimal method to obtain the optimal feedback control. In this framework, a two-layer DNN is used to construct the observer which can be implemented to systems with higher degrees of nonlinearity and without a priori knowledge of system dynamics. The optimal control scheme is implemented to online control the unknown nonlinear systems and the DNN observer simultaneously. An asymptotic stability of the closed-loop system is guaranteed using Lyapunov-based analysis. Simulation results are presented to demonstrate the effectiveness of the proposed control scheme.

Model-free finite-horizon optimal tracking control of discrete-time linear systems

A new efficient adaptive optimal control approach is presented in this paper based on the indirect model reference adaptive control (MRAC) architecture for improvement of adaptation and tracking performance of the uncertain system. The system accounts here for both matched and unmatched unknown uncertainties that can act as plant as well as input effectiveness failures or damages. For adaptation of the unknown parameters of these uncertainties, the frequency selective learning approach is used. Its idea is to compute a filtered expression of the system uncertainty using multiple filters based on online instantaneous information, which is used for augmentation of the update law. It is capable of adjusting a sudden change in system dynamics without depending on high adaptation gains and can satisfy exponential parameter error convergence under certain conditions in the presence of structured matched and unmatched uncertainties as well. Additionally, the controller of the MRAC system is designed using a new optimal control method. This method is a new linear quadratic regulator-based optimal control formulation for both output regulation and command tracking problems. It provides a closed-form control solution. The proposed overall approach is applied in a control of lateral dynamics of an unmanned aircraft problem to show its effectiveness.

In this paper, adaptive optimal control is proposed for linear discrete time-varying (LDTV) systems subject to unknown system dynamics. The idea of the method is a direct application of the Q-learning adaptive dynamic programming for time-varying systems. In order to derive the optimal control policy, an actor-critic structure is constructed and the time-varying least square method is adopted for parameter adaptation. The derived control policy robustly stabilizes the time-varying system and guarantees an optimal control performance. As no particular system information is required throughout the process, the proposed method provides a feasible solution to a large variety of applications. The validity of the proposed method is verified through simulation studies.

This paper presents an adaptive optimal control strategy focused on improving ride comfort by minimising the root mean square of accelerations measured at seven locations on the carbody floor. It requires preview knowledge of both the train speed and the standard track quality (not the deterministic track disturbances) and differs from other works in considering pointwise constraints (upper and lower secondary suspension travel limits). Adaptive control is achieved by varying the diameter of the restriction orifice which connects the pneumatic spring to the auxiliary air reservoir, which confers different dynamic stiffness and damping to the front and rear secondary suspensions. Decision maps with the front and rear optimal diameters for five track irregularity levels and six different speeds were obtained using a Monte Carlo strategy based on solving the optimal control problem for 240 stochastic realisations of time-varying track irregularities of 300 s. Results show that the adaptive optimal control provides a successful compromise between comfort and safety which overshadows the conventional passive approach: it improves comfort up to 50% with respect to stiff passive configurations and it guarantees constraint fulfilment unlike compliant passive configurations.

Voronoi-based adaptive area optimal coverage control for multiple manipulator systems with uncertain kinematics and dynamics

Most optimal control method aim at finding a control law to minimize the corresponding cost function, and the key problem is to solve the Hamilton–Jacobi–Bellman equation (HJBE). However, the solution of HJBE is hard and even unavailable to obtain for nonlinear system and, thus, the optimal control method can not be developed. To address this problem, an adaptive neural inverse optimal control method is proposed for nonlinear systems with external disturbances and actuator failures in this article. The cost function in this article is defined based on the controller and, therefore, once the controller is obtained, the corresponding cost function is minimized. Accordingly, the directly solution of HJBE is avoided. In more detail, the auxiliary system is constructed to develop the auxiliary controller, based on which the actual optimal fault-tolerant controller can be further obtained. The uncertainties in system dynamics are compensated by neural networks, and the unknown actuator failures are handled by adaptive control gain. It is proved that nonlinear system is input-to-state stabilizable (ISS) and the proposed controller can minimize the cost function that puts penalty on tracking error and system input. Two simulation examples are provided to demonstrate the effectiveness of the proposed control method.

Optimal adaptive control of fed-batch fermentation processes

In this paper, a novel strategy iterative method based on finite-time extended state observer is proposed to find the online adaptive optimal controller for a class of linear continuous-time systems with completely unknown system dynamics and unmeasurable state variables. In this method, finite-time state observer estimates the system state variables by using the input and output information of the system and the policy iterative method uses the estimated state variables and system input information to iteratively solve the algebraic Riccati equation until the adaptive optimal controller is obtained. It is worth noting that the method proposed in this paper only uses the input and output information of the system, without requiring the apriori knowledge of the system matrices and the state variables. A numerical simulation example will be provided to demonstrate the effectiveness of the method.

In this article, an observer-based fuzzy adaptive inverse optimal output feedback control problem is studied for a class of nonlinear systems in strict-feedback form. The considered nonlinear systems contain unknown nonlinear dynamics and their states are not measured directly. Fuzzy logic systems are applied to identify the unknown nonlinear dynamics and an auxiliary nonlinear system is constructed. Based on this auxiliary system, a fuzzy state observer is first designed to estimate the immeasurable states. By using the inverse optimal principle and adaptive backstepping design theory, an observer-based fuzzy adaptive inverse optimal output feedback control scheme is then developed. The proposed inverse optimal control scheme need not assume that the states are measurable. It also guarantees that the closed-loop system is semiglobally uniformly ultimately bounded, and achieves the optimal control objective as well. Finally, two simulation examples are provided to check the validity of the presented control method.

In this paper, a novel NN-based optimal adaptive consensus-based formation control scheme over finite horizon is presented for networked mobile robots or agents in the presence of uncertain robot/agent dynamics. The uncertain robot formation dynamics are approximated online by using an NN-based identifier and a suitable weight tuning law. In addition, a novel time-varying value function is derived by using the augmented error vector, which consists of the regulation and consensus-based formation errors of each robot. By using the value function approximation and the identified dynamics, the near optimal control input over finite horizon is derived. This finite horizon optimal control leads to a time-varying value function, which becomes the solution of the Hamilton–Jacobi–Bellman equation, and control input is approximated by a second NN with time-varying activation function. A novel weight update law for the NN value function is developed to tune the value function, satisfy the terminal constraint, and relax an initial admissible controller requirement. The Lyapunov stability method is utilized to demonstrate the consensus of the overall formation. Finally, simulation results are given to verify theoretical claims. Copyright © 2015 John Wiley & Sons, Ltd.