Quadratic Tracking Control of Linear Stochastic Systems with Unknown Dynamics Using Average Off-Policy Q-Learning Method

In this paper, reinforcement learning (RL) is employed to find a casual solution to the linear quadratic tracker (LQT) for continuous-time systems online in real time. Although several RL techniques are developed in the literature to solve the LQ regulator, to our knowledge, there is no rigorous result for using RL to solve the LQ tracker. This is mainly because of the requirement for computing a feedforward term in the tracker control which must be done in a noncausal manner backwards in time. To deal with this noncausality problem, an augmented system composed of the original system and the command generator dynamics is constructed, and an augmented LQT algebraic Riccati equation is derived for solving the LQT problem. In this formulation, one can apply RL techniques to solve the LQT problem, computing the feedforward term and the feedback term simultaneously online in real time. The convergence of the proposed online algorithms to the optimal control solution is verified. To show the efficiency of the proposed approach, a simulation example is provided.

This paper discusses the optimal preview control problem for a class of linear continuous stochastic control systems in the infinite horizon, based on the augmented error system method. Firstly, an assistant system is designed and the state equation is translated to the assistant system. Then, an integrator is introduced to construct a stochastic augmented error system. As a result, the tracking problem is converted to a regulation problem. Secondly, the optimal regulator is solved based on dynamic programming principle for the stochastic system, and the optimal preview controller of the original system is obtained. Compared with the finite horizon, we simplify the performance index. We also study the stability of the stochastic augmented error system and design the observer for the original stochastic system. Finally, the simulation example shows the effectiveness of the conclusion in this paper.

Large-scale binary integer programs occur frequently in many real-world applications. For some binary integer problems, finding an optimal solution or even a feasible solution is computationally expensive. In this paper, we develop a discrete meta-control procedure to approximately solve large-scale binary integer programs efficiently. The key idea is to map the vector of n binary decision variables into a scalar function defined over a time interval [0; n] and construct a linear quadratic tracking (LQT) problem that can be solved efficiently. We prove that an LQT formulation has an optimal binary solution, analogous to a classical bang-bang control in continuous time. Our LQT approach can provide advantages in reducing computation while generating a good approximate solution. Numerical examples are presented to demonstrate the usefulness of the proposed method.

This thesis reviews the ideas of reinforcement learning and looks into the detail of three popular methods: dynamic programming, Monte Carlo method and temporal-difference method. Dynamic programming is the core idea of reinforcement learning. Grid world and cliff walking problems show how reinforcement learning works to find the optimal path, or policy. The thesis also reviews the knowledge of linear quadratic control problem. Two common methods are reviewed. One is the standard method for LQR problem and the other one is a causal solution with an augmented system. Some concepts of them are related to dynamic programming. Related to this, some algorithms of solving linear quadratic tracking problem are presented based on reinforcement learning. To this end, augmented systems are created and the problem appears in the form of two system state vectors with different value matrices. Both offline algorithms with known system dynamics and online algorithms without system dynamics are investigated. According to the state vector of systems and the property of strategies, these algorithms are clearly classified. The simulation is done to testify the algorithms. The linear quadratic tracking control of motor speed is carried out with one second sampling time in an experimental setting. It turns out that the controller gain converges and the motor speed can track the reference perfectly.

Designing the optimal control for the practical systems is challenging due to the unknown system dynamics and unavoidable external disturbances. In this article, the $H_{\infty } $ optimal control problem is investigated for continuous-time linear systems with unknown dynamics. The existing reinforcement learning-based $H_{\infty } $ optimal control methods require persistence of excitation (PE) condition or data storage mechanism to guarantee the convergence of the algorithms. However, PE condition is hard to be monitored online and data storage mechanism requires to store huge amounts of past system data. In order to solve these problems, the initial excitation-based reinforcement learning algorithms are presented to learn the optimal control policy under an online-verifiable initial excitation condition. The properties of the initial excitation-based reinforcement learning algorithms are analyzed, which show that the presented algorithms converge to the optimum under the initial excitation condition. Numerical analysis is provided which demonstrates the correctness of the presented algorithms.

The ship steering system is a typical electro-hydraulic servo system, which has the characteristics of non self-balance, large inertia, heavy load and uncertain disturbance. The load of the ship determines the depth of the draught. The depth of the draught determines the load of the steering system. The load of the rudder directly affects the dynamic and static tracking accuracy of the steering angle of the rudder. Therefore, the ship load is an important part of the ship steering system, and the establishment of a complete and accurate mathematical model is the necessary condition for the accurate control of the system. Based on the characteristics of the ship steering system, a general mathematical description with an uncertain disturbance system is presented. In order to obtain good performance of tracking control and resistance to load disturbance, a feedforward linear quadratic optimal tracking control method is proposed. The ship steering system designed based on this method can not only make the steering angle fast and accurately track the given, but also effectively suppress the heavy load uncertain disturbance. The simulation results show that feedforward linear quadratic optimal tracking control system has good tracking control performance and strong resistance to heavy load disturbance.

This paper presents a landing controller for a fixed-wing aircraft during the landing phase, ensuring the aircraft reaches the touchdown point smoothly. The landing problem is converted to a finite-time linear quadratic tracking (LQT) problem in which an aircraft needs to track the desired landing path in the longitudinal-vertical plane while satisfying performance requirements and flight constraints. First, we design a smooth trajectory that meets flight performance requirements and constraints. Then, an optimal controller is designed to minimize the tracking error, while landing the aircraft within the desired time frame. For this purpose, a linearized model of an aircraft developed under the assumption of a small flight path angle and a constant approach speed is used. The resulting Differential Riccati equation is solved backward in time using the Dormand Prince algorithm. Simulation results show a satisfactory tracking performance and the finite-time convergence of tracking errors for different initial conditions of the flare-out phase of landing.

Model-free finite-horizon optimal tracking control of discrete-time linear systems

Reinforcement learning (RL) and approximate dynamic programming (ADP) have been recently studied to solve nonlinear optimal control problems (OCPs) of continuous‐time (CT) systems. However, online learning efficiency and reliability are two major concerns to be further improved. Motivated by the above issues, in this paper we propose a receding‐horizon reinforcement learning (RHRL) algorithm for near‐optimal control of CT systems under control constraints. Different from classic RL and ADP, in the proposed approach, the infinite‐horizon OCP is decomposed as a series of finite‐horizon ones solved with an actor‐critic structure according to the receding horizon strategy, which can improve the online learning efficiency and reliability. The unknown dynamics of the system are identified offline using a sparse kernel‐based neural network structure whose weights are also updated online in the RHRL framework to improve the control performance. Moreover, the convergence of the modeling error is proven. To verify the effectiveness of our approach, we apply the RHRL algorithm to the autonomous ground vehicle for realizing near‐optimal path‐tracking control. Compared with CT model predictive control using a nominal model and other model‐free tracking controllers such as pure pursuit, heuristic dual programming, and the soft actor‐critic algorithm, RHRL performs better in terms of control performance.

In this article, a linear quadratic tracking controller is designed for continuous‐time delay systems involving multiplicative noises. This problem is challenging since the separation principle is ineffective for Itô stochastic systems and it is not assumed that the control weighting matrix in the cost function is positive definite. In this work, both finite time horizon and infinite time horizon cases are considered. The optimal controller is designed such that quadratic cost function in terms of the tracking error of the states and the control input is minimized. It is noted that comparing with regular problem, that is, the weighting matrix of control input in cost function is positive semidefinite, there exists an extra term in the optimal controller which can be free to choose. The explicit analytical expressions for the optimal controller are provided through solving a couple of Riccati‐like equations. For the infinite horizon case, the stabilization problem for an Itô stochastic system containing the input time delay is also studied. The effectiveness of the obtained theoretical results is illustrated by a numerical example.

An ADDHP-based Q-learning algorithm for optimal tracking control of linear discrete-time systems with unknown dynamics

This article addresses the optimal tracking control problem with prescribed performance for uncertain nonlinear systems subject to input constraint and unknown disturbances. First, a fixed‐time monotonic convergence function is introduced to restrain tracking error, and a nonlinear mapping technique is employed to transform the constrained error into an unconstrained variable, then the fixed‐time output tracking issue is boiled down to the boundedness problem of the transformed variable. With the aid of a nonquadratic cost function, the input constraint is encoded into the optimization problem. To solve the unknown disturbances, an auxiliary system and an auxiliary disturbance policy are constructed, and the optimal control problem is formulated as a two‐player zero‐sum game. Moreover, a Hamilton–Jacobi–Isaacs (HJI) equation associated with this nonquadratic zero‐sum game is established to give the optimal control and the worst‐case disturbance policy solution. Subsequently, to avoid using knowledge of the system dynamics, three neural network approximators, namely, actor, critic, and disturbance, which are tuned online and simultaneously for approximating the solution of HJI, are constructed based on the integral reinforcement learning algorithm. Theoretical analysis shows that the reconstructed error system states and the weight estimation errors are semi‐globally uniformly ultimately bounded. Finally, the simulation study further tests the availability of the proposed control strategy.

In this paper, an adaptive optimal control is investigated for a stochastic linear discrete-time system with multiplicative state-dependent noise and control-dependent noise without knowledge of the system dynamics. With the framework of Q-learning, an off-policy state feedback solution for stochastic linear quadratic tracking (SLQT) problem has been proposed. First, an augmented system of the original system and the reference command generator is established to solve SLQT problem. Then, we present an optimal control by solving stochastic algebraic Riccati equation (SARE). Next, we present the on-policy and off-policy algorithms to achieve an adaptive optimal control without knowing the system dynamics. Finally, a simulation test is finally setup to verify the performance of the proposed adaptive optimal control.

This article presents a unified approach to simultaneous solution of optimization and optimal control problems in batch distillation, operating under different modes of operation: variable, constant, or optimal reflux. The simplified, computationally efficient short‐cut method and a novel algorithm to solve the optimal control problems in batch distillation is the basis of this unified approach. The short‐cut method identifies the feasible region of operation essential for optimization and optimal control problems, and provides analytical partial derivatives of the model parameters crucial to the solution.The new algorithm for the solution of optimal combination of the maximum principle and NLP optimization techniques. It circumvents the problems associated with the maximum principle approach (iterative solution of a two‐point boundary value problem, undounded control variables, and inability to handle the simultaneous optimization and optimal control problem), and the coupled ODE discretization‐NLP optimization scheme for nonlinear models (higher system nonlinearities, multiplicity of solutions, sensitivity of convergence to initial guesses). This algorithm reduces the dimensionality of the problem, and the nature of the algorithm allows a common platform to optimal solutions of different operating conditions. This article also shows that different categories of the optimal control problems in batch distillation essentially involve the solution of the maximum distillate problem.

1 A Survey on Computational Optimal Control.- Issues in the Direct Transcription of Optimal Control Problems to Sparse Nonlinear Programs.- Optimization in Control of Robots.- Large-scale SQP Methods and their Application in Trajectory Optimization.- Solving Optimal Control and Pursuit-Evasion Game Problems of High Complexity.- 2 Theoretical Aspects of Optimal Control and Nonlinear Programming.- Continuation Methods In Boundary Value Problems.- Second Order Optimality Conditions for Singular Extremals.- Synthesis of Adaptive Optimal Controls for Linear Dynamic Systems.- Control Applications of Reduced SQP Methods.- Time Optimal Control of Mechanical Systems.- 3 Algorithms for Optimal Control Calculations.- Second Order Algorithm for Time Optimal Control of a Linear System.- An SQP-type Solution Method for Constrained Discrete-Time Optimal Control Problems.- Numerical Methods for Solving Differential Games, Prospective Applications to Technical Problems.- Construction of the Optimal Feedback Controller for Constrained Optimal Control Problems with Unknown Disturbances.- Repetitive Optimization for Predictive Control of Dynamic Systems under Uncertainty.- Optimal Control of Multistage Systems Described by High-Index Differential-Algebraic Equations.- A New Class of a High Order Interior Point Method for the Solution of Convex Semiinfinite Optimization Problems.- A Structured Interior Point SQP Method for Nonlinear Optimal Control Problems.- 4 Software for Optimal Control Calculations.- Automated Approach for Optimizing Dynamic Systems.- ANDECS: A Computation Environment for Control Applications of Optimization.- Application of Automatic Differentiation to Optimal Control Problems.- OCCAL: A mixed symbolic-numeric Optimal Control CALculator.- 5 Applications of Optimal Control.- A Robotic Satellite with Simplified Design.- Nonlinear Control under Constraints of a Biological System.- An Object-Oriented Approach to Optimally Describe and Specify a SCADA System Applied to a Power Network.- Near-Optimal Flight Trajectories Generated by Neural Networks.- Performance of a Feedback Method with Respect to Changes in the Air-Density during the Ascent of a Two-Stage-To-Orbit Vehicle.- Linear Optimal Control for Reentry Flight.- Steady-State Modelling of Turbine Engine with Controllers.- Shortest Paths for Satellite Mounted Robot Manipulators.- Optimal Control of the Industrial Robot Manutec r3.