An ADDHP-based Q-learning algorithm for optimal tracking control of linear discrete-time systems with unknown dynamics

Model-free finite-horizon optimal tracking control of discrete-time linear systems

Dynamic compensator-based near-optimal control for unknown nonaffine systems via integral reinforcement learning

In this paper, the finite-horizon near optimal adaptive regulation of linear discrete-time systems with unknown system dynamics is presented in a forward-in-time manner by using adaptive dynamic programming and Q-learning. An adaptive estimator (AE) is introduced to relax the requirement of system dynamics, and it is tuned by using Q-learning. The time-varying solution to the Bellman equation in adaptive dynamic programming is handled by utilizing a time-dependent basis function, while the terminal constraint is incorporated as part of the update law of the AE. The Kalman gain is obtained by using the AE parameters, while the control input is calculated by using AE and the system state vector. Next, to relax the need for state availability, an adaptive observer is proposed so that the linear quadratic regulator design uses the reconstructed states and outputs. For the time-invariant linear discrete-time systems, the closed-loop dynamics becomes non-autonomous and involved but verified by using standard Lyapunov and geometric sequence theory. Effectiveness of the proposed approach is verified by using simulation results. The proposed linear quadratic regulator design for the uncertain linear system requires an initial admissible control input and yields a forward-in-time and online solution without needing value and/or policy iterations. Copyright © 2014 John Wiley & Sons, Ltd.

This thesis reviews the ideas of reinforcement learning and looks into the detail of three popular methods: dynamic programming, Monte Carlo method and temporal-difference method. Dynamic programming is the core idea of reinforcement learning. Grid world and cliff walking problems show how reinforcement learning works to find the optimal path, or policy. The thesis also reviews the knowledge of linear quadratic control problem. Two common methods are reviewed. One is the standard method for LQR problem and the other one is a causal solution with an augmented system. Some concepts of them are related to dynamic programming. Related to this, some algorithms of solving linear quadratic tracking problem are presented based on reinforcement learning. To this end, augmented systems are created and the problem appears in the form of two system state vectors with different value matrices. Both offline algorithms with known system dynamics and online algorithms without system dynamics are investigated. According to the state vector of systems and the property of strategies, these algorithms are clearly classified. The simulation is done to testify the algorithms. The linear quadratic tracking control of motor speed is carried out with one second sampling time in an experimental setting. It turns out that the controller gain converges and the motor speed can track the reference perfectly.

This article investigates the optimal tracking control problem for data-based stochastic discrete-time linear systems. An average off-policy Q-learning algorithm is proposed to solve the optimal control problem with random disturbances. Compared with the existing off-policy reinforcement learning (RL) algorithm, the proposed average off-policy Q-learning algorithm avoids the assumption of an initial stability control. First, a pole placement strategy is used to design an initial stable control for systems with unknown dynamics. Second, the initial stable control is used to design a data-based average off-policy Q-learning algorithm. Then, this algorithm is used to solve the stochastic linear quadratic tracking (LQT) problem, and a convergence proof of the algorithm is provided. Finally, numerical examples show that this algorithm outperforms other algorithms in a simulation.

Q-learning solution for optimal consensus control of discrete-time multiagent systems using reinforcement learning

This paper studies data-driven learning-based methods for the finite-horizon optimal control of linear time-varying discrete-time systems. First, a novel finite-horizon Policy Iteration (PI) method for linear time-varying discrete-time systems is presented. Its connections with existing infinite-horizon PI methods are discussed. Then, both data-driven off-policy PI and Value Iteration (VI) algorithms are derived to find approximate optimal controllers when the system dynamics is completely unknown. Under mild conditions, the proposed data-driven off-policy algorithms converge to the optimal solution. Finally, the effectiveness and feasibility of the developed methods are validated by a practical example of spacecraft attitude control.

This paper investigates the application of associative reinforcement learning techniques to the optimal control of linear discrete-time dynamic systems. Associative reinforcement learning involves the trial and error interaction with a dynamic system to determine the control actions that optimally achieve some desired performance index. The methodology can be applied either online or off-line and in a model based or model free manner. Associative reinforcement learning techniques are applied to the optimal regulator (LQR) control of discrete-time linear systems. Adaptive critic designs are implemented and the convergence speed compared for the different approaches. These methods can determine the optimal state and state/action value function and the optimal policy without requiring system models. (4 pages)

In this paper, reinforcement learning (RL) is employed to find a casual solution to the linear quadratic tracker (LQT) for continuous-time systems online in real time. Although several RL techniques are developed in the literature to solve the LQ regulator, to our knowledge, there is no rigorous result for using RL to solve the LQ tracker. This is mainly because of the requirement for computing a feedforward term in the tracker control which must be done in a noncausal manner backwards in time. To deal with this noncausality problem, an augmented system composed of the original system and the command generator dynamics is constructed, and an augmented LQT algebraic Riccati equation is derived for solving the LQT problem. In this formulation, one can apply RL techniques to solve the LQT problem, computing the feedforward term and the feedback term simultaneously online in real time. The convergence of the proposed online algorithms to the optimal control solution is verified. To show the efficiency of the proposed approach, a simulation example is provided.

In this paper, the problem about optimal iterative learning control of general nonlinear discrete-time systems has been studied. Based on contraction mapping principle, if some mild assumptions are given, we can obtain some sufficient conditions for the existence of optimal iterative learning control of general nonlinear discrete-time systems. These conditions are generalizations of the existence conditions of optimal iterative learning control of linear discrete-time systems.

Designing the optimal control for the practical systems is challenging due to the unknown system dynamics and unavoidable external disturbances. In this article, the $H_{\infty } $ optimal control problem is investigated for continuous-time linear systems with unknown dynamics. The existing reinforcement learning-based $H_{\infty } $ optimal control methods require persistence of excitation (PE) condition or data storage mechanism to guarantee the convergence of the algorithms. However, PE condition is hard to be monitored online and data storage mechanism requires to store huge amounts of past system data. In order to solve these problems, the initial excitation-based reinforcement learning algorithms are presented to learn the optimal control policy under an online-verifiable initial excitation condition. The properties of the initial excitation-based reinforcement learning algorithms are analyzed, which show that the presented algorithms converge to the optimum under the initial excitation condition. Numerical analysis is provided which demonstrates the correctness of the presented algorithms.

Integral Q-learning and explorized policy iteration for adaptive optimal control of continuous-time linear systems

To estimate the growth rate of matrix products $A_{n}··· A_{1}$ with factors from some set of matrices $\mathscr{A}$, such numeric quantities as the joint spectral radius $ρ(\mathscr{A})$ and the lower spectral radius $\mathord{\buildrel{\lower3pt\hbox{$\scriptscriptstyle\smile$}} \over \rho } (\mathscr{A})$ are traditionally used. The first of these quantities characterizes the maximum growth rate of the norms of the corresponding products, while the second one characterizes the minimal growth rate. In the theory of discrete-time linear switching systems, the inequality $ρ(\mathscr{A})<1$ serves as a criterion for the stability of a system, and the inequality $\mathord{\buildrel{\lower3pt\hbox{$\scriptscriptstyle\smile$}} \over \rho } (\mathscr{A})<1 $ as a criterion for stabilizability.Given a set $\mathscr{A}$ of $N×M$ matrices and a set $\mathscr{B}$ of $M×N$ matrices. Then, for matrix products $A_{n}B_{n}··· A_{1}B_{1}$ with factors $A_{i}∈\mathscr{A}$ and $B_{i}∈\mathscr{B}$, we introduce the quantities $μ(\mathscr{A},\mathscr{B})$ and $η(\mathscr{A},\mathscr{B})$, called the lower and upper minimax joint spectral radius of the pair $\{\mathscr{A},\mathscr{B}\}$, respectively, which characterize the maximum growth rate of the matrix products $A_{n}B_{n}··· A_{1}B_{1}$ over all sets of matrices $A_{i}∈\mathscr{A}$ and the minimal growth rate over all sets of matrices $B_{i}∈\mathscr{B}$. In this sense, the minimax joint spectral radii can be considered as generalizations of both the joint and lower spectral radii. As an application of the minimax joint spectral radii, it is shown how these quantities can be used to analyze the stabilizability of discrete-time linear switching control systems in the presence of uncontrolled external disturbances of the plant.

Model-free [formula omitted] control design for unknown linear discrete-time systems via Q-learning with LMI

This study is concerned with the robust energy-to-peak (l2–l∞) filtering problem for a class of uncertain linear discrete-time networked control systems with time-varying delays and randomly missing data. The system matrices are assumed to reside in a convex polytope, and the time-varying delays appearing in system states are assumed to lie in a given interval. Moreover, the random data missing is supposed to satisfy the Bernoulli random binary distribution. The authors' goal is to design a full-order filter such that the filtering error dynamic system is exponentially mean-square stable for all admissible time delays and randomly missing data while a prescribed l2–l∞ performance is achieved. Sufficient delay- and parameter-dependent conditions for the existence of the filter and for the solvability of the addressed problem are given in terms of a set of linear matrix inequalities. Finally, simulation examples and comparison studies are provided to demonstrate the effectiveness of the proposed approach.