Associative reinforcement learning for discrete-time optimal control

An ADDHP-based Q-learning algorithm for optimal tracking control of linear discrete-time systems with unknown dynamics

In this paper, the problem about optimal iterative learning control of general nonlinear discrete-time systems has been studied. Based on contraction mapping principle, if some mild assumptions are given, we can obtain some sufficient conditions for the existence of optimal iterative learning control of general nonlinear discrete-time systems. These conditions are generalizations of the existence conditions of optimal iterative learning control of linear discrete-time systems.

Methods from learning theory are used in the state space of linear dynamical and control systems in order to estimate relevant matrices and some relevant quantities such as the topological entropy. An application to stabilization via algebraic Riccati equations is included by viewing a control system as an autonomous system in an extended space of states and control inputs. Kernel methods are the main techniques used in this paper and the approach is illustrated via a series of numerical examples. The advantage of using kernel methods is that they allow to perform function approximation from data and, as illustrated in this paper, allow to approximate linear discrete-time autonomous and control systems from data.

1 A Survey on Computational Optimal Control.- Issues in the Direct Transcription of Optimal Control Problems to Sparse Nonlinear Programs.- Optimization in Control of Robots.- Large-scale SQP Methods and their Application in Trajectory Optimization.- Solving Optimal Control and Pursuit-Evasion Game Problems of High Complexity.- 2 Theoretical Aspects of Optimal Control and Nonlinear Programming.- Continuation Methods In Boundary Value Problems.- Second Order Optimality Conditions for Singular Extremals.- Synthesis of Adaptive Optimal Controls for Linear Dynamic Systems.- Control Applications of Reduced SQP Methods.- Time Optimal Control of Mechanical Systems.- 3 Algorithms for Optimal Control Calculations.- Second Order Algorithm for Time Optimal Control of a Linear System.- An SQP-type Solution Method for Constrained Discrete-Time Optimal Control Problems.- Numerical Methods for Solving Differential Games, Prospective Applications to Technical Problems.- Construction of the Optimal Feedback Controller for Constrained Optimal Control Problems with Unknown Disturbances.- Repetitive Optimization for Predictive Control of Dynamic Systems under Uncertainty.- Optimal Control of Multistage Systems Described by High-Index Differential-Algebraic Equations.- A New Class of a High Order Interior Point Method for the Solution of Convex Semiinfinite Optimization Problems.- A Structured Interior Point SQP Method for Nonlinear Optimal Control Problems.- 4 Software for Optimal Control Calculations.- Automated Approach for Optimizing Dynamic Systems.- ANDECS: A Computation Environment for Control Applications of Optimization.- Application of Automatic Differentiation to Optimal Control Problems.- OCCAL: A mixed symbolic-numeric Optimal Control CALculator.- 5 Applications of Optimal Control.- A Robotic Satellite with Simplified Design.- Nonlinear Control under Constraints of a Biological System.- An Object-Oriented Approach to Optimally Describe and Specify a SCADA System Applied to a Power Network.- Near-Optimal Flight Trajectories Generated by Neural Networks.- Performance of a Feedback Method with Respect to Changes in the Air-Density during the Ascent of a Two-Stage-To-Orbit Vehicle.- Linear Optimal Control for Reentry Flight.- Steady-State Modelling of Turbine Engine with Controllers.- Shortest Paths for Satellite Mounted Robot Manipulators.- Optimal Control of the Industrial Robot Manutec r3.

In this paper, a novel synchronous Q-learning method is proposed for solving discrete-time linear quadratic regulator (LQR) problems. To begin with, the Bellman equation corresponding to the optimal Q-function is reformulated into a consistency equation on the parameters of the optimal Q-function and the optimal controller. Then an actor-critic structure is introduced to learn the optimal Q-function and the optimal controller online in real time by using the state samples generated by the behavior policy. Particularly, the proposed synchronous Q-learning scheme simultaneously updates the Q-function approximation and the optimal controller approximation, rather than iterating between policy evaluation and policy improvement. The proposed control scheme is proved to be uniformly ultimately bounded (UUB) under appropriate learning rates, provided that certain persistence of excitation (PE) conditions are satisfied. Besides, the PE conditions can be easily met by injecting appropriate exploration noise into the behavior policy without causing any excitation noise bias. Finally, one simulation example is provided to verify the effectiveness of the proposed synchronous Q-learning method.

This paper is concerned with the structural controllability analysis for discrete-time linear control systems with time-delay. By adding virtual delay nodes, the linear systems with time-delay are transformed into corresponding linear systems without time-delay, and the structural controllability of them is equivalent. That is to say, the time-delay does not affect or change the controllability of the systems. Several examples are also presented to illustrate the theoretical results.

To estimate the growth rate of matrix products $A_{n}··· A_{1}$ with factors from some set of matrices $\mathscr{A}$, such numeric quantities as the joint spectral radius $ρ(\mathscr{A})$ and the lower spectral radius $\mathord{\buildrel{\lower3pt\hbox{$\scriptscriptstyle\smile$}} \over \rho } (\mathscr{A})$ are traditionally used. The first of these quantities characterizes the maximum growth rate of the norms of the corresponding products, while the second one characterizes the minimal growth rate. In the theory of discrete-time linear switching systems, the inequality $ρ(\mathscr{A})<1$ serves as a criterion for the stability of a system, and the inequality $\mathord{\buildrel{\lower3pt\hbox{$\scriptscriptstyle\smile$}} \over \rho } (\mathscr{A})<1 $ as a criterion for stabilizability.Given a set $\mathscr{A}$ of $N×M$ matrices and a set $\mathscr{B}$ of $M×N$ matrices. Then, for matrix products $A_{n}B_{n}··· A_{1}B_{1}$ with factors $A_{i}∈\mathscr{A}$ and $B_{i}∈\mathscr{B}$, we introduce the quantities $μ(\mathscr{A},\mathscr{B})$ and $η(\mathscr{A},\mathscr{B})$, called the lower and upper minimax joint spectral radius of the pair $\{\mathscr{A},\mathscr{B}\}$, respectively, which characterize the maximum growth rate of the matrix products $A_{n}B_{n}··· A_{1}B_{1}$ over all sets of matrices $A_{i}∈\mathscr{A}$ and the minimal growth rate over all sets of matrices $B_{i}∈\mathscr{B}$. In this sense, the minimax joint spectral radii can be considered as generalizations of both the joint and lower spectral radii. As an application of the minimax joint spectral radii, it is shown how these quantities can be used to analyze the stabilizability of discrete-time linear switching control systems in the presence of uncontrolled external disturbances of the plant.

Periodic event-triggered state-feedback control for discrete-time linear systems

Stochastic optimal control of unknown linear networked control system in the presence of random delays and packet losses

Model-free finite-horizon optimal tracking control of discrete-time linear systems

In this note, we consider the finite-horizon quadratic optimal control problem of discrete-time Markovian jump linear systems driven by a wide sense white noise sequence. We assume that the output variable and the jump parameters are available to the controller. It is desired to design a dynamic Markovian jump controller such that the closed-loop system minimizes the quadratic functional cost of the system over a finite horizon period of time. As in the case with no jumps, we show that an optimal controller can be obtained from two coupled Riccati difference equations, one associated to the optimal control problem when the state variable is available, and the other one associated to the optimal filtering problem. This is a principle of separation for the finite horizon quadratic optimal control problem for discrete-time Markovian jump linear systems. When there is only one mode of operation our results coincide with the traditional separation principle for the linear quadratic Gaussian control of discrete-time linear systems.

Estimation and Stochastic Control for Linear Infinite-Dimensional Systems

While a large amount of research over the past two decades has focused on discrete abstractions of infinite-state dynamical systems, many structural and algorithmic details of these abstractions remain unknown. To clarify the computational resources needed to perform discrete abstractions, this paper examines the algorithmic properties of an existing method for deriving finite-state systems that are bisimilar to linear discrete-time control systems. We explicitly find the structure of the finite-state system, show that it can be enormous compared to the original linear system, and give conditions to guarantee that the finite-state system is reasonably sized and efficiently computable. Though constructing the finite-state system is generally impractical, we see that special cases could be amenable to satisfiability based verification techniques.

The problem of optimal control of time-varying linear singular systems with quadratic performance index has been studied using the Runge–Kutta–Butcher algorithm. The results obtained using the Runge–Kutta (RK) method based on the arithmetic mean (RKAM) and the RK–Butcher algorithms are compared with the exact solutions of the time-varying optimal control of linear singular systems. It is observed that the result obtained using the RK–Butcher algorithm is closer to the true solution of the problem. Stability regions for the RKAM algorithm, the single-term Walsh series method and the RK–Butcher algorithms are presented. Error graphs for the simulated results and exact solutions are presented in graphical form to highlight the efficiency of the RK–Butcher algorithm. This algorithm can easily be implemented using a digital computer. An additional advantage of this method is that the solution can be obtained for any length of time for this type of optimal control of time-varying linear singular systems.

A general result in stochastic optimal control of nonlinear discrete-time systems with quadratic performance criteria