Integral Q-learning and explorized policy iteration for adaptive optimal control of continuous-time linear systems

This paper investigates the infinite-horizon linear quadratic regulator (LQR) problem of continuous-time LTI systems with random gains imposed on the input channels. The main novelty of this work originates from the point of view that in networked control, designing the channels and controller jointly often leads to an easier problem and meanwhile achieves better performance than designing them separately. Specifically, we formulate the LQR problem with random input gains as a channel/controller co-design problem. Such co-design can be realized by the twist of channel resource allocation, i.e., the channel capacities can be allocated among the input channels subject to an overall capacity constraint. The LQR problem is shown to be attainable under such co-design if and only if a modified algebraic Riccati equation (MARE) has a mean-square (MS) stabilizing solution. The optimal controller is given by a linear state feedback associated with the MS stabilizing solution. Moreover, a Newton's type iteration is developed for the computation of the MS stabilizing solution to the MARE.

In this paper, a novel synchronous Q-learning method is proposed for solving discrete-time linear quadratic regulator (LQR) problems. To begin with, the Bellman equation corresponding to the optimal Q-function is reformulated into a consistency equation on the parameters of the optimal Q-function and the optimal controller. Then an actor-critic structure is introduced to learn the optimal Q-function and the optimal controller online in real time by using the state samples generated by the behavior policy. Particularly, the proposed synchronous Q-learning scheme simultaneously updates the Q-function approximation and the optimal controller approximation, rather than iterating between policy evaluation and policy improvement. The proposed control scheme is proved to be uniformly ultimately bounded (UUB) under appropriate learning rates, provided that certain persistence of excitation (PE) conditions are satisfied. Besides, the PE conditions can be easily met by injecting appropriate exploration noise into the behavior policy without causing any excitation noise bias. Finally, one simulation example is provided to verify the effectiveness of the proposed synchronous Q-learning method.

Robust controller design for fuzzy parametric uncertain systems: An optimal control approach

In this paper, an optimal tracking control approach based on adaptive dynamic programming (ADP) algorithm is proposed to solve the linear quadratic regulation (LQR) problems for unknown discrete-time systems in an online fashion. First, we convert the optimal tracking problem into designing infinite-horizon optimal regulator for the tracking error dynamics based on the system transformation. Then we expand the error state equation by the history data of control and state. The iterative ADP algorithm of policy iteration (PI) and value iteration (VI) are introduced to solve the value function of the controlled system. It is shown that the proposed ADP algorithm solves the LQR without requiring any knowledge of the system dynamics. The simulation results show the convergence and effectiveness of the proposed control scheme.

The authors present a genetic approach to the optimal linear quadratic regulator (LQR) problem. Design of the optimal LQR depends on choosing weighting matrices. One must carry out a trial-and-error process to choose the proper weighting matrices in general. It is not a trivial problem. They show a new possibility using a genetic algorithm (GA) to find the proper weighting matrices and hence to obtain the optimal control gain. They apply the approach to the attitude control of an inverted pendulum system, which is known as the LQR problem. They perform computer simulations on the system with a 50 cm long inverted pendulum and obtain good results.

This paper considers the linear quadratic regulator (LQR) problem for two-dimensional (2D) systems. The general 2D system is assumed to be represented by the Givone-Roesser model. In attempting to solve the standard 2D LQR problem, it is shown that the notion of diagonal causality arises naturally. The system under consideration is redefined accordingly and it is demonstrated that the new 2D system can be cast as a one-dimensional (1D) non-square descriptor system. Existing results in 1D non-square descriptor system theory are then called upon to solve the modified 2D LQR problem.

Frequency-domain stability conditions for asynchronously sampled decentralized LTI systems

In this paper, we propose a novel Q-learning method based on multirate generalized policy iteration (MGPI) for unknown discrete-time (DT) linear quadratic regulation (LQR) problems. Q-learning is an effective scheme for unknown dynamical systems because it does not require any knowledge of the system dynamics to solve optimal control problems. By applying the MGPI concept, which is an extension of basic GPI with multirate time horizon steps, a new Q-learning algorithm is proposed for solving the LQR problem. Further, it is proven that the proposed algorithm converges to an optimal solution i.e., it learns the optimal control policy iteratively using the states and the control-input information. Finally, we employ the two degree-of-freedom helicopter model to verify the effectiveness of the proposed method and investigate its convergence properties.

An optimal control law for networked control systems with a discrete-time linear time-invariant (LTI) system as plant and networks between sensor and controller as well as between controller and actuator is proposed. This controller is designed by solving an optimization problem that is a generalization of the optimization problem used to obtain the Linear Quadratic Regulator (LQR) for deterministic discrete-time LTI systems. The networks are represented by random delays and drop outs of transmitted data packets.

This study investigates characteristic root equivalency relations between commensurate order and integer order Linear Time Invariant (LTI) systems. Author introduces some useful properties of a special class of commensurate order systems, which is called characteristic root equivalency class of LTI systems. These properties present potential to facilitate design and analysis efforts of this class of commensurate order systems. In this sense, straightforward stability checking procedures and design approaches for commensurate order root equivalent systems of the first and second order LTI systems are demonstrated. Findings of the study are validated by illustrative examples.

This paper studies data-driven learning-based methods for the finite-horizon optimal control of linear time-varying discrete-time systems. First, a novel finite-horizon Policy Iteration (PI) method for linear time-varying discrete-time systems is presented. Its connections with existing infinite-horizon PI methods are discussed. Then, both data-driven off-policy PI and Value Iteration (VI) algorithms are derived to find approximate optimal controllers when the system dynamics is completely unknown. Under mild conditions, the proposed data-driven off-policy algorithms converge to the optimal solution. Finally, the effectiveness and feasibility of the developed methods are validated by a practical example of spacecraft attitude control.

A EROSPACE engineering applications greatly stimulated the development of optimal control theory during the 1950s and 1960s, where the objective was to drive the system states in such a way that some defined cost was minimized. This turned out to have very useful applications in the design of regulators (where some steady state is to be maintained) and in tracking control strategies (where some predetermined state trajectory is to be followed). Among such applications was the problem of optimal flight trajectories for aircraft and space vehicles. Linear optimal control theory in particular has been very well documented and widely applied, where the plant that is controlled is assumed linear and the feedback controller is constrained to be linear with respect to its input. However, the availability of powerful low-cost microprocessors has spurred great advantages in the theory and applications of nonlinear control. The competitive era of rapid technological change, particularly in aerospace exploration, now demands stringent accuracy and cost requirements in nonlinear control systems. This has motivated the rapid development of nonlinear control theory for application to challenging, complex, dynamical real-world problems, particularly those that bear major practical significance in aerospace, marine, and defense industries. Infinite-time horizon nonlinear optimal control (ITHNOC) presents a viable option for synthesizing stabilizing controllers for nonlinear systems by making a state-input tradeoff, where the objective is to minimize the cost given by a performance index. The original theory of nonlinear optimal control dates from the 1960s. Various theoretical and practical aspects of the problem have been addressed in the literature over the decades since. In particular, the continuous-time nonlinear deterministic optimal control problem associated with autonomous (time-invariant) nonlinear regulator systems that are affine (linear) in the controls has been studied by many authors. The long-established theory of optimal control offers quite mature and well-documented techniques for solving this control-affine nonlinear optimization problem, based on dynamic programming or calculus of variations, but their application is generally a very tedious task. Bellman’s dynamic programming approach reduces to solving a nonlinear first-order partial differential equation (PDE), expressed by the Hamilton–Jacobi–Bellman (HJB) equation. The solution to the HJB equation gives the optimal performance/cost value (or storage) function and determines an optimal control in feedback form under some smoothness assumptions. Alternatively, in the classical calculus of variations, optimal control problems can be characterized locally in terms of the Hamiltonian dynamics arising from Pontryagin’s minimum principle. These are the characteristic equations of the HJB PDE, which result in a nonlinear, constrained two-point boundary value problem (TPBVP) that, in general, can only be solved by successive approximation of the optimal control input using iterative numerical techniques for each set of initial conditions. Numerically, even though the nonlinear TPBVP is somewhat easier to solve than the HJBPDE, control signals can only be determined offline and are thus best suited for feedforward control of plants for which the state trajectories are known a priori. Therefore, contrary to the dynamic programming approach, the resultant control law is not generally in feedback form. Open-loop control, however, is sensitive to random disturbances and requires that the initial state be on the optimal trajectory. In contrast, nonlinear optimal feedback has inherent robustness properties (inherent in the sense that it is obtained by ignoring uncertainty and disturbances). The potential difficulty with the HJB approach is that no efficient algorithm is available to solve the PDE when it is nonlinear and the problem dimension is high, making it impossible to derive exact expressions for optimal controls for most nontrivial problems of interest. The optimal can only be computed in special cases, such as linear dynamics and quadratic cost, or very low-dimensional systems. In particular, if the plant is linear time invariant (LTI) and the (infinite-time) performance index is quadratic, then the corresponding HJB equation for this infamous linear-quadratic regulator (LQR) problem reduces to an algebraic Riccati equation (ARE). Contrary to the well-developed and widely applied theory and computational tools for theRiccati equation (for example, see [1]), theHJB equation is difficult, if not impossible, to solve for most practical applications. The exact solution for the optimal control policies is very complex

Pang, Bo Jiang, Zhong-PingThis chapter studies the robustnessRobustness of reinforcement learningReinforcement Learning (RL) for discrete-time linear stochastic systemsStochastic system with multiplicative noiseMultiplicative noise evolving in continuous state and action spaces. As one of the popular methods in reinforcement learningReinforcement Learning (RL), the robustnessRobustness of policy iterationPolicy iteration is a longstanding open problem for the stochastic linear quadratic regulator (LQR)Linear Quadratic Regulator (LQR) problem with multiplicative noiseMultiplicative noise. A solution in the spirit of input-to-state stabilityInput-to-state stability is given, guaranteeing that the solutions of the policy iterationPolicy iteration algorithm are bounded and enter a small neighborhood of the optimal solution, whenever the error in each iteration is bounded and small. In addition, a novel off-policyOff-policy multiple-trajectory optimistic least-squaresLeast-squares algorithm policy iterationPolicy iteration algorithm is proposed, to learn a near-optimal solution of the stochastic LQRLinear Quadratic Regulator (LQR) problem directly from online input/state data, without explicitly identifying the system matrices. The efficacy of the proposed algorithm is supported by rigorous convergence analysis and numerical results on a second-order example.

In this paper, the cooperative stabilization problem of linear time-invariant (LTI) systems is studied based on a class of distributed Luenberger observers. A given LTI system is estimated by a network of observers. In case of observability configuration or external attack, each observer has to possess different measurement output of the LTI system. Classical distributed control strategies cannot be directly used because they are assumed to obtain identical information. In view of this problem, a class of distributed observers are designed under an undirected graph, which can recover the state of the LTI system by limited local information. By using any single channel of the estimated state, a control input is proposed to stabilize the LTI system. By analysing the stability of an augmented system which involves the LTI system and the estimation error system, some necessary and sufficient conditions are derived for the existence of control gains that guarantee the stabilization of the LTI system. In addition, some algorithms are proposed to specify the control gains. Lastly, some simulation examples are given to verify the effectiveness of the theoretical result.

Within the framework of the factorisation approach, the author presents parameterisation methods for stabilising controllers. The parameterisations of this study are characterised by an ideal finitely generated by some strictly causal stable transfer functions. Several typical examples of the parameterisations are also presented, namely (i) all stabilising controllers such that their relative degrees are more than or equal to some fixed number in the continuous-time linear time-invariant (LTI) system model; (ii) all stabilising controllers such that they must have at least some fixed number of delay operators in the discrete-time LTI system model; (iii) all stabilising controllers such that they must have some delay operators in the multidimensional system model. The parameterisation method of this study is also applied to (iv) a system that is stabilisable but that does not admit a doubly coprime factorisation.