Optimal control over a lossy communication network based on linear predictive compensation

This paper investigates the infinite-horizon linear quadratic regulator (LQR) problem of continuous-time LTI systems with random gains imposed on the input channels. The main novelty of this work originates from the point of view that in networked control, designing the channels and controller jointly often leads to an easier problem and meanwhile achieves better performance than designing them separately. Specifically, we formulate the LQR problem with random input gains as a channel/controller co-design problem. Such co-design can be realized by the twist of channel resource allocation, i.e., the channel capacities can be allocated among the input channels subject to an overall capacity constraint. The LQR problem is shown to be attainable under such co-design if and only if a modified algebraic Riccati equation (MARE) has a mean-square (MS) stabilizing solution. The optimal controller is given by a linear state feedback associated with the MS stabilizing solution. Moreover, a Newton's type iteration is developed for the computation of the MS stabilizing solution to the MARE.

The authors present a genetic approach to the optimal linear quadratic regulator (LQR) problem. Design of the optimal LQR depends on choosing weighting matrices. One must carry out a trial-and-error process to choose the proper weighting matrices in general. It is not a trivial problem. They show a new possibility using a genetic algorithm (GA) to find the proper weighting matrices and hence to obtain the optimal control gain. They apply the approach to the attitude control of an inverted pendulum system, which is known as the LQR problem. They perform computer simulations on the system with a 50 cm long inverted pendulum and obtain good results.

This paper considers the linear quadratic regulator (LQR) problem for two-dimensional (2D) systems. The general 2D system is assumed to be represented by the Givone-Roesser model. In attempting to solve the standard 2D LQR problem, it is shown that the notion of diagonal causality arises naturally. The system under consideration is redefined accordingly and it is demonstrated that the new 2D system can be cast as a one-dimensional (1D) non-square descriptor system. Existing results in 1D non-square descriptor system theory are then called upon to solve the modified 2D LQR problem.

ABSTRACTThis work proposes a new technique to characterize the suboptimality of stabilizing control laws for the infinite‐horizon linear quadratic regulation problem, wherein the norm of the state feedback gain matrix is conditioned to be lesser than a given bound. In contrast to existing techniques where the upper bound on the cost is pre‐specified, this work proposes to compute an upper bound associated with the suboptimal control law(s). To derive the norm‐bounded state feedback gain matrix, sufficient conditions of Krotov global optimal control framework are analyzed for the constrained linear quadratic problem, which typically renders the optimal control problem non‐convex. To address this issue, suitable convex optimization problems are formulated to compute the upper bound independent of initial conditions. The effectiveness of the proposed technique and its comparison with state‐of‐the‐art methods are demonstrated through numerical examples.

In this paper, the Linear Quadratic Regulator Problem with a positivity constraint on the admissible control set is addressed. Necessary and sufficient conditions for optimality are presented in terms of inner products, projections on closed convex sets, Pontryagin's maximum principle and dynamic programming. The main results are concerned with smoothness of the optimal control and the value function. The maximum principle will be extended to the infinite horizon case. Based on these analytical methods, we propose a numerical algorithm for the computation of the optimal controls for the finite and infinite horizon problem. The numerical methods will be justified by convergence properties between the finite and infinite horizon case on one side and discretized optimal controls and the true optimal control on the other.

A EROSPACE engineering applications greatly stimulated the development of optimal control theory during the 1950s and 1960s, where the objective was to drive the system states in such a way that some defined cost was minimized. This turned out to have very useful applications in the design of regulators (where some steady state is to be maintained) and in tracking control strategies (where some predetermined state trajectory is to be followed). Among such applications was the problem of optimal flight trajectories for aircraft and space vehicles. Linear optimal control theory in particular has been very well documented and widely applied, where the plant that is controlled is assumed linear and the feedback controller is constrained to be linear with respect to its input. However, the availability of powerful low-cost microprocessors has spurred great advantages in the theory and applications of nonlinear control. The competitive era of rapid technological change, particularly in aerospace exploration, now demands stringent accuracy and cost requirements in nonlinear control systems. This has motivated the rapid development of nonlinear control theory for application to challenging, complex, dynamical real-world problems, particularly those that bear major practical significance in aerospace, marine, and defense industries. Infinite-time horizon nonlinear optimal control (ITHNOC) presents a viable option for synthesizing stabilizing controllers for nonlinear systems by making a state-input tradeoff, where the objective is to minimize the cost given by a performance index. The original theory of nonlinear optimal control dates from the 1960s. Various theoretical and practical aspects of the problem have been addressed in the literature over the decades since. In particular, the continuous-time nonlinear deterministic optimal control problem associated with autonomous (time-invariant) nonlinear regulator systems that are affine (linear) in the controls has been studied by many authors. The long-established theory of optimal control offers quite mature and well-documented techniques for solving this control-affine nonlinear optimization problem, based on dynamic programming or calculus of variations, but their application is generally a very tedious task. Bellman’s dynamic programming approach reduces to solving a nonlinear first-order partial differential equation (PDE), expressed by the Hamilton–Jacobi–Bellman (HJB) equation. The solution to the HJB equation gives the optimal performance/cost value (or storage) function and determines an optimal control in feedback form under some smoothness assumptions. Alternatively, in the classical calculus of variations, optimal control problems can be characterized locally in terms of the Hamiltonian dynamics arising from Pontryagin’s minimum principle. These are the characteristic equations of the HJB PDE, which result in a nonlinear, constrained two-point boundary value problem (TPBVP) that, in general, can only be solved by successive approximation of the optimal control input using iterative numerical techniques for each set of initial conditions. Numerically, even though the nonlinear TPBVP is somewhat easier to solve than the HJBPDE, control signals can only be determined offline and are thus best suited for feedforward control of plants for which the state trajectories are known a priori. Therefore, contrary to the dynamic programming approach, the resultant control law is not generally in feedback form. Open-loop control, however, is sensitive to random disturbances and requires that the initial state be on the optimal trajectory. In contrast, nonlinear optimal feedback has inherent robustness properties (inherent in the sense that it is obtained by ignoring uncertainty and disturbances). The potential difficulty with the HJB approach is that no efficient algorithm is available to solve the PDE when it is nonlinear and the problem dimension is high, making it impossible to derive exact expressions for optimal controls for most nontrivial problems of interest. The optimal can only be computed in special cases, such as linear dynamics and quadratic cost, or very low-dimensional systems. In particular, if the plant is linear time invariant (LTI) and the (infinite-time) performance index is quadratic, then the corresponding HJB equation for this infamous linear-quadratic regulator (LQR) problem reduces to an algebraic Riccati equation (ARE). Contrary to the well-developed and widely applied theory and computational tools for theRiccati equation (for example, see [1]), theHJB equation is difficult, if not impossible, to solve for most practical applications. The exact solution for the optimal control policies is very complex

A Class of Linear Quadratic Gaussian Hybrid Optimal Control Problems with Realization–Independent Riccati Equations

In this paper, an optimal tracking control approach based on adaptive dynamic programming (ADP) algorithm is proposed to solve the linear quadratic regulation (LQR) problems for unknown discrete-time systems in an online fashion. First, we convert the optimal tracking problem into designing infinite-horizon optimal regulator for the tracking error dynamics based on the system transformation. Then we expand the error state equation by the history data of control and state. The iterative ADP algorithm of policy iteration (PI) and value iteration (VI) are introduced to solve the value function of the controlled system. It is shown that the proposed ADP algorithm solves the LQR without requiring any knowledge of the system dynamics. The simulation results show the convergence and effectiveness of the proposed control scheme.

Linear Quadratic Optimal Control of a Wave Equation with Boundary Damping and Pointwise Control Input

A linear quadratic regulator problem where a small parameter μ2multiplies the control cost is considered. As \mu \rightarrow 0 , it exhibits a dynamic behavior with multiple time-scales, whereas for \mu = 0 , it is a singular control problem. At first a multiparameter singularly perturbed model of the given problem is constructed by transferring the singularity that exists in the performance index to the dynamic equations. Then the problem is decomposed into several subproblems of minimal order, each pertaining to only one time-scale. The method developed permits under a single framework the characterization of asymptotic behavior of optimal closed-loop poles, state and control trajectories, performance index, and optimal transfer function as \mu \rightarrow 0 . Moreover, the asymptotic solution brings into focus 1) the nature of singular control and its connection to transmission zeros and 2) the presence of various types of impulses and other higher order distributions in both state and control. The approach unifies and extends significantly several ideas previously somewhat disjoint.

In this note, an explicit solution to the multivariable discrete linear quadratic (LQ) regulation problem is obtained in the limiting singular case where the input weighting matrix tends to zero. Such a solution follows from a suitable spectral factorization of the input spectrum density matrix under the assumption that the system is stabilizable and detectable, and its transfer function matrix is of full rank. The suitable spectral factor is shown to be the product of the system minimum-phase image and its unitary interactor matrix. The unitary interactor matrix defined here is a special case of the nilpotent interactor matrix defined in [5]. © 1992 IEEE

In this paper, a novel synchronous Q-learning method is proposed for solving discrete-time linear quadratic regulator (LQR) problems. To begin with, the Bellman equation corresponding to the optimal Q-function is reformulated into a consistency equation on the parameters of the optimal Q-function and the optimal controller. Then an actor-critic structure is introduced to learn the optimal Q-function and the optimal controller online in real time by using the state samples generated by the behavior policy. Particularly, the proposed synchronous Q-learning scheme simultaneously updates the Q-function approximation and the optimal controller approximation, rather than iterating between policy evaluation and policy improvement. The proposed control scheme is proved to be uniformly ultimately bounded (UUB) under appropriate learning rates, provided that certain persistence of excitation (PE) conditions are satisfied. Besides, the PE conditions can be easily met by injecting appropriate exploration noise into the behavior policy without causing any excitation noise bias. Finally, one simulation example is provided to verify the effectiveness of the proposed synchronous Q-learning method.

In this paper, we propose a method for estimating the algebraic Riccati equation (ARE) with respect to an unknown discrete-time system from the system state and input observation. The inverse optimal control (IOC) problem asks, “What objective function is optimized by a given control system?” The inverse linear quadratic regulator (ILQR) problem is an IOC problem that assumes a linear system and quadratic objective function. The ILQR problem can be solved by solving a linear matrix inequality that contains the ARE. However, the system model is required to obtain the ARE, and it is often unknown in fields in which the IOC problem occurs, for example, biological system analysis. Our method directly estimates the ARE from the observation data without identifying the system. This feature enables us to economize the observation data using prior information about the objective function. We provide a data condition that is sufficient for our method to estimate the ARE. We conducted a numerical experiment to demonstrate that our method can estimate the ARE with less data than system identification if the prior information is sufficient.

This paper investigates and discusses the method of dynamic programming in solving Bolza's cost form of Linear Quadratic Regulator Problems (LQRP). It is the desire of the authors of this paper to experiment numerically the solution of this class of problem using dynamic programming to solve for the optimal controls and the trajectories compared with other numerical methods with a view to further improving the results. The method uses the principle of optimality to reduce mathematically the number of calculations required to determine the optimal control law as well as the corresponding optimal cost functional.

Pang, Bo Jiang, Zhong-PingThis chapter studies the robustnessRobustness of reinforcement learningReinforcement Learning (RL) for discrete-time linear stochastic systemsStochastic system with multiplicative noiseMultiplicative noise evolving in continuous state and action spaces. As one of the popular methods in reinforcement learningReinforcement Learning (RL), the robustnessRobustness of policy iterationPolicy iteration is a longstanding open problem for the stochastic linear quadratic regulator (LQR)Linear Quadratic Regulator (LQR) problem with multiplicative noiseMultiplicative noise. A solution in the spirit of input-to-state stabilityInput-to-state stability is given, guaranteeing that the solutions of the policy iterationPolicy iteration algorithm are bounded and enter a small neighborhood of the optimal solution, whenever the error in each iteration is bounded and small. In addition, a novel off-policyOff-policy multiple-trajectory optimistic least-squaresLeast-squares algorithm policy iterationPolicy iteration algorithm is proposed, to learn a near-optimal solution of the stochastic LQRLinear Quadratic Regulator (LQR) problem directly from online input/state data, without explicitly identifying the system matrices. The efficacy of the proposed algorithm is supported by rigorous convergence analysis and numerical results on a second-order example.