An Efficient Policy Iteration Algorithm for Dynamic Programming Equations

This paper studies data-driven learning-based methods for the finite-horizon optimal control of linear time-varying discrete-time systems. First, a novel finite-horizon Policy Iteration (PI) method for linear time-varying discrete-time systems is presented. Its connections with existing infinite-horizon PI methods are discussed. Then, both data-driven off-policy PI and Value Iteration (VI) algorithms are derived to find approximate optimal controllers when the system dynamics is completely unknown. Under mild conditions, the proposed data-driven off-policy algorithms converge to the optimal solution. Finally, the effectiveness and feasibility of the developed methods are validated by a practical example of spacecraft attitude control.

Policy iteration and value iteration are at the core of many (approximate) dynamic programming methods. For Markov Decision Processes with finite state and action spaces, we show that they are instances of semismooth Newton-type methods for solving the Bellman equation. In particular, we prove that policy iteration is equivalent to the exact semismooth Newton method and enjoys a local quadratic convergence rate. This finding is corroborated by extensive numerical evidence in the fields of control and operations research, which confirms that policy iteration generally requires relatively few iterations to achieve convergence even in presence of a large number of admissible policies. We then show that value iteration is an instance of the fixed-point iteration method and develop a novel locally accelerated version of value iteration with global convergence guarantees and negligible extra computational costs.

Policy iteration, as an adaptive/approximate dynamic programming-based approach for optimal control is investigated. The context is optimal control of discrete-time nonlinear dynamics with undiscounted cost functions. Convergence of the learning iterations and uniqueness of the solution to the corresponding Bellman equation are established, leading to the optimality of the limit function, i.e., the learning results. Moreover, given the faster convergence of the learning under policy iteration compared with value iteration-based learning algorithms, some theoretical results are developed which prove that starting with a similar initial guess, policy iteration will not converge slower than value iteration. Finally, some numerical analyses are presented to demonstrate the results in practice.

We consider challenging dynamic programming models where the associated Bellman equation, and the value and policy iteration algorithms commonly exhibit complex and even pathological behavior. Our analysis is based on the new notion of regular policies. These are policies that are well-behaved with respect to value and policy iteration, and are patterned after proper policies, which are central in the theory of stochastic shortest path problems. We show that the optimal cost function over regular policies may have favorable value and policy iteration properties, which the optimal cost function over all policies need not have. We accordingly develop a unifying methodology to address long standing analytical and algorithmic issues in broad classes of undiscounted models, including stochastic and minimax shortest path problems, as well as positive cost, negative cost, risk-sensitive, and multiplicative cost problems.

This paper considers the maximization of certain equivalent reward generated by a Markov decision process with constant risk sensitivity. First, value iteration is used to optimize possibly time-varying processes of finite duration. Then a policy iteration procedure is developed to find the stationary policy with highest certain equivalent gain for the infinite duration case. A simple example demonstrates both procedures.

In this paper, we present an iterative technique for deriving the maximal solution of a set of discrete-time coupled algebraic Riccati equations, based on temporal difference methods, which are related to the optimal control of Markovian jump linear systems and have been studied extensively over the last few years. We trace a parallel with the theory of temporal difference algorithms for Markovian decision processes to develop a λ-policy iteration like algorithm for the maximal solution of these equations. For the special cases in which λ=0 and λ=1, we have the situation in which the algorithm reduces to the iterations of the Riccati difference equations (value iteration) and quasilinearization method (policy iteration), respectively. The advantage of the proposed method is that an appropriate choice of λ between 0 and 1 can speed up the convergence of the policy evaluation step of the policy iteration method by using value iteration.

The goal of this article is to investigate new and simple convergence analysis of dynamic programming for the linear–quadratic regulator problem of discrete-time linear time-invariant systems. In particular, bounds on errors are given in terms of both matrix inequalities and matrix norm. Under a mild assumption on the initial parameter, we prove that the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$Q$</tex-math></inline-formula> -value iteration exponentially converges to the optimal solution. Moreover, a global asymptotic convergence is also presented. These results are then extended to the policy iteration. We prove that in contrast to the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$Q$</tex-math></inline-formula> -value iteration, the policy iteration always converges exponentially fast. An example is given to illustrate the results.

In this article, a data-driven distributed control method is proposed to solve the cooperative optimal output regulation problem of leader-follower multiagent systems. Different from traditional studies on cooperative output regulation, a distributed adaptive internal model is originally developed, which includes a distributed internal model and a distributed observer to estimate the leader's dynamics. Without relying on the dynamics of multiagent systems, we have proposed two reinforcement learning algorithms, policy iteration and value iteration, to learn the optimal controller through online input and state data, and estimated values of the leader's state. By combining these methods, we have established a basis for connecting data-distributed control methods with adaptive dynamic programming approaches in general since these are the theoretical foundation from which they are built.

In this paper, an optimal tracking control approach based on adaptive dynamic programming (ADP) algorithm is proposed to solve the linear quadratic regulation (LQR) problems for unknown discrete-time systems in an online fashion. First, we convert the optimal tracking problem into designing infinite-horizon optimal regulator for the tracking error dynamics based on the system transformation. Then we expand the error state equation by the history data of control and state. The iterative ADP algorithm of policy iteration (PI) and value iteration (VI) are introduced to solve the value function of the controlled system. It is shown that the proposed ADP algorithm solves the LQR without requiring any knowledge of the system dynamics. The simulation results show the convergence and effectiveness of the proposed control scheme.

Output-feedback adaptive optimal control of interconnected systems based on robust adaptive dynamic programming

In this paper, the Bellman equation is used to solve the optimal adaptive control of quantized linear discrete-time system with unknown dynamics. To mitigate the effect of the quantization errors, the dynamics of the quantization error bound and an update law for tuning the range of the dynamic quantizer are derived. Subsequently, by using adaptive dynamic programming technique, the infinite horizon optimal regulation problem of the uncertain quantized linear discrete-time system is solved in a forward-in-time manner without using value and/or policy iterations. The asymptotic stability of the closed-loop system is verified by standard Lyapunov stability approach in the presence of state and input quantizers. The effectiveness of the proposed method is verified via simulation.

In this article, the λ -policy iteration ( λ -PI) method for the optimal control problem of discrete-time linear systems is reconsidered and restated from a novel aspect. First, the traditional λ -PI method is recalled, and some new properties of the traditional λ -PI are proposed. Based on these new properties, a modified λ -PI algorithm is introduced with its convergence proven. Compared with the existing results, the initial condition is further relaxed. The data-driven implementation is then constructed with a new matrix rank condition for verifying the feasibility of the proposed data-driven implementation. A simulation example verifies the effectiveness of the proposed method.

In this chapter, by using neural network (NN)-based online learning optimal control approach, a decentralized control strategy is developed to stabilize a class of continuous-time large-scale interconnected nonlinear systems . It is proven that the decentralized control strategy of the overall system can be established by adding appropriate feedback gains to the optimal control laws of the isolated subsystems . Then, an online policy iteration (PI) algorithm is developed to solve the Hamilton–Jacobi–Bellman equations related to the optimal control problem. By constructing a set of critic NNs, the cost functions can be obtained by NN approximation, followed by the control laws. Furthermore, as a generalization, an NN-based decentralized control law is developed to stabilize the large-scale interconnected nonlinear systems using an online model-free integral PI algorithm. The model-free PI approach can solve the decentralized control problem for large-scale interconnected nonlinear systems with unknown dynamics. Finally, two simulation examples are provided to illustrate the effectiveness of the present decentralized control scheme.

We provide a formal description of the discounted reward MDP framework in Chap. 1, including both the finite- and the infinite-horizon settings and summarizing the associated optimality equations. We then present the well-known exact solution algorithms, value iteration and policy iteration, and outline a framework of rolling-horizon control (also called receding-horizon control) as an approximate solution methodology for solving MDPs, in conjunction with simulation-based approaches covered later in the book. We conclude with a brief survey of other recently proposed MDP solution techniques designed to break the curse of dimensionality.KeywordsOptimal PolicyMarkov Decision ProcessPolicy IterationApproximate Dynamic ProgrammingDynamic Programming EquationThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

The convergence evaluation of the discrete linear quadratic regulator (DLQR) to map the Z-stable plane, is the main target of this research that is oriented to the development of tuning method for multivariable systems. The tuning procedures is based on strategies to select the weighting matrices and dynamic programming. The solutions of DLQR are presented, since Bellman formulations until Riccati and Lyapunov recurrences and are based on the Generalized Policy Iteration, Policy Iteration and Value Iteration. The algorithms and the proposed heuristic method are developed from Riccati and Lyapunov recurrences and are implemented to map the closed loop dynamic eingen values in the Z plane. A fourth order model is used to evaluate the convergence and its ability to map the plan Z by selection of the weighting matrices of Optimal Control.