Regular Policies in Abstract Dynamic Programming

This paper studies data-driven learning-based methods for the finite-horizon optimal control of linear time-varying discrete-time systems. First, a novel finite-horizon Policy Iteration (PI) method for linear time-varying discrete-time systems is presented. Its connections with existing infinite-horizon PI methods are discussed. Then, both data-driven off-policy PI and Value Iteration (VI) algorithms are derived to find approximate optimal controllers when the system dynamics is completely unknown. Under mild conditions, the proposed data-driven off-policy algorithms converge to the optimal solution. Finally, the effectiveness and feasibility of the developed methods are validated by a practical example of spacecraft attitude control.

ABSTRACTIn this article, the initial excitation‐based inverse reinforcement learning methods are presented for continuous‐time linear non‐zero‐sum games. The policy iteration and value iteration algorithms are presented for the inverse reinforcement learning problems, and an online‐verifiable initial excitation condition is given to guarantee the convergence of the presented algorithms. Comparing with the traditional inverse reinforcement learning algorithms for linear non‐zero‐sum games, the presented algorithms relax the requirement on data‐storage mechanism. Furthermore, the requirement on the initial stabilizing state feedback matrices is relaxed in the presented initial excitation‐based value iteration algorithm. The properties of the presented initial excitation‐based policy iteration and value iteration algorithms are analyzed. Simulation results show the efficiency of the presented algorithms.

Policy iteration and value iteration are at the core of many (approximate) dynamic programming methods. For Markov Decision Processes with finite state and action spaces, we show that they are instances of semismooth Newton-type methods for solving the Bellman equation. In particular, we prove that policy iteration is equivalent to the exact semismooth Newton method and enjoys a local quadratic convergence rate. This finding is corroborated by extensive numerical evidence in the fields of control and operations research, which confirms that policy iteration generally requires relatively few iterations to achieve convergence even in presence of a large number of admissible policies. We then show that value iteration is an instance of the fixed-point iteration method and develop a novel locally accelerated version of value iteration with global convergence guarantees and negligible extra computational costs.

We present an accelerated algorithm for the solution of static Hamilton--Jacobi--Bellman equations related to optimal control problems. Our scheme is based on a classic policy iteration procedure, which is known to have superlinear convergence in many relevant cases provided the initial guess is sufficiently close to the solution. This limitation often degenerates into a behavior similar to a value iteration method, with an increased computation time. The new scheme circumvents this problem by combining the advantages of both algorithms with an efficient coupling. The method starts with a coarse-mesh value iteration phase and then switches to a fine-mesh policy iteration procedure when a certain error threshold is reached. A delicate point is to determine this threshold in order to avoid cumbersome computations with the value iteration and at the same time to ensure the convergence of the policy iteration method to the optimal solution. We analyze the methods and efficient coupling in a number of examples in different dimensions, illustrating their properties.

The convergence evaluation of the discrete linear quadratic regulator (DLQR) to map the Z-stable plane, is the main target of this research that is oriented to the development of tuning method for multivariable systems. The tuning procedures is based on strategies to select the weighting matrices and dynamic programming. The solutions of DLQR are presented, since Bellman formulations until Riccati and Lyapunov recurrences and are based on the Generalized Policy Iteration, Policy Iteration and Value Iteration. The algorithms and the proposed heuristic method are developed from Riccati and Lyapunov recurrences and are implemented to map the closed loop dynamic eingen values in the Z plane. A fourth order model is used to evaluate the convergence and its ability to map the plan Z by selection of the weighting matrices of Optimal Control.

Policy iteration, as an adaptive/approximate dynamic programming-based approach for optimal control is investigated. The context is optimal control of discrete-time nonlinear dynamics with undiscounted cost functions. Convergence of the learning iterations and uniqueness of the solution to the corresponding Bellman equation are established, leading to the optimality of the limit function, i.e., the learning results. Moreover, given the faster convergence of the learning under policy iteration compared with value iteration-based learning algorithms, some theoretical results are developed which prove that starting with a similar initial guess, policy iteration will not converge slower than value iteration. Finally, some numerical analyses are presented to demonstrate the results in practice.

In order to simplify computational methods based on dynamic programming, a relaxed procedure based on upper and lower bounds of the optimal cost was recently introduced. The convergence properties of this procedure are analysed here. In particular, it is shown that the computational effort in finding an approximately optimal control law by relaxed value iteration is related to the polynomial degree that is needed to approximate the optimal cost. This gives a rigorous foundation for the claim that the search for optimal control laws requires complex computations only if the optimal cost function is complex. A computational example is given for switching control on a graph with 60 nodes, 120 edges and 30 continuous states.

The goal of this article is to investigate new and simple convergence analysis of dynamic programming for the linear–quadratic regulator problem of discrete-time linear time-invariant systems. In particular, bounds on errors are given in terms of both matrix inequalities and matrix norm. Under a mild assumption on the initial parameter, we prove that the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$Q$</tex-math></inline-formula> -value iteration exponentially converges to the optimal solution. Moreover, a global asymptotic convergence is also presented. These results are then extended to the policy iteration. We prove that in contrast to the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$Q$</tex-math></inline-formula> -value iteration, the policy iteration always converges exponentially fast. An example is given to illustrate the results.

We provide a formal description of the discounted reward MDP framework in Chap. 1, including both the finite- and the infinite-horizon settings and summarizing the associated optimality equations. We then present the well-known exact solution algorithms, value iteration and policy iteration, and outline a framework of rolling-horizon control (also called receding-horizon control) as an approximate solution methodology for solving MDPs, in conjunction with simulation-based approaches covered later in the book. We conclude with a brief survey of other recently proposed MDP solution techniques designed to break the curse of dimensionality.KeywordsOptimal PolicyMarkov Decision ProcessPolicy IterationApproximate Dynamic ProgrammingDynamic Programming EquationThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

In this paper, we propose a new lower approximation scheme for POMDP with discounted and average cost criterion. The approximating functions are determined by their values at a finite number of belief points, and can be computed efficiently using value iteration algorithms for finite-state MDP. While for discounted problems several lower approximation schemes have been proposed earlier, ours seems the first of its kind for average cost problems. We focus primarily on the average cost case, and we show that the corresponding approximation can be computed efficiently using multi-chain algorithms for finite-state MDP. We give a preliminary analysis showing that regardless of the existence of the optimal average cost J* in the POMDP, the approximation obtained is a lower bound of the liminf optimal average cost function, and can also be used to calculate an upper bound on the limsup optimal average cost function, as well as bounds on the cost of executing the stationary policy associated with the approximation. We show the convergence of the cost approximation, when the optimal average cost is constant and the optimal differential cost is continuous.

We consider a totally asynchronous stochastic approximation algorithm, Q-learning, for solving finite space stochastic shortest path (SSP) problems, which are undiscounted, total cost Markov decision processes with an absorbing and cost-free state. For the most commonly used SSP models, existing convergence proofs assume that the sequence of Q-learning iterates is bounded with probability one, or some other condition that guarantees boundedness. We prove that the sequence of iterates is naturally bounded with probability one, thus furnishing the boundedness condition in the convergence proof by Tsitsiklis [Tsitsiklis JN (1994) Asynchronous stochastic approximation and Q-learning. Machine Learn. 16:185–202] and establishing completely the convergence of Q-learning for these SSP models.

In this paper, we present an iterative technique for deriving the maximal solution of a set of discrete-time coupled algebraic Riccati equations, based on temporal difference methods, which are related to the optimal control of Markovian jump linear systems and have been studied extensively over the last few years. We trace a parallel with the theory of temporal difference algorithms for Markovian decision processes to develop a λ-policy iteration like algorithm for the maximal solution of these equations. For the special cases in which λ=0 and λ=1, we have the situation in which the algorithm reduces to the iterations of the Riccati difference equations (value iteration) and quasilinearization method (policy iteration), respectively. The advantage of the proposed method is that an appropriate choice of λ between 0 and 1 can speed up the convergence of the policy evaluation step of the policy iteration method by using value iteration.

Leasing computational resources for on-demand computing is now a viable option for providers of network services. Temporary spikes or lulls in demand for a service can be accommodated by flexible leasing arrangements. From the service provider's perspective, the problem is how many resources to lease and for how long. In this paper we formulate and solve the resource leasing problem for the case of a single service. The objective is to minimize the cost of leasing resources while still maintaining an adequate quality of service, which we measure by the average wait time of requests. Demand for the service and execution times of service requests are modeled as random variables. The problem is formulated as a continuous-time, infinite-horizon Markov decision problem. We use the dynamic programming method of value iteration for its solution and we characterize the resulting optimal cost function. We find that the cost of providing a service is convex-like in the number of resources leased and nondecreasing in the number of requests in the system. Close examination of the optimal cost function shows that the cost of providing a service is more sensitive to underdeployment than to overdeployment. Thus, when demand for the service is known to exist, but is unpredictable, it is better to lease more resources than fewer resources.

We consider shortest path problems defined on graphs with random arc costs. We assume that information on arc cost values is accumulated as the graph is being traversed. The objective is to devise a policy that leads from an origin to a destination node with minimal expected cost. We provide dynamic programming algorithms, estimates for their complexity, negative complexity results, and analysis of some possible heuristic algorithms. © 1996 John Wiley & Sons, Inc.

In this article, a data-driven distributed control method is proposed to solve the cooperative optimal output regulation problem of leader-follower multiagent systems. Different from traditional studies on cooperative output regulation, a distributed adaptive internal model is originally developed, which includes a distributed internal model and a distributed observer to estimate the leader's dynamics. Without relying on the dynamics of multiagent systems, we have proposed two reinforcement learning algorithms, policy iteration and value iteration, to learn the optimal controller through online input and state data, and estimated values of the leader's state. By combining these methods, we have established a basis for connecting data-distributed control methods with adaptive dynamic programming approaches in general since these are the theoretical foundation from which they are built.