Transient policies in discrete dynamic programming: Linear programming including suboptimality tests and additional constraints

An asymptotic error analysis of the conventional discrete dynamic programming (DDP) method is presented, and upper bounds of the error in the control policy (i.e., the difference of the estimated and true optimal control) at each operation period are computed. This error is shown to be of the order of the state discretization interval (ΔS), a result with significant implications in the optimization of multistate systems where the “curse of dimensionality” restricts the number of states to a relatively small number. The error in the optimal cost varies with ΔS2. The analysis provides useful insights into the effects of state discretization on calculated control and cost functions, the comparability of results from different discretizations, and criteria about the required number of nodes. In an effort to reduce the discretization error in the case of smooth cost functions, a new discrete dynamic programming method, termed gradient dynamic programming (GDP), is proposed. GDP uses a piecewise Hermite interpolation of the cost‐to‐go function, at each stage, which preserves the values of the cost‐to‐go function and of its first derivatives at the discretization nodes. The error in the control policy is shown to be of the order of (ΔS)3 and the error in the cost to vary with ΔS4. Thus as ΔS decreases, GDP converges to the true optimum much more rapidly than DDP. Another major advantage of the new methodology is that it facilitates the use of Newton‐type iterative methods in the solution of the nonlinear optimization problems at each stage. The linear convergence of DDP and the superlinear convergence of GDP are illustrated in an example.

Parallel dynamic programming and automata theory

Pneumatic Artificial Muscle (PAM) is an attractive device to be used as an actuator for humanoid robots because of its high power-to-weight ratio and good flexibility. However, both the modeling and the controlling of PAM-driven robots are challenging due to the high nonlinearities of a PAM's air pressure dynamics and its mechanical structure. This paper focuses on applying Reinforcement Learning (RL) to the control of a PAM-driven robots without our knowledge of its model. We propose a new RL algorithm, Local Update Dynamic Policy Programming (LUDPP), as an extension of Dynamic Policy Programming (DPP). This algorithm exploits the nature of smooth policy update of DPP to considerably reduce the computational complexity in both time and space: at each iteration, this algorithm only updates the value function locally throughout the whole state-action space. We applied LUDPP to control one finger (2 DOFs with a 12-dimensional state-action space) of Shadow Dexterous Hand, a PAM-driven humanoid robot hand. Experimental results suggest that our method can achieve successful control of such a robot with a limited computational resource whereas other conventional value function based RL algorithms (DPP, LSPI) cannot.

Previous article Next article Existence of an Optimal Stationary Policy in a Markov Decision ProcessI. V. RomanovskiiI. V. Romanovskiihttps://doi.org/10.1137/1110010PDFBibTexSections ToolsAdd to favoritesExport CitationTrack CitationsEmail SectionsAbout[1] Ronald A. Howard, Dynamic programming and Markov processes, The Technology Press of M.I.T., Cambridge, Mass., 1960viii+136 MR0118514 0091.16001 Google Scholar[2] David Blackwell, Discrete dynamic programming, Ann. Math. Statist., 33 (1962), 719–726 MR0149965 0133.12906 CrossrefGoogle Scholar[3] S. Vajda, The theory of games and linear programming, Methuen and Co. Ltd., London, 1956v+106 MR0086716 0074.34402 Google Scholar Previous article Next article FiguresRelatedReferencesCited ByDetails Optimal control of stationary Markov processesStochastic Processes and their Applications, Vol. 1, No. 3 | 1 Jul 1973 Cross Ref A Linear Programming Problem Connected with Optimal Stationary Control in a Dynamic Decision ProblemB. G. Pittel’Theory of Probability & Its Applications, Vol. 16, No. 4 | 17 July 2006AbstractPDF (498 KB)Optimal Strategy in Games with an Unbounded Sequence of MovesYu. I. KiferTheory of Probability & Its Applications, Vol. 14, No. 2 | 17 July 2006AbstractPDF (774 KB) Volume 10, Issue 1| 1965Theory of Probability & Its Applications1-192 History Submitted:10 October 1964Published online:17 July 2006 InformationCopyright © Society for Industrial and Applied MathematicsPDF Download Article & Publication DataArticle DOI:10.1137/1110010Article page range:pp. 120-122ISSN (print):0040-585XISSN (online):1095-7219Publisher:Society for Industrial and Applied Mathematics

Bellman’s discrete dynamic programming is one of the most general approaches to solve optimal control problems. For discrete-time dynamical systems, it is, at least theoretically, capable to determine globally optimal control laws. In most practical cases, both state and control variables are subject to constraints. Due to the necessity for gridding of the range of both state and control variables in numerical implementations of dynamic programming, the computational effort grows exponentially with increasing system dimensions. This fact is well known as the curse of dimensionality. Furthermore, gridding of intervals representing uncertain system parameters is inevitable, if dynamic programming is used for the design of optimal controllers for systems with uncertainties. In this contribution, an interval arithmetic procedure for the design of optimal and robust controllers is presented. This procedure relies on the basic concepts of dynamic programming. Sophisticated techniques for the exclusion of non-optimal control strategies significantly reduce the computational burden. Since interval techniques can be applied to both continuous-time and discrete-time dynamical systems, the interval arithmetic optimization approach presented in this chapter is applicable to both cases. In addition, the inclusion of effects of uncertain parameters in the underlying optimality criteria is demonstrated. For that purpose, interval arithmetic routines for analysis and design of optimal and robust controllers have been developed. Details about computationally efficient implementations of interval arithmetic optimization procedures and numerical results for a mechanical positioning system with statedependent switchings between different dynamical models for viscous and Coulomb friction are summarized.

Gear shifting strategy of vehicle is important aid for the acquisition of dynamic performance and high economy. A dynamic programming (DP) algorithm is used to optimize the gear shifting schedule for off‐road vehicle by using an objective function that weighs fuel use and trip time. The optimization is accomplished through discrete dynamic programming and a trade‐off between trip time and fuel consumption is analyzed. By using concave and convex surface road as road profile, an optimal gear shifting strategy is used to control the longitudinal behavior of the vehicle. Simulation results show that the trip time can be reduced by powerful gear shifting strategy and fuel consumption can achieve high economy with economical gear shifting strategy in different initial conditions and route cases.

Dynamic programming algorithms are developed for optimal capital allocation subject to budget constraints. We extend the work of Weingartner [Weingartner, H. M. 1966. Capital budgeting of interrelated projects: Survey and synthesis. Management Sci. 12(7, March) 485–516.] and Weingartner and Ness [Weingartner, H. M., D. N. Ness. 1967. Methods for the solution of the multi-dimensional 0/1 knapsack problem. Oper. Res. 15(1, January–February) 83–108.] by including multilevel projects, reinvesting returns, borrowing and lending, capital deferrals, and project interactions. We are able to handle dynamic programming models with several state variables because the optimal returns are monotone non-decreasing step functions. Computational experience with a variety of problems is reported.

7 - Numerical optimization in allocation, storage, and recovery of thermal energy and resources

A projection and division method for problems of linear dynamic programming

We consider challenging dynamic programming models where the associated Bellman equation, and the value and policy iteration algorithms commonly exhibit complex and even pathological behavior. Our analysis is based on the new notion of regular policies. These are policies that are well-behaved with respect to value and policy iteration, and are patterned after proper policies, which are central in the theory of stochastic shortest path problems. We show that the optimal cost function over regular policies may have favorable value and policy iteration properties, which the optimal cost function over all policies need not have. We accordingly develop a unifying methodology to address long standing analytical and algorithmic issues in broad classes of undiscounted models, including stochastic and minimax shortest path problems, as well as positive cost, negative cost, risk-sensitive, and multiplicative cost problems.

Scalable reinforcement learning for plant-wide control of vinyl acetate monomer process

Schedule optimization is crucial to reduce energy consumption of flexible manufacturing systems (FMSs) with shared resources and route flexibility. Based on the weighted p-timed Petri Net (WTPN) models of FMS, this paper considers a scheduling problem which minimizes both productive and idle energy consumption subjected to general production constraints. The considered problem is proven to be a nonconvex mixed integer nonlinear program (MINLP). A new reachability graph (RG)-based discrete dynamic programming (DP) approach is proposed for generating near energy-optimal schedules within adequate computational time. The nonconvex MINLP is sampled, and the reduced RG is constructed such that only reachable paths are retained for computation of the energy-optimal path. Each scheduling subproblem is linearized, and each optimal substructure is computed to store in a routing table. It is proven that the sampling-induced error is bounded, and this upper bound can be reduced by increasing the sampling frequency. Experiment results on an industrial stamping system show the effectiveness of our proposed scheduling method in terms of computational complexity and deviation from optimality.

Dynamic expediting of an urgent order with uncertain progress

This paper presents a general dynamic programming algorithm for the solution of optimal stochastic control problems concerning a class of discrete event systems.

Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming