Parallel dynamic programming and automata theory

An asymptotic error analysis of the conventional discrete dynamic programming (DDP) method is presented, and upper bounds of the error in the control policy (i.e., the difference of the estimated and true optimal control) at each operation period are computed. This error is shown to be of the order of the state discretization interval (ΔS), a result with significant implications in the optimization of multistate systems where the “curse of dimensionality” restricts the number of states to a relatively small number. The error in the optimal cost varies with ΔS2. The analysis provides useful insights into the effects of state discretization on calculated control and cost functions, the comparability of results from different discretizations, and criteria about the required number of nodes. In an effort to reduce the discretization error in the case of smooth cost functions, a new discrete dynamic programming method, termed gradient dynamic programming (GDP), is proposed. GDP uses a piecewise Hermite interpolation of the cost‐to‐go function, at each stage, which preserves the values of the cost‐to‐go function and of its first derivatives at the discretization nodes. The error in the control policy is shown to be of the order of (ΔS)3 and the error in the cost to vary with ΔS4. Thus as ΔS decreases, GDP converges to the true optimum much more rapidly than DDP. Another major advantage of the new methodology is that it facilitates the use of Newton‐type iterative methods in the solution of the nonlinear optimization problems at each stage. The linear convergence of DDP and the superlinear convergence of GDP are illustrated in an example.

Schedule optimization is crucial to reduce energy consumption of flexible manufacturing systems (FMSs) with shared resources and route flexibility. Based on the weighted p-timed Petri Net (WTPN) models of FMS, this paper considers a scheduling problem which minimizes both productive and idle energy consumption subjected to general production constraints. The considered problem is proven to be a nonconvex mixed integer nonlinear program (MINLP). A new reachability graph (RG)-based discrete dynamic programming (DP) approach is proposed for generating near energy-optimal schedules within adequate computational time. The nonconvex MINLP is sampled, and the reduced RG is constructed such that only reachable paths are retained for computation of the energy-optimal path. Each scheduling subproblem is linearized, and each optimal substructure is computed to store in a routing table. It is proven that the sampling-induced error is bounded, and this upper bound can be reduced by increasing the sampling frequency. Experiment results on an industrial stamping system show the effectiveness of our proposed scheduling method in terms of computational complexity and deviation from optimality.

This paper investigates the computation of transient-optimal policies in discrete dynamic programming. The model, is quite general: it may contain transient as well as nontransient policies. and the transition matrices are not necessarily substochastic. A functional equation for the so-called transient-value-vector is derived and the concept of superharmonicity is introduced. This concept provides the linear program to compute the transientvalue-vector and a transient-optimal policy. We also discuss the elimination of suboptimal actions, the solution of problems with additional constraints, and the computation of an efficient policy for a multiple objective dynamic programming problem.

Previous article Next article Existence of an Optimal Stationary Policy in a Markov Decision ProcessI. V. RomanovskiiI. V. Romanovskiihttps://doi.org/10.1137/1110010PDFBibTexSections ToolsAdd to favoritesExport CitationTrack CitationsEmail SectionsAbout[1] Ronald A. Howard, Dynamic programming and Markov processes, The Technology Press of M.I.T., Cambridge, Mass., 1960viii+136 MR0118514 0091.16001 Google Scholar[2] David Blackwell, Discrete dynamic programming, Ann. Math. Statist., 33 (1962), 719–726 MR0149965 0133.12906 CrossrefGoogle Scholar[3] S. Vajda, The theory of games and linear programming, Methuen and Co. Ltd., London, 1956v+106 MR0086716 0074.34402 Google Scholar Previous article Next article FiguresRelatedReferencesCited ByDetails Optimal control of stationary Markov processesStochastic Processes and their Applications, Vol. 1, No. 3 | 1 Jul 1973 Cross Ref A Linear Programming Problem Connected with Optimal Stationary Control in a Dynamic Decision ProblemB. G. Pittel’Theory of Probability & Its Applications, Vol. 16, No. 4 | 17 July 2006AbstractPDF (498 KB)Optimal Strategy in Games with an Unbounded Sequence of MovesYu. I. KiferTheory of Probability & Its Applications, Vol. 14, No. 2 | 17 July 2006AbstractPDF (774 KB) Volume 10, Issue 1| 1965Theory of Probability & Its Applications1-192 History Submitted:10 October 1964Published online:17 July 2006 InformationCopyright © Society for Industrial and Applied MathematicsPDF Download Article & Publication DataArticle DOI:10.1137/1110010Article page range:pp. 120-122ISSN (print):0040-585XISSN (online):1095-7219Publisher:Society for Industrial and Applied Mathematics

This research is about the development of a dynamic programming model for solving fuzzy linear programming problems. Initially, fuzzy dynamic linear programming model FDLP is developed. This research revises the established dynamic programming model for solving linear programming problems in a crisp environment. The mentioned approach is upgraded to address the problem in an uncertain environment. Dynamic programming model can either be passing forward or backward. In the proposed approach backward dynamic programming approach is adopted to address the problem. It is then followed by implementing the proposed method on the education system of Pakistan. The education system of Pakistan comprises of the Primary, Middle, Secondary, and Tertiary education stages. The problem is to maximize the efficiency of the education system while achieving the targets with minimum usage of the constrained resources. Likewise the model tries to maximize the enrollment in the Primary, Middle, Secondary and Tertiary educational categories, subject to the total available resources in a fuzzy uncertain environment. The solution proposes that the enrollment can be increased by an amount 9997130, by increasing the enrollment in the Middle and Tertiary educational categories. Thus the proposed method contributes to increase the objective function value by 30%. Moreover, the proposed solutions violate none of the constraints. In other words, the problem of resources allocation in education system is efficiently managed to increase efficiency while remaining in the available constrained resources. The motivation behind using the dynamic programming methodology is that it always possesses a numerical solution, unlike the other approaches having no solution at certain times. The proposed fuzzy model takes into account uncertainty in the linear programming modeling process and is more robust, flexible and practicable.

Bellman’s discrete dynamic programming is one of the most general approaches to solve optimal control problems. For discrete-time dynamical systems, it is, at least theoretically, capable to determine globally optimal control laws. In most practical cases, both state and control variables are subject to constraints. Due to the necessity for gridding of the range of both state and control variables in numerical implementations of dynamic programming, the computational effort grows exponentially with increasing system dimensions. This fact is well known as the curse of dimensionality. Furthermore, gridding of intervals representing uncertain system parameters is inevitable, if dynamic programming is used for the design of optimal controllers for systems with uncertainties. In this contribution, an interval arithmetic procedure for the design of optimal and robust controllers is presented. This procedure relies on the basic concepts of dynamic programming. Sophisticated techniques for the exclusion of non-optimal control strategies significantly reduce the computational burden. Since interval techniques can be applied to both continuous-time and discrete-time dynamical systems, the interval arithmetic optimization approach presented in this chapter is applicable to both cases. In addition, the inclusion of effects of uncertain parameters in the underlying optimality criteria is demonstrated. For that purpose, interval arithmetic routines for analysis and design of optimal and robust controllers have been developed. Details about computationally efficient implementations of interval arithmetic optimization procedures and numerical results for a mechanical positioning system with statedependent switchings between different dynamical models for viscous and Coulomb friction are summarized.

Gear shifting strategy of vehicle is important aid for the acquisition of dynamic performance and high economy. A dynamic programming (DP) algorithm is used to optimize the gear shifting schedule for off‐road vehicle by using an objective function that weighs fuel use and trip time. The optimization is accomplished through discrete dynamic programming and a trade‐off between trip time and fuel consumption is analyzed. By using concave and convex surface road as road profile, an optimal gear shifting strategy is used to control the longitudinal behavior of the vehicle. Simulation results show that the trip time can be reduced by powerful gear shifting strategy and fuel consumption can achieve high economy with economical gear shifting strategy in different initial conditions and route cases.

Dynamic programming algorithms are developed for optimal capital allocation subject to budget constraints. We extend the work of Weingartner [Weingartner, H. M. 1966. Capital budgeting of interrelated projects: Survey and synthesis. Management Sci. 12(7, March) 485–516.] and Weingartner and Ness [Weingartner, H. M., D. N. Ness. 1967. Methods for the solution of the multi-dimensional 0/1 knapsack problem. Oper. Res. 15(1, January–February) 83–108.] by including multilevel projects, reinvesting returns, borrowing and lending, capital deferrals, and project interactions. We are able to handle dynamic programming models with several state variables because the optimal returns are monotone non-decreasing step functions. Computational experience with a variety of problems is reported.

7 - Numerical optimization in allocation, storage, and recovery of thermal energy and resources

In this paper we introduce the notion of derivation in product (direct product) of additively idempotent semirings. These semirings are useful and important tools in diverse areas such as design of switching circuits, automata theory, information systems, dynamic programming and decision theory. We study hereditary derivations, inner derivations and obtain a representation of an arbitrary derivations which are zero on the multipliers.

The dynamic programming (DP) theory is applied in designing energy management control strategy of Hybrid Tracked Vehicle. As hybrid tracked vehicles use dual-energy sources, the engine-generator sets and power batteries. We applied the dynamic programming theory to optimize the distribution between the dual-energy sources, so we can make the fuel consumption minimize. Through the analysis of the optimization results, near-optimal rules are extracted. Then we do some simulations to verify the control rules based on MATLAB software. It was found that the fuel economic performance improved significant, compared to former rule-based control strategy.

This article reviews the history and theory of dynamic programming (DP), a recursive method of solving sequential decision problems under uncertainty. It discusses computational algorithms for the numerical solution of DP problems, and an important limitation in our ability to solve realistic large-scale dynamic programming problems, the ‘curse of dimensionality’. It also summarizes recent research in complexity theory that delineates situations where the curse can be broken (allowing us to solve DPs using fast polynomial time algorithms), and situations where it is insuperable. The literature on econometric estimation and testing of DP models is reviewed, as is another ‘scientific limit to knowledge’, namely, the identification problem.

This paper examines long-term optimal operation using dynamic programming for a large hydropower system of 10 reservoirs in Northeast China. Besides considering flow and hydraulic head, the optimization explicitly includes time-varying electricity market prices to maximize benefit. Two techniques are used to reduce the 'curse of dimensionality' of dynamic programming with many reservoirs. Discrete differential dynamic programming (DDDP) reduces the search space and computer memory needed. Object-oriented programming (OOP) and the ability to dynamically allocate and release memory with the C++ language greatly reduces the cumulative effect of computer memory for solving multi-dimensional dynamic programming models. The case study shows that the model can reduce the 'curse of dimensionality' and achieve satisfactory results.

In this paper we propose a robust formulation for discrete time dynamic programming (DP). The objective of the robust formulation is to systematically mitigate the sensitivity of the DP optimal policy to ambiguity in the underlying transition probabilities. The ambiguity is modeled by associating a set of conditional measures with each state-action pair. Consequently, in the robust formulation each policy has a set of measures associated with it. We prove that when this set of measures has a certain “rectangularity” property, all of the main results for finite and infinite horizon DP extend to natural robust counterparts. We discuss techniques from Nilim and El Ghaoui [17] for constructing suitable sets of conditional measures that allow one to efficiently solve for the optimal robust policy. We also show that robust DP is equivalent to stochastic zero-sum games with perfect information.

This paper presents a new dynamic programming (DP) approach to the optimal path following problem. The method rests on an interpolation in the phase plane so that the resulting joint accelerations and joint torques are continuous. This allows for taking into account limits on the joint jerks and torque rates in addition to joint velocities, accelerations, torques, and the mechanical power. Most methods proposed in the literature yield values of optimal trajectories at discrete sampling times so that this must be interpolated subsequently to the trajectory optimization. This causes violations of the joint jerk and torque rate limits. The proposed method does not suffer from this problem, which is a main feature of this approach. Unlike most of the previously proposed methods, joint jerk and torque rate limits are addressed with a reasonable increase in the computation time of DP algorithms. Special attention is given to experimental validation of the optimization results. The presented experimental results confirm the importance of taking the motor torque characteristics as well as the Coulomb and viscous friction into account. Neglecting these effects (as most previous publications did) leads to trajectories that cannot be performed by real manipulators. Apart from DP, other approaches with smaller computation times exist. However, most of these methods are either limited to the time optimal case (which might not always be the desired criterion) and are not able to handle all earlier mentioned constraints or cannot take into account viscous friction effects. Apart from that a sequential convex programming (SCP) approach exists, which accounts for the same constraints as the presented approach. Therefore, the proposed DP approach is compared to this SCP method and an example is presented where the time optimal trajectory is performed by a real manipulator.