Simplifying dynamic programming via mode‐directed tabling

In the dynamic programming paradigm the value of an optimal solution is recursively defined in terms of optimal solutions to subproblems. Such dynamic programming definitions can be very tricky and error-prone to specify. This paper presents a novel, elegant method based on tabled logic programming that simplifies the specification of such dynamic programming solutions. Our method introduces a new mode declaration for tabled predicates. The arguments of each tabled predicate are divided into indexed and non-indexed ones so that tabled predicates can be regarded as functions: indexed arguments represent input values and non-indexed arguments represent output values. The non-indexed arguments in a tabled predicate can be further declared to be aggregate, e.g., the minimum, so that while generating answers, the global table will dynamically maintain the smallest value for that argument. This mode declaration scheme, coupled with recursion, provides a considerably easy-to-use method for dynamic programming: there is no need to define the value of an optimal solution recursively, instead, defining a general solution suffices. The optimal value as well as its corresponding concrete solution can be derived implicitly and automatically using tabled logic programming systems. Experimental results are shown to indicate that the mode declaration improves both time and space performances in solving dynamic programming problems on tabled LP systems. Additionally, our mode declaration scheme provides an alternative implementation vehicle for preference logic programming.

Optimal load clipping with time of use rates

The mathematical model of optimal water quantity allocation for a single main canal in a large-scale irrigation area was constructed that took the minimal sum of the squared deviation of water shortage for water receiving areas controlled by the single main canal in one given irrigation period as the study target, and the total irrigation quantity of the single main canal as a constraint condition. Taking the optimal allocation of water quantity of each branch canal as decision variables, and several branch canals under the irrigation sequence of the main canal as a state variable, this model was solved by the one-dimensional dynamic programming (DP) method, by which the minimal water shortage and corresponding optimal water quantity allocation of each branch canal was calculated. The proposed method could provide a decision-making reference for optimal water resources allocation of single main canal irrigation areas, and also provide the theoretical basis for optimal water quantity allocation of a main canal with rotation irrigation by strips or with segmented rotation irrigation mode in China’s large-scale irrigation areas. Taking Hengliu Main Canal of Zhouqiao Irrigation Area in Jiangsu Province as a study case, optimization results showed that in a medium drought year (p = 75%) and a special drought year (p = 95%), minimal water shortage for water receiving areas controlled by Hengliu Main Canal was respectively 2.57 × 104 m3 and 23.31 × 104 m3 during the ponding period of rice. The corresponding water quantity allocation for each branch canal has reflected a compellent model solution precision and efficiency.

Dynamic Programming (DP) is a useful approach to multi-stage decision problems. On the basis of Bellman’s optimality principle such a problem can be decomposed into subproblems through recursive formulae. Bellman (1957), 1957a and Dantzig (1957) introduced a DP method for integer-variable linear programs. This method yields pseudo-polynomial algorithms when the number of constraints is fixed (Papadimitriou (1981)). In the class of linear programs, problems with a staircase structure can be efficiently treated by DP methods which can be interpreted as iterative predictor-corrector processes using a pricing mechanism for subproblems at individual stages (e.g., Dantzig (1963), Ho and Manne (1974), Ho (1978), Ho and Loute (1980), Abrahamson (1981) . Various applications of DP have also been developed in production planning (Manne (1958) , Wagner and Whitin (1959), Clark and Scarf (1960), Veinott (1965), 1969), Bessler and Veinott (1966), Zangwill (1965), 1966) (1969), Konno (1973), (1988), Bitran and Yanasse (1982), Bitran et al. (1984),... In particular, polynomial algorithms have been obtained for concave cost lotsizing problems and their extensions (e.g., Wagner and Whitin (1959), Zangwill (1965), (1966), (1959), Dreyfus and Law (1977).

This paper mainly studies the decision-making problems encountered by enterprises in the production process, including whether to inspect parts and finished products, and how to handle unqualified parts and finished products. By applying operations research and statistical knowledge, and adopting dynamic programming and one-sided hypothesis testing methods, relevant decision-making problems are effectively solved. In this paper, the authors propose a sampling inspection method to parameterize the defect rate of parts and optimize decision-making schemes for different production stages. Finally, the study analyzes the impact of various decisions on the economic benefits of enterprises, constructs corresponding models using dynamic programming, and derives optimal solutions under various decisions. For the first problem, based on the basic principles of statistics, this paper uses one-sided hypothesis testing to detect whether the defect rate of parts exceeds the nominal value. In the case of a large sample size, the central limit theorem is cleverly applied to approximate the binomial distribution to the normal distribution, thereby simplifying the calculation process. And based on the principle of sampling inspection, detailed judgments are made on whether to accept or reject parts under different confidence levels. When studying the second problem, this paper cleverly adopts the dynamic programming method to conduct detailed decision analysis on the three key stages of the enterprise production process: part inspection, finished product inspection, and unqualified product handling. And through the reverse analysis method, starting from the final product, the decision-making of each stage is gradually optimized to ensure that the risk is minimized while controlling costs. By evaluating the relationship between inspection costs and potential losses, as well as the specific impact of different handling methods for unqualified products on the economic benefits of enterprises, the goal is to maximize economic benefits while ensuring product quality. The third problem further enhances the complexity of decision-making based on the second problem, considering multiple processes and multiple parts to optimize decision-making in the multi-stage production process. This problem still adopts the reverse analysis method of dynamic programming to construct a more complex dynamic programming model, comprehensively considering the inspection costs, unqualified product handling costs, and potential market risks of each production stage. In the process of model construction, in-depth analysis is conducted on how to effectively handle parts, semi-finished products, unqualified semi-finished products, finished products, and unqualified finished products at different production stages, including different decisions for each stage. Through careful analysis, it provides enterprises with an optimal decision-making scheme for multiple processes and multiple parts.

The applicability of the dynamic programming method to two-dimensional slope stability analyses is studied. The critical slip surface is defined as the slip surface that yields the minimum value of an optimal function. The only assumption regarding the shape of the critical slip surface is that the surface is an assemblage of linear segments. Stresses acting along the critical slip surface are computed using a finite element stress analysis. Assumptions associated with limit equilibrium methods of slices related to the shape of the critical slip surface and the relationship between interslice forces are no longer required. A computer program named DYNPROG was developed based on the proposed analytical procedure, and numerous example problems have been analyzed. Results obtained when using DYNPROG were compared with those obtained when using several well-known limit equilibrium methods. The comparisons demonstrate that the dynamic programming method provides a superior solution when compared with conventional limit equilibrium methods. Analyses conducted also show that factors of safety computed when using the dynamic programming method are generally slightly lower than those computed using conventional limit equilibrium methods of slices; however, as Poisson's ratio approaches 0.5, the computed factors of safety from the dynamic programming method and the limit equilibrium method appear to become similar.Key words: dynamic programming, slope stability, stress analysis, optimization theory, limit equilibrium methods of slices.

General Imbedding Theory

In any complex or large scale sequential decision making problem, there is a crucial need to use function approximation to represent the relevant functions such as the value function or the policy. The Dynamic Programming (DP) and Reinforcement Learning (RL) methods introduced in previous chapters make the implicit assumption that the value function can be perfectly represented (i.e. kept in memory), for example by using a look-up table (with a finite number of entries) assigning a value to all possible states (assumed to be finite) of the system. Those methods are called exact because they provide an exact computation of the optimal solution of the considered problem (or at least, enable the computations to converge to this optimal solution). However, such methods often apply to toy problems only, since in most interesting applications, the number of possible states is so large (and possibly infinite if we consider continuous spaces) that a perfect representation of the function at all states is impossible. It becomes necessary to approximate the function by using a moderate number of coefficients (which can be stored in a computer), and therefore extend the range of DP and RL to methods using such approximate representations. These approximate methods combine DP and RL methods with function approximation tools.

This article mainly focuses on exploring the current application status, concrete things implemented by dynamic programming and advantages of dynamic programming in problems (the concept and operation of dynamic programming and the role it plays in programs) in different fields, future development trends and areas for improvement. It also introduces the gaps and problems currently encountered by dynamic programming, then provides corresponding solutions to make it more helpful and efficient in the future. In this regard, by consulting different references and online resources written by previous scientists, it finds out that they all have different points of view on dynamic programming and different solutions to its shortcomings. In the current research, dynamic programming is the most widely used algorithm with few limitations that can achieve the global optimal solution at any time; however, it does not mean that it can achieve the local optimal solution at any time, which also indirectly results in the memory space it requires being larger than other optimization algorithms. After analyzing the literature and researching and testing multiple applications, it is obvious that dynamic programming can make the code run easier and faster than traditional methods, even though it will take up more space.

This paper presents a novel method to find the profit-maximizing Nash equilibriums in allocating generation amounts with consideration of ramp-rates under competitive market environment. In order to find the Nash equilibriums it is necessary to search all the feasible combinations of generators' outputs which satisfy various constraints. The procedure to eliminate the dominated strategies can be formulated using Bellman's optimality principle of dynamic programming problem and hence the backward or forward search algorithm of dynamic programming can be easily applied. Therefore, the Nash equilibriums are found using dynamic programming method and we found that there exist several Nash equilibriums in the generation allocation problem. Individual generators participate in a game to maximize its profit through competitions and play aa game with bidding strategies of its generation quantities in a spot market. The ramp-rate physically or technically limits generators to increase or decrease outputs in its range and restricts the number of bidding strategies of each generator. We suggest the dynamic programming to find the Nash equilibriums while removing the dominated strategies in each stage (or each time). In the case studies, we analyzed the generation allocation game for a 12-hour multi-stage and compared it with the results of dynamic economic dispatch. Both of the two cases were considered generator's ramp-rate effects.

In this chapter we present a class of discrete-time structural models of labour force dynamics which are estimable using the same methodology. There have been a number of such models discussed in the literature, but only a very few have been estimated in their structural form. Although the models are categorized as either fitting into labour force participation, job search, or job matching frameworks, they all share in common the reservation wage property, namely that there is a change in state only when the random wage draw moves above or below a threshold value. They key to estimation is the numerical solution of the dynamic programming problem for the reservation wage path. In the finite horizon case considered here reservation wages are time varying and all of the economic theory is contained in the restrictions on the shape of the reservation wage path; only a few parameters fully describe the entire path, although not generally in closed form. The likelihood function based on observed labour force states is also a function of the reservation wage path. Thus, the parameters which determine the reservation wage path can be estimated by iterating between the dynamic programming solution and the likelihood function.

ABSTRACT: In this paper, the maximum principle of optimal control theory and the method of dynamic programming are discussed in relation to the minimization of fuel consumption in ship routing. The connection between the two methods is indicated for the case in which ship routing is treated as a continuous process, meaning that the sailing paths are not restricted to arcs of a grid as in the discrete dynamic programming method, but can vary continuously in the navigation area. Practical aspects are also discussed, such as the discrete approach of dynamic programming, as well as the finite version of the continuous approach and the limited predictability of the weather. Results are presented showing least-time routes, which are obtained with computational methods based on the maximum principle and the corresponding continuous type of dynamic programming.

An asymptotic error analysis of the conventional discrete dynamic programming (DDP) method is presented, and upper bounds of the error in the control policy (i.e., the difference of the estimated and true optimal control) at each operation period are computed. This error is shown to be of the order of the state discretization interval (ΔS), a result with significant implications in the optimization of multistate systems where the “curse of dimensionality” restricts the number of states to a relatively small number. The error in the optimal cost varies with ΔS2. The analysis provides useful insights into the effects of state discretization on calculated control and cost functions, the comparability of results from different discretizations, and criteria about the required number of nodes. In an effort to reduce the discretization error in the case of smooth cost functions, a new discrete dynamic programming method, termed gradient dynamic programming (GDP), is proposed. GDP uses a piecewise Hermite interpolation of the cost‐to‐go function, at each stage, which preserves the values of the cost‐to‐go function and of its first derivatives at the discretization nodes. The error in the control policy is shown to be of the order of (ΔS)3 and the error in the cost to vary with ΔS4. Thus as ΔS decreases, GDP converges to the true optimum much more rapidly than DDP. Another major advantage of the new methodology is that it facilitates the use of Newton‐type iterative methods in the solution of the nonlinear optimization problems at each stage. The linear convergence of DDP and the superlinear convergence of GDP are illustrated in an example.

A technique called dynamic programming can be used to identify an optimal rebalancing schedule, which significantly reduces rebalancing and sub-optimality costs. Dynamic programming provides solutions to multi-stage decision processes in which the decisions made in prior periods affect the choices available in later periods. Dynamic programming provides the optimal year-by-year decision policy by working backwards from year 10. The results of the test of the relative efficacy of dynamic programming and the MvD heuristic with data on domestic equities, domestic fixed income, non-US equities, non-US fixed income, and emerging market equities, show that the MvD heuristic performs quite well compared to the dynamic programming solution for the two-asset case and substantially better than other heuristics. The increase in the number of assets reduces the advantage of dynamic programming over the MvD heuristic and is reversed at the level of five assets. Dynamic programming cannot be applied beyond five assets, but the MvD heuristic can be extended up to 100 assets. The MvD heuristic reduces total costs relative to all of the other heuristics by substantial amounts. The performance of the MvD heuristic improves relative to the dynamic programming solution as more assets are added but this improvement reflects a growing reliance on an approximation for the dynamic programming approach.

In this paper, we present a modified dynamic programming (DP) method. The method is basically the same as the value iteration method (VI), a representative DP method, except the preprocess of a system's state transition model for reducing its complexity, and is called the dynamic programming on reduced models (DPRM). That reduction is achieved by imaginarily considering causes of the probabilistic behavior of a system, and then cutting off some causes with low occurring probabilities. In computational illustrations, VI, DPRM, and the real-time Q-learning method (RTQ) are applied to elevator operation problems, which can be modeled by using Markov decision processes. The results show that DPRM can compute quasi-optimal value functions which bring more effective allocations of elevators than value functions by RTQ in less computational times than VI. This characteristic is notable when the traffic pattern is complicated.