Non-Serial Dynamic Programming — A Survey

We address the problem of determining optimal stepsizes for estimating parameters in the context of approximate dynamic programming. The sufficient conditions for convergence of the stepsize rules have been known for 50 years, but practical computational work tends to use formulas with parameters that have to be tuned for specific applications. The problem is that in most applications in dynamic programming, observations for estimating a value function typically come from a data series that can be initially highly transient. The degree of transience affects the choice of stepsize parameters that produce the fastest convergence. In addition, the degree of initial transience can vary widely among the value function parameters for the same dynamic program. This paper reviews the literature on deterministic and stochastic stepsize rules, and derives formulas for optimal stepsizes for minimizing estimation error. This formula assumes certain parameters are known, and an approximation is proposed for the case where the parameters are unknown. Experimental work shows that the approximation provides faster convergence than other popular formulas.

In this paper, we develop an optimization method for solving the problem of the curse of dimensionality in dynamic programming. The method has the advantage of providing a cost for decision-making and eliminates the curse of dimensionality, which restricts the application of dynamic programming to only small classes of problems and, therefore, restricts the real-life application of dynamic programming. We established some relationships between geometric programming and dynamic programming parameters. We applied the method to a problem on course allocation and obtained the cost for course allocation and the optimal decision policy to be (0.1, 0.2, 1, 3).

In this paper we have studied the Dynamic programming problem and major area of applications of this approach has been introduced. Dynamic programming provides a means for determining optimal long-term crop management plans. However, most applications and their analysis on annual time steps with fixed strategies within the year, effectively ignoring conditional responses during the year. We suggest an alternative approach that captures the strategic responses within a cropping season to random weather variables as they unfold, reflecting farmers’ ability to adapt to weather realizations. Multistage decision problems a problem of dynamic programming problem there is numerically challenging. So for the analytical results, dynamic programming is able to obtain the optimal agricultural product problem, and also decides how many it consumes and how many it saves in material and permanently store in each period economically. However, in this study, the problem is considered deterministic in which all input parameters are constant. The objective is to find a sequence of actions (a so-called policy) that minimizes the total cost over the decision making horizon the purpose of this paper has been to introduce application of dynamic programming techniques by way of example. The end result of the model formulation reveals the applicability of dynamic programming in resolving long time of the problem.

Dynamic Programming (DP) is a useful approach to multi-stage decision problems. On the basis of Bellman’s optimality principle such a problem can be decomposed into subproblems through recursive formulae. Bellman (1957), 1957a and Dantzig (1957) introduced a DP method for integer-variable linear programs. This method yields pseudo-polynomial algorithms when the number of constraints is fixed (Papadimitriou (1981)). In the class of linear programs, problems with a staircase structure can be efficiently treated by DP methods which can be interpreted as iterative predictor-corrector processes using a pricing mechanism for subproblems at individual stages (e.g., Dantzig (1963), Ho and Manne (1974), Ho (1978), Ho and Loute (1980), Abrahamson (1981) . Various applications of DP have also been developed in production planning (Manne (1958) , Wagner and Whitin (1959), Clark and Scarf (1960), Veinott (1965), 1969), Bessler and Veinott (1966), Zangwill (1965), 1966) (1969), Konno (1973), (1988), Bitran and Yanasse (1982), Bitran et al. (1984),... In particular, polynomial algorithms have been obtained for concave cost lotsizing problems and their extensions (e.g., Wagner and Whitin (1959), Zangwill (1965), (1966), (1959), Dreyfus and Law (1977).

The industry and academe broadly use linear programming to analyze the logistics or supply chain solutions, which is based on the hypothesis of fixed environment and restricted procedures. However, in the dynamic market, the supply chain driven by the market faces multi units such as customers, suppliers, manufactures, distributors and carriers and multi uncertain factors such as physical distribution, manufacturing support and purchase. Through a comparative analysis of application of linear programming and dynamic programming in supply chain policy decision, the paper concludes that dynamic programming is more suitable for logistics and supply chain system policy decision study under changing market environments.

We consider discrete optimization problems in which the only exploitable feature of the objective function is a limited form of decomposability. “Nonoverlapping comparison algorithms” are defined as a model of procedures which decompose the problem and apply Bellman’s principle of optimality. Nonserial dynamic programming (DP), a simple elimination procedure, is shown to be optimal among all nonoverlapping comparison algorithms, including nondeterministic algorithms. These results can give an exponential lower bound on the shortest admissible proof that a solution is optimal. Furthermore, if part of the search space is ruled out, a subset of the comparisons made by DP optimally searches the remainder. We suggest that the running time of DP is a useful measure of the “interaction complexity” of a problem, and that because of its simplicity DP is of practical as well as theoretical interest.

Generalized dynamic programming for multicriteria optimization

Waste problems often occur in big cities such as Jakarta, Surabaya, Medan, Bandung, Yogyakarta and Semarang. Urban waste is waste produced by human activities in urban areas, including various types of materials such as plastic, paper, metal and organic materials. Therefore, waste needs to be collected, transported and disposed of immediately to prevent negative impacts on the environment. With the increase in population in urban areas, it is likely that the amount of waste will also increase. The solution that can be used to solve this problem is to optimize the distance using a dynamic programming model forward recursive equation algorithm. The application of dynamic programming can be used to solve various problems such as determining the shortest distance, optimal flow, task or resource scheduling, production planning, network optimization, and problems in research and development projects. Based on determining the optimal route for transporting waste in Medan Johor District using the dynamic programming algorithm forward recursive equation model, the total distance is 26,2km.

The applicability of the dynamic programming method to two-dimensional slope stability analyses is studied. The critical slip surface is defined as the slip surface that yields the minimum value of an optimal function. The only assumption regarding the shape of the critical slip surface is that the surface is an assemblage of linear segments. Stresses acting along the critical slip surface are computed using a finite element stress analysis. Assumptions associated with limit equilibrium methods of slices related to the shape of the critical slip surface and the relationship between interslice forces are no longer required. A computer program named DYNPROG was developed based on the proposed analytical procedure, and numerous example problems have been analyzed. Results obtained when using DYNPROG were compared with those obtained when using several well-known limit equilibrium methods. The comparisons demonstrate that the dynamic programming method provides a superior solution when compared with conventional limit equilibrium methods. Analyses conducted also show that factors of safety computed when using the dynamic programming method are generally slightly lower than those computed using conventional limit equilibrium methods of slices; however, as Poisson's ratio approaches 0.5, the computed factors of safety from the dynamic programming method and the limit equilibrium method appear to become similar.Key words: dynamic programming, slope stability, stress analysis, optimization theory, limit equilibrium methods of slices.

The structure of many sequential optimization problems over a finite or infinite horizon can be summarized in the mapping that defines the related dynamic programming algorithm. In this paper we take as a starting point this mapping and obtain results that are applicable to a broad class of problems. This approach has also been taken earlier by Denardo under contraction assumptions. The analysis here is carried out without contraction assumptions and thus the results obtained can be applied, for example, to the positive and negative dynamic programming models of Blackwell and Strauch. Most of the existing results for these models are generalized and several new results are obtained relating mostly to the convergence of the dynamic programming algorithm and the existence of optimal stationary policies.

In this chapter, following the theoretical fundamentals, the applications of dynamic programming (DP) on several HEV types including series, parallel and series–parallel are specifically given. ADVISOR-based simulation results prove that the DP algorithm is capable of finding optimal power which is from the battery to the electric motor and then the power ratio between them. Quadratic programming (QP) is also contained in discussion of this chapter due to its advantages of saving computational time in real-time realization. The power management strategy on basis of QP algorithm for fuel optimization of PHEV is depicted in details.

Usually, discrete optimization problems (DOPs) from applications have a special structure, and the matrices of constraints for large-scale problems have a lot of zero elements (sparse matrices). One of the promising ways to exploit sparsity in the interaction graph of the DOP is nonserial dynamic programming (NSDP), which allows to compute a solution in stages such that each of them uses results from previous stages. The drawback of NSDP methods consists on exponential time and space complexity that is exponential in the induced width of the DOP’s interaction graph. This causes an expediency and an urgency of development of tools that could help to cope with this difficulty. In this paper is shown that NSDP algorithm generates a family of related DOPs that differ from each other in their right-hand sides. For solving this family of related problems postoptimal and sensitivity analysis methods are proposed.

<p>A potentially parallel iterative algorithm for the solution of the unconstrained N-stage decision problem of Dynamic Programming is developed. This new solution method, known as Variable Metric Dynamic Programming, is based on the use of variable metric minimisation techniques to develop quadratic approximations to the optimal cost function for each stage. The algorithm is applied to various test problems, and a comparison with an existing similar algorithm proves favourable. The Variable Metric Dynamic Programming solution method is used in the implementation of an adaptive highlevel scheduling mechanism on a multiprogrammed computer in a university environment. This demonstrates a practical application of the new algorithm. More importantly, the application of Variable Metric Dynamic Programming to a scheduling problem illustrates how Mathematical Programming may be used in complex computer scheduling problems to provide in a natural way the required dynamic feedback mechanisms.</p>

The Option Value Model in the Retirement Literature: The Trade-Off between Computational Complexity and Predictive Validity

In this paper the Dynamic Programming technique and the HMMS rule are applied on a production control model of an assembly factory.