Approximate dynamic programming: Application to process supply chain management

Open AccessOpen Access licenseAboutSectionsView PDF ToolsAdd to favoritesDownload CitationsTrack Citations ShareShare onFacebookTwitterLinked InEmail Go to SectionOpen AccessOpen Access license HomeStochastic SystemsVol. 3, No. 2 Tuning Approximate Dynamic Programming Policies for Ambulance Redeployment via Direct SearchMatthew S. Maxwell, Shane G. Henderson, Huseyin TopalogluMatthew S. Maxwell, Shane G. Henderson, Huseyin TopalogluPublished Online:15 Nov 2013https://doi.org/10.1287/10-SSY020AbstractIn this paper we consider approximate dynamic programming methods for ambulance redeployment. We first demonstrate through simple examples how typical value function fitting techniques, such as approximate policy iteration and linear programming, may not be able to locate a high-quality policy even when the value function approximation architecture is rich enough to provide the optimal policy. To make up for this potential shortcoming, we show how to use direct search methods to tune the parameters in a value function approximation architecture so as to obtain high-quality policies. Direct search is computationally intensive. We therefore use a post-decision state dynamic programming formulation of ambulance redeployment that, together with direct search, requires far less computation with no noticeable performance loss. We provide further theoretical support for the post-decision state formulation of the ambulance-deployment problem by showing that this formulation can be obtained through a limiting argument on the original dynamic programming formulation. Back to Top Next FiguresReferencesRelatedInformationCited byInpatient Overflow: An Approximate Dynamic Programming ApproachJ. G. Dai, Pengyi Shi16 May 2019 | Manufacturing & Service Operations Management, Vol. 21, No. 4A Unified Framework for Optimization Under UncertaintyWarren B. Powell4 November 2016The Minimum Expected Penalty Relocation Problem for the Computation of Compliance Tables for Ambulance VehiclesThije van Barneveld21 April 2016 | INFORMS Journal on Computing, Vol. 28, No. 2Discrete Optimization Models for Homeland Security and Disaster ManagementLaura Albert McLay26 October 2015Simulation-Based Approximate Policy Iteration with Generalized Logistic FunctionsAntoine Sauré, Jonathan Patrick, Martin L. Puterman28 September 2015 | INFORMS Journal on Computing, Vol. 27, No. 3Technical Note—Trading Off Quick versus Slow Actions in Optimal SearchSteven M. Shechter, Farhad Ghassemi, Yasin Gocgun, Martin L. Puterman6 March 2015 | Operations Research, Vol. 63, No. 2A Bound on the Performance of an Optimal Ambulance Redeployment PolicyMatthew S. Maxwell, Eric Cao Ni, Chaoxu Tong, Shane G. Henderson, Huseyin Topaloglu, Susan R. Hunter16 July 2014 | Operations Research, Vol. 62, No. 5 Volume 3, Issue 2December 2013Pages 322-633 Article Information Metrics Information Received:November 01, 2010Published Online:November 15, 2013 Copyright © 2013, The author(s)Cite asMatthew S. Maxwell, Shane G. Henderson, Huseyin Topaloglu (2013) Tuning Approximate Dynamic Programming Policies for Ambulance Redeployment via Direct Search. Stochastic Systems 3(2):322-361. https://doi.org/10.1287/10-SSY020 PDF download

Reservoir management is a highly complex problem, which includes the uncertainty in the inflows as well as in the objectives. Stochastic dynamic programming (SDP) has been widely used to develop the optimal operating policies of multipurpose reservoirs. In the recent years, fuzzy optimization has been used to incorporate the uncertainty caused due to the imprecise nature of objectives. In the present study, a comparison is made between a conventional SDP model and fuzzy dynamic programming models for optimal operation of a multipurpose reservoir. The two models are developed for Hirakud Reservoir on River Mahanadi in the State of Orissa in India. The performance of the reservoir is evaluated under the operating policies derived from these models through simulation. The results indicate that the fuzzy dynamic programming model performs better in achieving the flood control objective and thus in the overall performan1ce of reservoir.

The natural flow regime can sustain the ecological integrity of riverine ecosystems. Different reservoir operation polices differ in their effects on the degree of alteration of natural flow regimes. Dynamic programming plays an important role in developing operation policies. When using dynamic programming models to develop operation policies, the discrete number of storage states (DNSS), which is a key factor affecting the reservoirs operation policies, is always been determined based on computational efficiency and economic benefits. Little consideration has been given to the ecological disturbance caused by different DNSS‐based operation policies. To analyze the impact of DNSS, we built a deterministic dynamic programming model to explore the relationship among DNSS, the flow regime alteration (ecological disturbance), and the cumulative annual power generation (economic benefits) by setting a range of DNSS scenarios. We used three reservoirs with different storage coefficients (ratios of usable storage to annual average runoff) as examples and used the range of variability approach to assess the ecological disturbance under these scenarios. We compared the results with those of a stochastic dynamic programming (SDP) model and a Bayesian SDP (BSDP) model. We found that when DNSS is low, increasing DNSS improves economic benefits but causes a more severe ecological disturbance; when DNSS is high, increasing DNSS improves the economic benefits only slightly, without exacerbating the ecological disturbance; for a given DNSS, the BSDP model provides higher economic benefits than the SDP model and a similar disturbance of the riverine ecosystem; and larger reservoirs more often cause more severe disturbance of riverine ecosystems because monthly mean flows and annual extreme flows change more drastically. Our results will help to protect the riverine ecosystems and improve economic benefits if reservoir operation managers consider DNSS using dynamic programming models.

Approximate dynamic programming based approach to process control and scheduling

A Least-Squares Temporal Difference based method for solving resource allocation problems

With globalization and rapid technological-economic development accelerating the market dynamics, consumers' demand is becoming more volatile and diverse. In this situation, capacity adjustment as an operational strategic decision plays a major role to ensure supply chain responsiveness while maintaining costs at a reasonable norm. This study contributes to the literature by developing computationally efficient approximate dynamic programming approaches for production capacity planning considering uncertainties and demand interdependence in a multi-factory multi-product supply chain setting. For this purpose, the k-Nearest-Neighbor-based Approximate Dynamic Programming and the Rolling-Horizon-based Approximate Dynamic Programming are developed to enable real-time decision support while ensuring the robustness of the outcomes in stochastic decision environments. Given the market volatilities in the Thin Film Transistor-Liquid Crystal Display industry, a real case from this sector is investigated to evaluate the applicability of the developed approach and provide insights for other industry situations. The developed method is less complex to implement, and numerical experiments showed that it is also computationally more efficient compared to Stochastic Dynamic Programming.

<p>The increasing stress on water resource systems has prompted researchers to look for ways to improve the performance of reservoir operations. Changes in demand, various hydrological inputs, and new environmental stresses are among issues that water managers face. These concerns have sparked interest in applying different techniques to determine reservoir operation policy to improve reservoir system performance. As the resolution of analysis rises, it becomes more difficult to effectively represent a real-world system using currently available approaches for determining the best reservoir operation policy. One of the challenges is the "curse of dimensionality," which occurs when the discretization of the state and action spaces becomes finer or when more state or action variables are taken into account. Because of the dimensionality curse, the number of state-action variables is limited, rendering dynamic programming (DP) and stochastic DP (SDP) ineffective in handling complex reservoir optimization issues. Reinforcement learning (RL) is one way to overcome the aforementioned curses of stochastic optimization of water resources systems. RL is a well-known and influential technique in machine learning research that can solve a wide range of optimization and simulation challenges. In this study, a novel continuous-action deep RL algorithm called Deep Deterministic Policy Gradients (DDPG) is applied to solve the DP problem for the Folsom Reservoir system located in California, US. Without requiring any model simplifications or surrendering any of the critical characteristics of DP, the employed continuous action-space RL method effectively overcomes dimensionality concerns. The system model employs an iterative learning method that takes into account delayed rewards without requiring an explicit probabilistic model of hydrologic processes, and it can learn the best actions that maximize total expected reward by interacting with a simulated environment. This research is funded by the US Geological Survey.</p>

Dynamic programming is a very useful tool in solving optimization and optimal control problems. Here, the Approximate Dynamic Programming (ADP) and the notion of neural networks based predictive control are combined with a model-free control method based on SPSA (Simultaneous perturbation stochastic approximation), and a novel ADP based model-free predictive control strategy for nonlinear systems is proposed. Dynamic programming is used to adjust the control parameters in the novel model-free control method and the notion of predictive control is introduced to modify the whole control structure. Finally, the proposed ADP based model-free predictive control strategy is applied to solve nonlinear tracking problems and the effectiveness of this novel control method is fully illustrated though simulation tests on two typical nonlinear systems.

Big data and the curse of dimensionality are common vocabularies that researchers in different communities have recently been dealing with, e.g. dynamic programming (DP) in automatic control system society. A novel unweighted sampled based least square projection approach is proposed in this study to address the issue of the large state space in the DP optimisation problem. The method, in particular, takes into account both contraction mapping and monotonicity properties of the DP algorithm for value function approximation. Specifically, the batch of samples are gathered by uniform probability distribution at first, and an unweighted LS sub‐problem in the subspace is solved. As the case study, a new Markov decision process model associated with a resource allocation problem is considered to illustrate the technique and evaluate its effectiveness. It is noted that the approach can be employed for different applications as well. Moreover, a MATLAB based software is developed to implement and examine different parts of the proposed method. Simulation examples are considered to support the results of the approach via developed software. The idea makes a connection between the recent advances in big data analysis and approximate DP as well.

This paper treats a class of finite-stage dynamic programming (DP) problems whose state space is an interval in the one-dimensional Euclidean space R1. By introducing a notion of “reward space” we can define, in this class, an inverse problem (called the inverse DP) to a given DP (called the main DP), because of the monotonicity and continuity of the reward functions and state transformations. Roughly speaking the inverse DP to the main DP is defined by replacing “max”, “reward functions” and “state transformations” with “min”, “inverse (in some sense) to the state transformations” and “inverse (in some sense) to the reward functions”, respectively, by replacing “terminal reward function” with “inverse to itself”, and by alternating “reward spaces” with “state spaces”, but by letting “action spaces” remain as they are. According to the monotonicity of the reward functions and state transformations we clarify the relation between the main DP and the inverse DP separately for four cases. Our main result is stated as follows. The pair of optimal reward functions and optimal policy for the main DP characterizes, in an inverse sense, the pair of optimal reward functions and optimal policy for the inverse DP, and vice versa (Section 2). Further we show that our DP represents a mathematical programming problem which has the property “recursiveness with monotonicity” in [5] and conversely such a problem can also be represented by our DP (Section 3). The last section is devoted to the illustration of several examples.

A robust optimization approach to reduce the bullwhip effect of supply chains with vendor order placement lead time delays in an uncertain environment

In this study, operation policies were obtained for a reservoir in Michoacan, Mexico, used for irrigation and domestic water supplies. The main purpose of these policies is to optimize the uses of the water, an increasingly scarce resource everywhere. Two optimization methodologies were used; stochastic dynamic programming, that provides release decisions for each stage, and genetic algorithms coupled with a reservoir operation simulation program, to achieve annual release curves. The operation of the reservoir was evaluated using historical inflow records. Monthly requirements for crop cycles, as well as the volumes of spills and deficits were examined. Both methodologies gave inverse relationships between deficits and spilled volumes. While both methodologies proved efficient in achieving the objectives, the results of the stochastic dynamic programming showed a better performance for this system.

Dynamic programming techniques have proven much more flexible than calculus of variations and other techniques in performing redundancy resolution through global optimization of performance indices. When the state and input spaces are discrete, and the time horizon is finite, they can easily accommodate generic constraints and objective functions and find Pareto-optimal sets. Several implementations have been proposed in previous works, but either they do not ensure the achievement of the globally optimal solution, or they have not been demonstrated on robots of practical relevance. In this communication, recent advances in dynamic programming redundancy resolution, so far only demonstrated on simple planar robots, are extended to be used with generic kinematic structures. This is done by expanding the Robot Operating System (ROS) and proposing a novel architecture meeting the requirements of maintainability, re-usability, modularity and flexibility that are usually required to robotic software libraries. The proposed ROS extension integrates seamlessly with the other software components of the ROS ecosystem, so as to encourage the reuse of the available visualization and analysis tools. The new architecture is demonstrated on a 7-DOF robot with a six-dimensional task, and topological analyses are carried out on both its state space and resulting joint-space solution.

Approximate dynamic programming for lateral transshipment problems in multi-location inventory systems

We study the interplay between inventory replenishment policies and information sharing in the context of a two-tier supply chain with a single supplier and a single retailer serving an independent and identically distributed Gaussian market demand. We investigate how the retailer’s inventory policy impacts the supply chain’s cumulative expected long-term average inventory costs [Formula: see text] in two extreme information-sharing cases: (a) full information sharing and (b) no information sharing. To find the retailer’s inventory policy that minimizes [Formula: see text], we formulate an infinite-dimensional optimization problem whose decision variables are the MA([Formula: see text]) coefficients that characterize a stationary ordering policy. Under full information sharing, the optimization problem admits a simple solution and the optimal policy is given by an MA(1) process. On the other hand, to solve the optimization problem under no information sharing, we reformulate the optimization from its time domain formulation to an equivalent z-transform formulation in which the decision variables correspond to elements of the Hardy space H2. This alternative representation allows us to use a number of results from H2 theory to compute the optimal value of [Formula: see text] and characterize a sequence of ϵ-optimal inventory policies under some mild technical conditions. By comparing the optimal solution under full information sharing and no information sharing, we derive a number of important practical takeaways. For instance, we show that there is value in information sharing if and only if the retailer’s optimal policy under full information sharing is not invertible with respect to the sequence of demand shocks. Furthermore, we derive a fundamental mathematical identity that reveals the value of information sharing by exploiting the canonical Smirnov–Beurling inner–outer factorization of the retailer’s orders when viewed as an element of H2. We also show that the value of information sharing can grow unboundedly when the cumulative supply chain costs are dominated by the supplier’s inventory costs. Funding: R. Caldentey acknowledges the University of Chicago Booth School of Business for financial support. Supplemental Material: The online appendix is available at https://doi.org/10.1287/moor.2023.0008 .