Approximate dynamic programming for lateral transshipment problems in multi-location inventory systems

An effective lateral transshipment policy to improve service level in the supply chain

This study explores the optimization of centralized inventory systems through lateral transshipments, addressing supply chain disruptions caused by unpredictable demand. Lateral transshipments, which involve reallocating stock among retailers at the same distribution level, were investigated as a cost-effective and service-enhancing alternative to traditional emergency restocking methods. Using simulation and optimization techniques, we analyzed various transshipment policies, including Complete-Pooling and Partial-Pooling, across multiple scenarios. Our findings highlight that Partial-Pooling with an optimized threshold outperformed other policies by balancing collaboration and efficiency, reducing total costs, and improving service levels. The results underscore the importance of hybrid transshipment strategies, particularly for high-demand and cost-sensitive inventory systems. The research provides actionable insights for supply chain managers how tailored transshipment policies enhance centralized system overall supply chain resilience, profitability, and customer satisfaction.

Inventory models with lateral transshipments: A review

Many continuous process industries are required to operate multiproduct supply chains under significant uncertainty in market demands and prices, and mathematical programming formulations have been developed for the deterministic version of the problem. However, available extensions to the stochastic case cannot be readily used to obtain an optimal operating policy in practice because the uncertainty results in an explosion of the scenarios to be considered over an extended time horizon. In this article we represent the uncertainty through Markov chains and use an approach based on stochastic dynamic programming (DP), which can generate a dynamic operating policy that incorporates information about the uncertainty in the problem at each time step. DP is computationally infeasible because of the size of the state and action space for realistic problems. We propose a novel stochastic optimization algorithmic framework called “DP in a heuristically restricted state space.” The confined state space is generated by simulating various potential scenarios under centralized dynamic inventory and production policy generated by combining static local inventory policy heuristics. The resulting DP policy responds to the time‐varying demand for products by appropriately stitching together decisions made by the local static policies. This is a new formulation for effectively combining heuristic policies. © 2006 American Institute of Chemical Engineers AIChE J, 2006

Dynamic programming is a very useful tool in solving optimization and optimal control problems. Here, the Approximate Dynamic Programming (ADP) and the notion of neural networks based predictive control are combined with a model-free control method based on SPSA (Simultaneous perturbation stochastic approximation), and a novel ADP based model-free predictive control strategy for nonlinear systems is proposed. Dynamic programming is used to adjust the control parameters in the novel model-free control method and the notion of predictive control is introduced to modify the whole control structure. Finally, the proposed ADP based model-free predictive control strategy is applied to solve nonlinear tracking problems and the effectiveness of this novel control method is fully illustrated though simulation tests on two typical nonlinear systems.

Malaria is a major health concern for many developing countries. Designing strategies for efficient distribution of malaria medications, such as Artemesinin Combination Therapies, is a key challenge in resource constrained countries. This paper develops a solution methodology that integrates strategic‐level and tactical‐level models to better manage pharmaceutical distribution through a three‐tier centralized health system, which is common to sub‐Saharan African countries. At the strategic level, we develop a two‐stage stochastic programming approach to address the problem of demand uncertainty. In the first stage, an initial round of shipments is sent before the malaria season to each local clinic from district hospitals, which receive medications from regional warehouses. After the malaria season begins, a recourse action is triggered to avoid shortages in the form of (i) lateral transshipment or (ii) delayed shipment. The optimal solutions developed by the strategic model identify small clinic clusters possessing exclusive transshipment policies. Therefore, we decompose the problem at the tactical level, solving each clinic cluster independently using a Markov decision process approach to determine optimal periodic transshipment policies. A case study of our proposed distribution system is performed for 290 facilities controlled by the Malawi Ministry of Health. Numerical analysis of Malawi's distribution system indicates that our proposed cluster‐based decomposition method could near optimally reduce shortage incidents. Moreover, such an approach is robust to challenges of developing countries such as slow paper‐based inventory review, uncertain transportation infrastructure, the need for equitable distribution, and seasonal and correlated demand associated with malaria transmission dynamics.

This paper advances a neural-network-based approximate dynamic programming control mechanism that can be applied to complex control problems such as helicopter flight control design. Based on direct neural dynamic programming (DNDP), an approximate dynamic programming methodology, the control system is tailored to learn to maneuver a helicopter. The paper consists of a comprehensive treatise of this DNDP-based tracking control framework and extensive simulation studies for an Apache helicopter. A trim network is developed and seamlessly integrated into the neural dynamic programming (NDP) controller as part of a baseline structure for controlling complex nonlinear systems such as a helicopter. Design robustness is addressed by performing simulations under various disturbance conditions. All designs are tested using FLYRT, a sophisticated industrial scale nonlinear validated model of the Apache helicopter. This is probably the first time that an approximate dynamic programming methodology has been systematically applied to, and evaluated on, a complex, continuous state, multiple-input multiple-output nonlinear system with uncertainty. Though illustrated for helicopters, the DNDP control system framework should be applicable to general purpose tracking control.

With the development of marine science, aeronautics and astronautics, energy, chemical industry, biomedicine and management science, many complex systems face the problem of optimization and control. Approximate dynamic programming solves the curse of dimensionality problem of dynamic programming, and it is a new kind of approximate optimization solution that emerges in recent years. Based on the analysis of optimization system, this paper proposes a nonlinear multi-input multi-output, online learning, and data-driven approximate dynamic programming structure and its learning algorithm. The method is achieved from the following three aspects: 1) the critic function of multi-dimensional input critic module of the approximate dynamic programming is approximated with a data-driven k-nearest neighbor method; 2) the multi-output policy iteration of the approximate dynamic programming actor module is calculated with an exponential convergence performance; 3) The critic and actor modules are learned synchronously, and achieve the online optimal and control effect. The optimal control for the longitudinal motion of a thermal underwater glider is used to show the effect of the proposed method. This work can lay a foundation for the theory and application of a nonlinear data-driven multi-input multi-output approximate dynamic programming method. It's also the consensus needs in optimization control and artificial intelligence of many scientific and engineering fields, such as energy conservation, emission reduction, decision support and operational management etc.

Optimal Exercise Patterns and Valuation of Employee Stock Options: An Approximate Dynamic Programming Approach

Open AccessOpen Access licenseAboutSectionsView PDF ToolsAdd to favoritesDownload CitationsTrack Citations ShareShare onFacebookTwitterLinked InEmail Go to SectionOpen AccessOpen Access license HomeStochastic SystemsVol. 3, No. 2 Tuning Approximate Dynamic Programming Policies for Ambulance Redeployment via Direct SearchMatthew S. Maxwell, Shane G. Henderson, Huseyin TopalogluMatthew S. Maxwell, Shane G. Henderson, Huseyin TopalogluPublished Online:15 Nov 2013https://doi.org/10.1287/10-SSY020AbstractIn this paper we consider approximate dynamic programming methods for ambulance redeployment. We first demonstrate through simple examples how typical value function fitting techniques, such as approximate policy iteration and linear programming, may not be able to locate a high-quality policy even when the value function approximation architecture is rich enough to provide the optimal policy. To make up for this potential shortcoming, we show how to use direct search methods to tune the parameters in a value function approximation architecture so as to obtain high-quality policies. Direct search is computationally intensive. We therefore use a post-decision state dynamic programming formulation of ambulance redeployment that, together with direct search, requires far less computation with no noticeable performance loss. We provide further theoretical support for the post-decision state formulation of the ambulance-deployment problem by showing that this formulation can be obtained through a limiting argument on the original dynamic programming formulation. Back to Top Next FiguresReferencesRelatedInformationCited byInpatient Overflow: An Approximate Dynamic Programming ApproachJ. G. Dai, Pengyi Shi16 May 2019 | Manufacturing & Service Operations Management, Vol. 21, No. 4A Unified Framework for Optimization Under UncertaintyWarren B. Powell4 November 2016The Minimum Expected Penalty Relocation Problem for the Computation of Compliance Tables for Ambulance VehiclesThije van Barneveld21 April 2016 | INFORMS Journal on Computing, Vol. 28, No. 2Discrete Optimization Models for Homeland Security and Disaster ManagementLaura Albert McLay26 October 2015Simulation-Based Approximate Policy Iteration with Generalized Logistic FunctionsAntoine Sauré, Jonathan Patrick, Martin L. Puterman28 September 2015 | INFORMS Journal on Computing, Vol. 27, No. 3Technical Note—Trading Off Quick versus Slow Actions in Optimal SearchSteven M. Shechter, Farhad Ghassemi, Yasin Gocgun, Martin L. Puterman6 March 2015 | Operations Research, Vol. 63, No. 2A Bound on the Performance of an Optimal Ambulance Redeployment PolicyMatthew S. Maxwell, Eric Cao Ni, Chaoxu Tong, Shane G. Henderson, Huseyin Topaloglu, Susan R. Hunter16 July 2014 | Operations Research, Vol. 62, No. 5 Volume 3, Issue 2December 2013Pages 322-633 Article Information Metrics Information Received:November 01, 2010Published Online:November 15, 2013 Copyright © 2013, The author(s)Cite asMatthew S. Maxwell, Shane G. Henderson, Huseyin Topaloglu (2013) Tuning Approximate Dynamic Programming Policies for Ambulance Redeployment via Direct Search. Stochastic Systems 3(2):322-361. https://doi.org/10.1287/10-SSY020 PDF download

Approximate dynamic programming based approach to process control and scheduling

Sensory data have becoming widely available in large volume and variety due to the increasing presence and adoption of the Internet of Things. Such data can be tremendously useful if they are processed properly in a timely fashion. They could play a key role in the coordination of industrial production. It is thus desirable to explore an effective and efficient scheme to support data tracking and monitoring. This paper intends to propose a novel automatic learning scheme to improve the tracking efficiency while maintaining or improving the data tracking accuracy. A core strategy in the proposed scheme is the design of Laguerre neural network (LaNN)-based approximate dynamic programming (ADP). As a traditional optimal learning strategy, ADP is a popular approach for data processing. The action neural network (NN) and the critic NN as two important components in ADP have big impact on the performance of ADP. In this paper, a LaNN is employed as the implementation of the action NN in ADP considering Laguerre polynomials' approximation capability. In addition, this LaNN-based ADP is integrated into an online parameter-tuning framework to optimize those parameters of characteristic model that is used to trace the data in the tracking control system. Meanwhile, this article provides an associated Lyapunov convergence analysis to guarantee a uniformly ultimately boundedness property for tracking errors in the proposed approach. Furthermore, the proposed LaNN-based ADP optimal online parameter-tuning scheme is validated using a temperature dynamic tracking control task. The simulation results demonstrate that the scheme has satisfactory learning performance over time.

While Approximate Dynamic Programming has successfully been used in many applications involving discrete states and inputs such as playing the games of Tetris or chess, it has not been used in many continuous state and input space applications. In this paper, we combine Approximate Dynamic Programming techniques and apply them to the continuous, nonlinear and high dimensional dynamics of a quadcopter vehicle. We use a polynomial approximation of the dynamics and sum-of-squares programming techniques to compute a family of polynomial value function approximations for different tuning parameters. The resulting approximations to the optimal value function are combined in a point-wise maximum approach, which is used to compute the online policy. The success of the method is demonstrated in both simulations and experiments on a quadcopter. The control performance is compared to a linear time-varying Model Predictive Controller (MPC). The two methods are then combined to keep the computational benefits of a short horizon MPC and the long term performance benefits of the Approximate Dynamic Programming value function as the terminal cost.

While Approximate Dynamic Programming has successfully been used in many applications involving discrete states and inputs such as playing the games of Tetris or chess, it has not been used in many continuous state and input space applications. In this paper, we combine Approximate Dynamic Programming techniques and apply them to the continuous, nonlinear and high dimensional dynamics of a quadcopter vehicle. We use a polynomial approximation of the dynamics and sum-of-squares programming techniques to compute a family of polynomial value function approximations for different tuning parameters. The resulting approximations to the optimal value function are combined in a point-wise maximum approach, which is used to compute the online policy. The success of the method is demonstrated in both simulations and experiments on a quadcopter. The control performance is compared to a linear time-varying Model Predictive Controller (MPC). The two methods are then combined to keep the computational benefits of a short horizon MPC and the long term performance benefits of the Approximate Dynamic Programming value function as the terminal cost.

We study the approximate dynamic programming approach to revenue management in the context of attended home delivery. We draw on results from dynamic programming theory for Markov decision problems, convex optimisation and discrete convex analysis to show that the underlying dynamic programming operator has a unique fixed point. Moreover, we also show that - under certain assumptions - for all time steps in the dynamic program, the value function admits a continuous extension, which is a finite-valued, concave function of its state variables. This result opens the road for achieving scalable implementations of the proposed formulation, as it allows making informed choices of basis functions in an approximate dynamic programming context. We illustrate our findings using a simple numerical example and conclude with suggestions on how our results can be exploited in future work to obtain closer approximations of the value function.