Platelet Inventory Management with Approximate Dynamic Programming

In this paper, we consider a finite-horizon Markov decision process (MDP) for which the objective at each stage is to minimize a quantile-based risk measure (QBRM) of the sequence of future costs; we call the overall objective a dynamic quantile-based risk measure (DQBRM). In particular, we consider optimizing dynamic risk measures where the one-step risk measures are QBRMs, a class of risk measures that includes the popular value at risk (VaR) and the conditional value at risk (CVaR). Although there is considerable theoretical development of risk-averse MDPs in the literature, the computational challenges have not been explored as thoroughly. We propose data-driven and simulation-based approximate dynamic programming (ADP) algorithms to solve the risk-averse sequential decision problem. We address the issue of inefficient sampling for risk applications in simulated settings and present a procedure, based on importance sampling, to direct samples toward the “risky region” as the ADP algorithm progresses. Finally, we show numerical results of our algorithms in the context of an application involving risk-averse bidding for energy storage.The online appendix is available at https://doi.org/10.1287/moor.2017.0872 .

New closed-loop approximate dynamic programming for solving stochastic decentralized multi-project scheduling problem with resource transfers

ABSTRACTResource planning (RP) in a professional service organization matches workforce resources with project tasks while considering a myriad of factors such as skill requirements, service delivery role, skill type, workforce proficiency level, and geographical location. The multiperiod stochastic resource planning studied in this article extends the one‐period deterministic resource planning by explicitly coping with both internal resource attrition and project demand uncertainty in a sequential decision‐making framework. It allows resource managers to make effective use of their internal resources and identify the need to outsource to external contingent resources. We model the multiperiod stochastic resource planning as a Markov decision process and implement an approximate dynamic programming algorithm to obtain dynamic and adaptive solutions in reasonable computation times. A comprehensive computational study shows that our approximate dynamic programming algorithm achieves higher profitability and internal resource utilization compared to the rolling horizon approach used as a benchmark.

This paper focuses on economical operation of a grid-connected microgrid in real-time. An online optimization model is developed to achieve the microgrid operation in the process of dynamic optimal control, and through information feedback and online optimization to reduce the impact of uncertainty in a MG. The online optimization model has the characteristics of nonlinear and discontinuous, so an approximate dynamic programming (ADP) algorithm is used to solve this online model. Finally, the proposed online optimization model is tested in the European benchmark microgrid system. The simulation results show that the online optimization model overcomes the impacts of random fluctuations of renewable energy and loads and reduces the operating cost of the microgrid.

This article examines a major maritime disaster scenario in which the objective is to maximize the number of survivors. To achieve this objective, we seek to optimize the decision policy to load individuals onto a helicopter for transport from an evacuation site to a forward operating base. Our contributions are twofold. First, we formulate the loading problem as a finite-horizon Markov Decision Process. Second, since the curse of dimensionality renders exact methods not applicable, we use an Approximate Dynamic Programming (ADP) approach to explore the efficacy of these methods to the problem. We generate three ADP policies, each using a value function approximation based on a unique post-decision state variable and a lookup table representation. We compare these ADP policies to a random policy, a myopic policy, and Policy Function Approximation (PFA) which evacuates survivors in order of triage category starting with the most critically ill. Our results show that an ADP-generated policy which prioritizes the evacuation of healthy individuals improved performance by 34 ± 5% versus the random policy, 15 ± 3% versus the myopic policy, and 42 ± 3% versus the PFA policy.

Dialysis is the most common therapy for patients afflicted with chronic kidney failure. Currently, little is known about the relationship between the timing of dialysis initiation and the therapy's cost and effectiveness. This paper examines the cost-effective initiation of dialysis and compares standard initiation criteria from the clinical literature to computationally derived strategies. Comparisons make use of a simulation model that integrates submodels of disease progression, hospitalization, transplantation, cost, and quality of life. The simulation model is also used by an approximate dynamic programming (ADP) algorithm to derive approximately optimal strategies that maximize patient welfare. Patient welfare is measured from the society's perspective and is defined as the product of the expected discounted quality-adjusted life years (QALYs) and a “value-of-life” parameter, minus the expected total discounted medical expenditures. Also considered is an alternative formulation in which the goal is to minimize the expected total discounted medical expenditures without affecting patient QALYs relative to current medical practice. Numerical results show that: (i) standard early initiation strategies, where once started on dialysis patients are kept on a fixed weekly program, have a limited potential, and (ii) early dialysis at an incrementally increasing dose customized to each patient can yield a significant cost advantage. These findings demonstrate computationally intensive models of disease progression, and therapy effectiveness can identify novel strategies for managing expensive medical therapies and a more efficient use of scarce health-care resources.

This paper investigates the single-evader and multi-pursuer (SEMP) reach-avoid differential graphical (RADG) games of nonlinear heterogeneous players subject to saturated input and limited communication channels. The evader’s objective is to reach a designated target area while avoiding the pursuers, whose aim is to intercept the evader. First, we reformulate the SEMP RADG game as an optimal control problem within a weighted communication topology graph by incorporating the interception and control errors. Then, optimal control strategies that account for input saturation are derived by solving the coupled Hamilton-Jacobi (HJ) equations. These strategies are shown to constitute the Nash Equilibrium (NE) of the game. In addition, four types of pursuers, namely, isolated, passive, invisible, and regular pursuers, are defined based on the communication topology, and the conditions for achieving an Interactive Nash Equilibrium (INE), which is proposed for SEMP RADG games, are analyzed. Moreover, a single-network approximate dynamic programming (ADP) algorithm using concurrent learning (CL) is proposed to provide the near-optimal solutions to the coupled HJ equations. Asymptotic capture conditions are established through an examination of equilibrium points, and extensions to general pursuit-evasion (PE) games and half-space targets are further discussed. Our results are validated through numerical simulations.

New heuristics for the RCPSP with multiple overlapping modes

In this work, time-driven learning refers to the machine learning method that updates parameters in a prediction model continuously as new data arrives. Among existing approximate dynamic programming (ADP) and reinforcement learning (RL) algorithms, the direct heuristic dynamic programming (dHDP) has been shown an effective tool as demonstrated in solving several complex learning control problems. It continuously updates the control policy and the critic as system states continuously evolve. It is therefore desirable to prevent the time-driven dHDP from updating due to insignificant system event such as noise. Toward this goal, we propose a new event-driven dHDP. By constructing a Lyapunov function candidate, we prove the uniformly ultimately boundedness (UUB) of the system states and the weights in the critic and the control policy networks. Consequently, we show the approximate control and cost-to-go function approaching Bellman optimality within a finite bound. We also illustrate how the event-driven dHDP algorithm works in comparison to the original time-driven dHDP.

The cumulative capacitated vehicle routing problem (CCVRP) is a variant of the classical capacitated vehicle routing problem in which the objective is the minimization of customers’ total waiting time prior to service. Inspired by the first-mile pick-up service within 3rd-Party Logistics operations, we study a dynamic and stochastic variant of the CCVRP. In this variant, a capacitated vehicle undertakes a single tour to collect customers’ parcels, taking into account stochastic service requests, stochastic service cancellations, and stochastic demand (i.e., changes in parcel size) simultaneously. Conditional on maximizing the usage of the vehicle’s capacity, we minimize the total waiting time of both customers who are served and those who request services but are not served due to cancellation or capacity violation. We formulate this problem as a Markov decision process and solve it using a hybrid approach that combines approximate dynamic programming and rollout algorithm. Our main contributions include ([Formula: see text]) the construction of the [Formula: see text] [Formula: see text] that leverages a property of the Poisson process, and ([Formula: see text]) consideration of service fairness-based acceptance/rejection rule in response to excessive stochastic service requests. In the numerical study, our approach is examined to outperform an 2-Opt-based re-optimization policy in terms of both customers’ waiting time and service-request satisfying rate. More importantly, computational results heuristically show that there is no cost to adopting the service fairness-based rule, which is obviously a valuable managerial insight.

With the gradual opening of the power market, the advantages of virtual power plant (VPP) in aggregating demand side resources are becoming increasingly prominent. Therefore, it is urgent to study the market mechanism and clearing strategy of their participation in integrated demand response (IDR). An IDR market mechanism and clearing strategy considering the flexible capacity of VPP are proposed in this paper. Firstly, the mechanism of VPP participating in the IDR market is analyzed, and the IDR resource models considering the internal resources of VPP is established. Furthermore, the mathematical model of flexible capacity of virtual power plant is proposed and solved by approximate dynamic programming algorithm. Finally, the IDR market clearing strategy considering the flexible capacity of multiple VPPs is proposed, and the interactive clearing method of the superior distribution network and the VPP operator optimization strategy is designed. The results show that the IDR market clearing strategy in this paper can reduce the total cost on the basis of ensuring the interests of VPP operators, and achieve peak load shifting and new energy accommodation.

An online adaptive optimal control is proposed for continuous-time nonlinear systems with completely unknown dynamics, which is achieved by developing a novel identifier-critic-based approximate dynamic programming algorithm with a dual neural network (NN) approximation structure. First, an adaptive NN identifier is designed to obviate the requirement of complete knowledge of system dynamics, and a critic NN is employed to approximate the optimal value function. Then, the optimal control law is computed based on the information from the identifier NN and the critic NN, so that the actor NN is not needed. In particular, a novel adaptive law design method with the parameter estimation error is proposed to online update the weights of both identifier NN and critic NN simultaneously, which converge to small neighbourhoods around their ideal values. The closed-loop system stability and the convergence to small vicinity around the optimal solution are all proved by means of the Lyapunov theory. The proposed adaptation algorithm is also improved to achieve finite-time convergence of the NN weights. Finally, simulation results are provided to exemplify the efficacy of the proposed methods.

ABSTRACTLarge-scale disasters and catastrophic events typically result in a significant shortage of critical resources, posing a great challenge to allocating limited resources among different affected areas to improve the quality of emergency logistics operations. This article pays attention to the performance of resource allocation, which includes three metrics: efficiency, effectiveness, and equity, respectively corresponding to economic cost, service quality, and fairness. In particular, the effectiveness metric considers human suffering by depicting it as deprivation cost, an economic valuation measurement that has been recently proposed and the equity metric concerns about the service equality at the end of planning horizon. A nonlinear integer model is first proposed and then an equivalent dynamic programming model is developed to avoid the nonlinear terms created by the introduction of the deprivation cost. The dynamic programming method can solve small-scale problems to optimality but meets difficulty when solving medium- and large-scale problems, due to the curse of dimensionality. Therefore, an approximate dynamic programming algorithm, called the rollout algorithm, is proposed to overcome this computational difficulty. The computational complexity of the proposed algorithm is theoretically analyzed. Furthermore, a modified version of the rollout algorithm is presented, with its computational complexity analyzed. Extensive numerical experiments are conducted to test the performance of the proposed algorithms, and the experimental results demonstrate that the initially proposed rollout algorithm yields optimal or near-optimal solutions within a reasonable amount of time. In addition, the impacts of some important parameters are investigated and managerial insights are drawn.

Multiple stage stochastic planning of integrated electricity and gas system based on distributed approximate dynamic programming

Abstract: Online resource management in cloud computing must respond to rapidly changing demand and system conditions, often exceeding the adaptability of static or periodically recomputed optimisation plans. This study formulates cloud resource management as a Markov decision process (MDP) and investigates deep reinforcement learning (DRL) as an approximate dynamic programming (ADP) approach for deriving adaptive control policies. The system state encompasses workload, capacity utilisation, and performance indicators, while actions govern virtual machine placement, scaling, and server power-state transitions. Reward functions combine energy and operating costs with penalties associated with service-level agreement violations. DRL algorithms, including deep Q-networks, advantage actor–critic, and proximal policy optimization, are implemented in PyTorch and trained through interaction with a CloudSim-based simulation environment calibrated to realistic load and power profiles. The learned policies are evaluated against deterministic and scenario-based stochastic optimisation baselines using metrics such as average cost, energy consumption, SLA violations, and robustness to non-stationary demand. Results show that DRL policies can approach or surpass the performance of stochastic optimisation solutions in highly dynamic scenarios, while maintaining feasible control actions. The study demonstrates how DRL can be positioned within an OR/ADP framework and highlights conditions under which learning-based methods offer significant advantages for online cloud control. Keywords: deep reinforcement learning, approximate dynamic programming, cloud computing, Markov decision processes, online resource management, energy-aware control, PyTorch, simulation