Online Optimal Operation of Microgrid Using Approximate Dynamic Programming Under Uncertain Environment

We study a stochastic perishable inventory control problem with endogenous (decision-dependent) uncertainty in shelf life of units. Our primary motivation is determining ordering policies for blood platelets. Hospitals typically order their required platelets from a central supplier, and as such, the shelf life of units at the time of delivery can be subject to significant variability. Determining optimal ordering quantities is a challenging task because of the short maximum shelf life of platelets (three to five days after testing) and high uncertainty in daily demand. We formulate the problem as an infinite-horizon discounted Markov decision process (MDP). The model captures salient features observed in our data from a network of Canadian hospitals and allows for fixed ordering costs. We show that with uncertainty in shelf life, the value function of the MDP is nonconvex and key structural properties valid under deterministic shelf life no longer hold. Hence, we propose an approximate dynamic programming (ADP) algorithm to find approximate policies. We approximate the value function using a linear combination of basis functions and tune the parameters using a simulation-based policy iteration algorithm. We evaluate the performance of the proposed policy using extensive numerical experiments in parameter regimes relevant to the platelet inventory management problem. We further leverage the ADP algorithm to evaluate the impact of ignoring shelf-life uncertainty. Finally, we evaluate the out-of-sample performance of the ADP algorithm in a case study using real data and compare it with the historical hospital performance and other benchmarks. After tuning the parameters, the ADP policy can be computed online in a few seconds and results in more than 50% lower expiration and shortage rates compared with the historical rates. In addition, it performs better or as well as other benchmarks, including an exact policy that ignores uncertainty in shelf life and becomes hard to compute for larger instances of the problem. History: Accepted by Paul Brooks, Area Editor for Applications in Biology, Medicine, & Healthcare. Funding: V. Sarhangian was supported by the Natural Sciences and Engineering Research Council of Canada [Grant RGPIN-2018-04518]. H. Abouee-Mehrizi was supported by the Natural Sciences and Engineering Council of Canada [Grant RGPIN-2019-05625]. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2023.0245 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2023.0245 ). The complete IJOC Software and Data Repository is available at https://informsjoc.github.io/ .

Sensory data have becoming widely available in large volume and variety due to the increasing presence and adoption of the Internet of Things. Such data can be tremendously useful if they are processed properly in a timely fashion. They could play a key role in the coordination of industrial production. It is thus desirable to explore an effective and efficient scheme to support data tracking and monitoring. This paper intends to propose a novel automatic learning scheme to improve the tracking efficiency while maintaining or improving the data tracking accuracy. A core strategy in the proposed scheme is the design of Laguerre neural network (LaNN)-based approximate dynamic programming (ADP). As a traditional optimal learning strategy, ADP is a popular approach for data processing. The action neural network (NN) and the critic NN as two important components in ADP have big impact on the performance of ADP. In this paper, a LaNN is employed as the implementation of the action NN in ADP considering Laguerre polynomials' approximation capability. In addition, this LaNN-based ADP is integrated into an online parameter-tuning framework to optimize those parameters of characteristic model that is used to trace the data in the tracking control system. Meanwhile, this article provides an associated Lyapunov convergence analysis to guarantee a uniformly ultimately boundedness property for tracking errors in the proposed approach. Furthermore, the proposed LaNN-based ADP optimal online parameter-tuning scheme is validated using a temperature dynamic tracking control task. The simulation results demonstrate that the scheme has satisfactory learning performance over time.

New closed-loop approximate dynamic programming for solving stochastic decentralized multi-project scheduling problem with resource transfers

In this paper, we consider a finite-horizon Markov decision process (MDP) for which the objective at each stage is to minimize a quantile-based risk measure (QBRM) of the sequence of future costs; we call the overall objective a dynamic quantile-based risk measure (DQBRM). In particular, we consider optimizing dynamic risk measures where the one-step risk measures are QBRMs, a class of risk measures that includes the popular value at risk (VaR) and the conditional value at risk (CVaR). Although there is considerable theoretical development of risk-averse MDPs in the literature, the computational challenges have not been explored as thoroughly. We propose data-driven and simulation-based approximate dynamic programming (ADP) algorithms to solve the risk-averse sequential decision problem. We address the issue of inefficient sampling for risk applications in simulated settings and present a procedure, based on importance sampling, to direct samples toward the “risky region” as the ADP algorithm progresses. Finally, we show numerical results of our algorithms in the context of an application involving risk-averse bidding for energy storage.The online appendix is available at https://doi.org/10.1287/moor.2017.0872 .

Dialysis is the most common therapy for patients afflicted with chronic kidney failure. Currently, little is known about the relationship between the timing of dialysis initiation and the therapy's cost and effectiveness. This paper examines the cost-effective initiation of dialysis and compares standard initiation criteria from the clinical literature to computationally derived strategies. Comparisons make use of a simulation model that integrates submodels of disease progression, hospitalization, transplantation, cost, and quality of life. The simulation model is also used by an approximate dynamic programming (ADP) algorithm to derive approximately optimal strategies that maximize patient welfare. Patient welfare is measured from the society's perspective and is defined as the product of the expected discounted quality-adjusted life years (QALYs) and a “value-of-life” parameter, minus the expected total discounted medical expenditures. Also considered is an alternative formulation in which the goal is to minimize the expected total discounted medical expenditures without affecting patient QALYs relative to current medical practice. Numerical results show that: (i) standard early initiation strategies, where once started on dialysis patients are kept on a fixed weekly program, have a limited potential, and (ii) early dialysis at an incrementally increasing dose customized to each patient can yield a significant cost advantage. These findings demonstrate computationally intensive models of disease progression, and therapy effectiveness can identify novel strategies for managing expensive medical therapies and a more efficient use of scarce health-care resources.

This paper investigates the single-evader and multi-pursuer (SEMP) reach-avoid differential graphical (RADG) games of nonlinear heterogeneous players subject to saturated input and limited communication channels. The evader’s objective is to reach a designated target area while avoiding the pursuers, whose aim is to intercept the evader. First, we reformulate the SEMP RADG game as an optimal control problem within a weighted communication topology graph by incorporating the interception and control errors. Then, optimal control strategies that account for input saturation are derived by solving the coupled Hamilton-Jacobi (HJ) equations. These strategies are shown to constitute the Nash Equilibrium (NE) of the game. In addition, four types of pursuers, namely, isolated, passive, invisible, and regular pursuers, are defined based on the communication topology, and the conditions for achieving an Interactive Nash Equilibrium (INE), which is proposed for SEMP RADG games, are analyzed. Moreover, a single-network approximate dynamic programming (ADP) algorithm using concurrent learning (CL) is proposed to provide the near-optimal solutions to the coupled HJ equations. Asymptotic capture conditions are established through an examination of equilibrium points, and extensions to general pursuit-evasion (PE) games and half-space targets are further discussed. Our results are validated through numerical simulations.

New heuristics for the RCPSP with multiple overlapping modes

In this work, time-driven learning refers to the machine learning method that updates parameters in a prediction model continuously as new data arrives. Among existing approximate dynamic programming (ADP) and reinforcement learning (RL) algorithms, the direct heuristic dynamic programming (dHDP) has been shown an effective tool as demonstrated in solving several complex learning control problems. It continuously updates the control policy and the critic as system states continuously evolve. It is therefore desirable to prevent the time-driven dHDP from updating due to insignificant system event such as noise. Toward this goal, we propose a new event-driven dHDP. By constructing a Lyapunov function candidate, we prove the uniformly ultimately boundedness (UUB) of the system states and the weights in the critic and the control policy networks. Consequently, we show the approximate control and cost-to-go function approaching Bellman optimality within a finite bound. We also illustrate how the event-driven dHDP algorithm works in comparison to the original time-driven dHDP.

Approximate dynamic programming for an energy-efficient parallel machine scheduling problem

More recently, with the increasing demand of web services on the World Wide Web used in the Internet of Things (IoTs), there has been a growing interest in the study of efficient web service quality evaluation approaches through the use of prediction strategies to obtain accurate quality-of-service (QoS) values. The unpredictable network environment imposes very challenging obstacles to web service QoS prediction. Most of the traditional web service QoS prediction approaches are implemented only using a set of static model parameters with the help of designer's priori knowledge. Unlike the traditional QoS prediction approaches, our algorithm in this paper is realized by incorporating approximate dynamic programming (ADP)-based online parameter tuning strategy into the QoS prediction approach. Through online learning and optimization, the proposed approach implements the QoS prediction with automatic parameter tuning capability, and the prior knowledge or identification of the prediction model is not required. The near-optimal performance of QoS prediction can therefore be achieved. Experiment studies are carried out to demonstrate the effectiveness of the proposed ADP-based prediction approach.

This paper deals with on-line coordination of public transport systems under disruptions. An on-line optimization model is proposed in order to support decisions about how to balance all the fleet of transit lines in the public transport system and also to minimize waiting time caused by disruption. A fast heuristic algorithm is developed for the on-line problem and a numerical study of the regional train network of Madrid is carried out.

Maximum wind power tracking based on cloud RBF neural network

Approximate dynamic programming (ADP) has been widely studied from several important perspectives: algorithm development, learning efficiency measured by success or failure statistics, convergence rate, and learning error bounds. Given that many learning benchmarks used in ADP or reinforcement learning studies are control problems, it is important and necessary to examine the learning controllers from a control-theoretic perspective. This paper makes use of direct heuristic dynamic programming (direct HDP) and three typical benchmark examples to introduce a unique analytical framework that can be applied to other learning control paradigms and other complex control problems. The sensitivity analysis and the linear quadratic regulator (LQR) design are used in the paper for two purposes: to quantify direct HDP performances and to provide guidance toward designing better learning controllers. The use of LQR however does not limit the direct HDP to be a learning controller that addresses nonlinear dynamic system control issues. Toward this end, applications of the direct HDP for nonlinear control problems beyond sensitivity analysis and the confines of LQR have been developed and compared whenever appropriate to an LQR design.

Geosteering, a key component of drilling operations, traditionally involves manual interpretation of various data sources such as well-log data. This introduces subjective biases and inconsistent procedures. Academic attempts to solve geosteering decision optimization with greedy optimization and approximate dynamic programming (ADP) showed promise but lacked adaptivity to realistic diverse scenarios. Reinforcement learning (RL) offers a solution to these challenges, facilitating optimal decision-making through reward-based iterative learning. State estimation methods, e.g., particle filter (PF), provide a complementary strategy for geosteering decision-making based on online information. We introduce RL-Estimation, a method that integrates an RL-based geosteering framework with PF to address realistic geosteering scenarios. RL-Estimation deploys PF to process real-time well-log data to estimate the location of the well relative to the stratigraphic layers, which then informs the RL-based decision-making process. Our findings indicate that RL-Estimation achieves at least 20% higher reservoir contact compared to using RL or PF alone. Additionally, RL-Estimation performs within 2% of the theoretically optimal benchmark, which assumes access to the true state instead of relying on estimates from PF. These results demonstrate the synergy between RL and PF, highlighting the method’s effectiveness and near-optimal performance in realistic geosteering scenarios.

Multiple stage stochastic planning of integrated electricity and gas system based on distributed approximate dynamic programming