Simulation-based Algorithms for Markov Decision Processes

AlphaGo and its successors AlphaGo Zero and AlphaZero made international headlines with their incredible successes in game playing, which have been touted as further evidence of the immense potential of artificial intelligence, and in particular, machine learning. AlphaGo defeated the reigning human world champion Go player Lee Sedol 4 games to 1, in March 2016 in Seoul, Korea, an achievement that surpassed previous computer game-playing program milestones by IBM’s Deep Blue in chess and by IBM’s Watson in the U.S. TV game show Jeopardy. AlphaGo then followed this up by defeating the world’s number one Go player Ke Jie 3-0 at the Future of Go Summit in Wuzhen, China in May 2017. Then, in December 2017, AlphaZero stunned the chess world by dominating the top computer chess program Stockfish (which has a far higher rating than any human) in a 100-game match by winning 28 games and losing none (72 draws) after training from scratch for just four hours! The deep neural networks of AlphaGo, AlphaZero, and all their incarnations are trained using a technique called Monte Carlo tree search (MCTS), whose roots can be traced back to an adaptive multistage sampling (AMS) simulation-based algorithm for Markov decision processes (MDPs) published in Operations Research back in 2005 [Chang, HS, MC Fu, J Hu and SI Marcus (2005). An adaptive sampling algorithm for solving Markov decision processes. Operations Research, 53, 126–139.] (and introduced even earlier in 2002). After reviewing the history and background of AlphaGo through AlphaZero, the origins of MCTS are traced back to simulation-based algorithms for MDPs, and its role in training the neural networks that essentially carry out the value/policy function approximation used in approximate dynamic programming, reinforcement learning, and neuro-dynamic programming is discussed, including some recently proposed enhancements building on statistical ranking & selection research in the operations research simulation community.

Faustmann's formula gives the land value, or the forest value of land with trees, under deterministic assumptions regarding future stand growth and prices, over an infinite horizon. Markov decision process (MDP) models generalize Faustmann's approach by recognizing that future stand states and prices are known only as probabilistic distributions. The objective function is then the expected discounted value of returns, over an infinite horizon. It gives the land or the forest value in a stochastic environment. In MDP models, the laws of motion between stand-price states are Markov chains. Faustmann's formula is a special case where the probability of movement from one state to another is equal to unity. MDP models apply whether the stand state is bare land, or any state with trees, be it even- or uneven-aged. Decisions change the transition probabilities between stand states through silvicultural interventions. Decisions that maximize land or forest value depend only on the stand-price state, independently of how it was reached. Furthermore, to each stand-price state corresponds one single best decision. The solution of the MDP gives simultaneously the best decision for each state, and the forest value (land plus trees), given the stand state and following the best policy. Numerical solutions use either successive approximation, or linear programming. Examples with deterministic and stochastic cases show in particular the convergence of the MDP model to Faustmann's formula when the future is assumed known with certainty. In this deterministic environment, Faustmann's rule is independent of the distribution of stands in the forest. FOR. SCI. 47(4):466–474.

Towards enhanced threat modelling and analysis using a Markov Decision Process

Markov decision process (MDP) models were effective in analyzing forest management policies. Even the simplest standard results gave useful insights into forest ecology, such as how landscape diversity is shaped by natural catastrophes, and how forests mature through successional phases. The methods were also useful to predict the effects of different management policies on ecological and economic criteria. Optimization augmented the usefulness of the approach, suggesting that income from Wisconsin's maple-birch forests could be increased without ruining their diversity of landscape, tree size, and tree species. It showed that maximizing species diversity, defined by the distribution of trees in shade-tolerance classes, would require some harvest. Instead, maximum tree size diversity occurred in unmanaged forests, but this gave a less diverse landscape and no income. The MDP method allowed for the design of compromise policies that would maximize income while keeping diversity above specified limits. The opportunity cost of increasing tree size diversity was found to be much higher than for species diversity. Comparing the maximum timber income owners could have got with what they actually cut suggested that the amenity value of forests was four times that of timber. Advantages of the methods reside in the ability to model complex ecosystem processes with simple probability matrices, and in the rich MDP theory and algorithms. Limitations include the difficulty of defining a space set large enough for accurate discretization, but small enough for practical application.

Invisible stock loss often occurs in inventory management systems, which is usually caused by quality deterioration, theft, and misplacement and makes inventory record inaccurate. Inaccurate inventory record will bring significant negative impact on the performance of inventory control and leads to severe stock-out. To eliminate this negative impact, in this paper, we develop a Markov decision process (MDP) model for an inventory control system with invisible stock loss, inaccurate record, and replenishment lead-time. Based on the MDP model, a robust inventory control policy is constructed by using the policy iteration technique. From numerical experiments, we find that the robust policy still has a reorder point/order-up-to level structure i.e., an (s, S) structure, and can significantly reduce the impact of inventory inaccuracy.

Linear programming formulation for non-stationary, finite-horizon Markov decision process models

The past decade has witnessed the shift from calling taxis within sight to using applications to send location information which enables e-hailing platforms to match passengers to drivers out of sight. After receiving orders, e-hailing platforms search for a suitable driver to accept the order to complete the matching. At present, platforms mainly set a fixed matching distance according to passengers’ locations for finding available drivers in this distance, but they do not consider if there are enough drivers available for their passengers. In order to find the properest driver which enables the decrease of passengers’ waiting time (i.e., from the time the passenger places the order to the time the driver accepts the order), we propose a model to dynamically change matching distance. The purpose is to find the best matching distance in different supply (i.e., the number of drivers) and demand (i.e., the number of passengers) circumstances. We obtain the required probability information by extracting data from multiple datasets in different supply and demand regions. Later, a Markov Decision Process (MDP) model is established to solve the optimal dynamically changing matching distance policy. At the same time, we propose a reward function to measure the waiting time of passengers and the distance between the driver and the passenger boarding point. Finally, we compared MDP model with three benchmark models, the model of fixed matching distance, dynamic programming matching distance, and linear matching distance. Through comparison, it is concluded that the policy of dynamically changing matching distance based on the MDP model we proposed is best applied. The policy we proposed balance the waiting time of passengers and the distance between the driver and the passenger boarding point, improve platform efficiency.KeywordsMarkov decision processDynamic matching distanceOrder dispatchUrban computingE-hailing

Dynamic selling of quality-graded products under demand uncertainties

Average case analysis of the classical algorithm for Markov decision processes with Büchi objectives

Women with BRCA1/2 mutations have higher risk for breast and ovarian cancers. Available intervention actions include prophylactic surgeries and breast screening, which vary significantly in cost, cancer prevention, and in resulting death from other causes. We present a model designed to yield optimal intervention strategies for mutation carriers between the ages of 30 and 65 and any prior intervention history. A Markov decision process (MDP) model is developed that considers yearly state transitions for the mutation carriers and state dependent intervention actions. State is defined as a vector comprising mutation type, health states, prior intervention actions, and age. A discounted value iteration algorithm is used to obtain optimal strategies from the MDP model using both cost and quality-adjusted life years (QALYs) as rewards. The results from MDP model show that for 30-year-old women with BRCA1 mutation and no prior intervention history, the cost-optimal strategy is a combination of prophylactic mastectomy (PM) and prophylactic oophorectomy (PO) at age 30 with no screening afterwards. Whereas, the QALYs-optimal strategy suggests PO at age 30 and PM at age 50 with screening afterwards. For BRCA2 mutation carriers at age 30, the cost-optimal strategy is PO at age 30, PM at age 40, and yearly screening only after age 56. Corresponding QALYs-optimal strategy is PM at age 40 with screening. Strategies for all other ages (31 to 65) are obtained and presented. It is also demonstrated that the cost-optimal strategies offer near maximum survival rate and near minimum cancer incidence rates by age 70, when compared to other ad hoc strategies.

As a typical scenario in future generation communication network applications, UAV-assisted communication can perform autonomous data delivery for massive machine type communication (mMTC), where the data generated from Internet of Things (IoT) devices can be carried and delivered to the corresponding locations with no direct communication channels to the IoT devices. Wireless energy transfer technique can recharge the UAV when the system is in operation, assisting the UAV to continuously collect and deliver data. In this work, we formulate a Markov decision process (MDP) model to describe the energy and data transfer optimization problem for the UAV. To maximize the long-term utility of the UAV, the MDP model is solved by value iteration algorithm to obtain the optimal strategies of the UAV to collect data, deliver data, and receive transferred energy to replenish on-device battery energy storage. Furthermore, to tackle the issues of system state uncertainties, partially observable states, and large state space in UAV-assisted communication systems, we extend the MDP model and solve it by using a Q -learning and a deep reinforcement learning (DRL) schemes. Simulations and numerical results validate that, compared with baseline schemes, the proposed MDP model with DRL based scheme can achieve better wireless energy and data transfer strategies in terms of the higher long-term utility of the UAV.

We consider approximate policy evaluation for finite state and action Markov de- cision processes (MDP) with the least squares temporal difference (LSTD) algorithm, LSTD(λ), in an exploration-enhanced learning context, where policy costs are computed from observations of a Markov chain different from the one corresponding to the policy under evaluation. We establish for the discounted cost criterion that LSTD(λ) converges almost surely under mild, minimal conditions. We also analyze other properties of the iterates involved in the algorithm, including convergence in mean and boundedness. Our analysis draws on theories of both finite space Markov chains and weak Feller Markov chains on a topological space. Our results can be applied to other temporal difference algorithms and MDP models. As examples, we give a convergence analysis of a TD(λ) algorithm and extensions to MDP with compact state and action spaces, as well as a convergence proof of a new LSTD algorithm with state-dependent λ-parameters.

This research proposes an algorithm to solve overbooking scheduling problem for outpatient hospitals with multiple providers and high patient demand. Assuming there is reward and system cost for serving each patient, and the decision maker in the hospital can decide the amount of resources to assign to each patient. Regardless of the random patient arrivals and departures, a novel model of the Continuous-Time Markov Decision Process (CTMDP) is explored, our objective in this paper is to find an optimal policy to achieve maximum total discounted expected reward starting from any initial states. Further more, to solve the computation complexity of CTMDP models when the action and system state space is large, genetic algorithm (GA) is proposed to search the optimal solution, which can be calculated in a parallel way thus reducing the computation time for the optimal policy.KeywordsOverbooking appointment schedulingOutpatient servicesGenetic algorithmsOptimal control policyContinuous-Time Markov Decision Process (CTMDP)

Given a model and a specification, the fundamental model-checking problem asks for algorithmic verification of whether the model satisfies the specification. We consider graphs and Markov decision processes (MDPs), which are fundamental models for reactive systems. One of the very basic specifications that arise in verification of reactive systems is the strong fairness (aka Streett) objective. Given different types of requests and corresponding grants, the objective requires that for each type, if the request event happens infinitely often, then the corresponding grant event must also happen infinitely often. All \(\omega \)-regular objectives can be expressed as Streett objectives and hence they are canonical in verification. To handle the state-space explosion, symbolic algorithms are required that operate on a succinct implicit representation of the system rather than explicitly accessing the system. While explicit algorithms for graphs and MDPs with Streett objectives have been widely studied, there has been no improvement of the basic symbolic algorithms. The worst-case numbers of symbolic steps required for the basic symbolic algorithms are as follows: quadratic for graphs and cubic for MDPs. In this work we present the first sub-quadratic symbolic algorithm for graphs with Streett objectives, and our algorithm is sub-quadratic even for MDPs. Based on our algorithmic insights we present an implementation of the new symbolic approach and show that it improves the existing approach on several academic benchmark examples.

This paper is the first of two papers that present and evaluate an approach for determining suboptimal policies for large-scale Markov decision processes (MDP). Part 1 is devoted to the determination of bounds that motivate the development and indicate the quality of the suboptimal design approach; Part 2 is concerned with the implementation and evaluation of the suboptimal design approach. The specific MDP considered is the infinite-horizon, expected total discounted cost MDP with finite state and action spaces. The approach can be described as follows. First, the original MDP is approximated by a specially structured MDP. The special structure suggests how to construct associated smaller, more computationally tractable MDP's. The suboptimal policy for the original MDP is then constructed from the solutions of these smaller MDP's. The key feature of this approach is that the state and action space cardinalities of the smaller MDP's are exponential reductions of the state and action space cardinalities of the original MDP.