A Rollout Algorithm for Multichain Markov Decision Processes with Average Cost

Markov decision processes (MDPs) are widely used to model stochastic systems to deduce optimal decision-making policies. As the transition probabilities are usually unknown in MDPs, simulation-based policy improvement (SBPI) using a base policy to derive optimal policies when the state transition probabilities are unknown is suggested. However, estimating the Q-value of each action to determine the best action in each state requires many simulations, which results in efficiency problems for SBPI. In this study, we propose a method to improve the overall efficiency of SBPI using optimal computing budget allocation (OCBA) based on accumulated samples. Previous works have mainly focused on improving SBPI efficiency for a single state and without using the previous simulation samples. In contrast, the proposed method improves the overall efficiency until an optimal policy can be found in consideration of the state traversal property of the SBPI. The proposed method accumulates simulation samples across states to estimate the unknown transition probabilities. These probabilities are then used to estimate the mean and variance of the Q-value for each action, which allows the OCBA to allocate the simulation budget efficiently to find the best action in each state. As the SBPI traverses the state, the accumulated samples allow appropriate allocation of OCBA; thus, the optimal policy can be obtained with a lower budget. The experimental results demonstrate the improved efficiency of the proposed method compared to previous works.

This paper is the first of two papers that present and evaluate an approach for determining suboptimal policies for large-scale Markov decision processes (MDP). Part 1 is devoted to the determination of bounds that motivate the development and indicate the quality of the suboptimal design approach; Part 2 is concerned with the implementation and evaluation of the suboptimal design approach. The specific MDP considered is the infinite-horizon, expected total discounted cost MDP with finite state and action spaces. The approach can be described as follows. First, the original MDP is approximated by a specially structured MDP. The special structure suggests how to construct associated smaller, more computationally tractable MDP's. The suboptimal policy for the original MDP is then constructed from the solutions of these smaller MDP's. The key feature of this approach is that the state and action space cardinalities of the smaller MDP's are exponential reductions of the state and action space cardinalities of the original MDP.

Selecting a subset of the best solutions among large-scale problems is an important area of research. When the alternative solutions are stochastic in nature, then it puts more burden on the problem. The objective of this paper is to select a set that is likely to contain the actual best solutions with high probability. If the selected set contains all the best solutions, then the selection is denoted as correct selection. We are interested in maximizing the probability of this selection; P(CS). In many cases, the available computation budget for simulating the solution set in order to maximize P(CS) is limited. Therefore, instead of distributing these computational efforts equally likely among the alternatives, the optimal computing budget allocation (OCBA) procedure came to put more effort on the solutions that have more impact on the selected set. In this paper, we derive formulas of how to distribute the available budget asymptotically to find the approximation of P(CS). We then present a procedure that uses OCBA with the ordinal optimization (OO) in order to select the set of best solutions. The properties and performance of the proposed procedure are illustrated through a numerical example. Overall results indicate that the procedure is able to select a subset of the best systems with high probability of correct selection using small number of simulation samples under different parameter settings.

Choosing suitable buffer setups for network-flow production lines of automated manufacturing systems to augment throughputs is a pragmatic issue. In this work, an approach merging artificial immune system (AIS) and ordinal optimization (OO) is developed to determine an optimal buffer resource allocation of a network-flow production line for maximizing the throughput. The proposed approach consists of two levels. The first level is using AIS assisted by a rough estimation to select a candidate subset of solutions. The second level is to identify an excellent solution among the solutions obtained from the first level using optimal computing budget allocation (OCBA) technique. We have tested the proposed approach on a three-stage ten-node network-flow production line and compared the test results with those obtained by three swarm intelligence methods with accurate estimation. Test results demonstrate that the proposed approach can obtain an excellent solution within a reasonable computing time and outperforms the three swarm intelligence methods.

For the design of call admission control (CAC) policy in wireless networks, how to decrease the average system cost is a key issue. In this paper, we study the optimal admission policy for minimizing the system cost. By modeling the admission control problem into a Markov decision process (MDP) and analyzing the corresponding value function, we obtain some monotonicity properties of the optimal policy. These properties suggest that the optimal admission control policy for the bandwidth asymmetry wireless networks should have a threshold structure and the threshold specified for a class of calls may change with the system state. Due to the prohibitively high complexity for computing the thresholds in a system with large state space, we propose a heuristic CAC policy called call-rate- based dynamic threshold (CRDT) policy to approximate the theoretical optimal policy based on the insights we obtain from the modeling and the analytical study on the properties of the optimal policy. The CRDT policy is efficient and can be easily implemented. The numerical results show that the performance of average system cost of the proposed CRDT policy is close to that of the optimal policy from the MDP model and is better than that of some known existing CAC schemes, including those performing well in bandwidth asymmetry wireless networks.

An Operational Framework for the Adoption and Integration of New Diagnostic Tests into Emergency Department Workflow

Economic MPC of Markov Decision Processes: Dissipativity in undiscounted infinite-horizon optimal control

This paper extends finite state and action space Markov Decision Process (MDP) models by introducing a new type of measurement for the outcomes of actions. The new measurement allows to sequentially observe the next-state transition for taking an action, i.e., the actions are ordered and the next action outcome in the sequence is observed only if the current action is not chosen. The sequentially-observed MDP (SO-MDP) shares some properties with a standard MDP: among history dependent policies, Markovian ones are still optimal. SO-MDP policies have the advantage of producing better rewards than standard optimal MDP policies due to additional measurements. Computing these policies, on the other hand, is more complex and we present a linear programming based synthesis of the optimal decision policies for the finite horizon SO-MDPs. A simulation example of multiple autonomous agents is also provided to demonstrate the SO-MDP model and the proposed policy synthesis method.

Ordinal optimization is an efficient technique to choose and rank various engineering designs that require time-consuming discrete-event simulations. Optimal computing budget allocation (OCBA) has been an important tool to enhance its efficiency such that the best design is selected in a timely fashion. It, however, fails to address the issue of selecting the best and worst designs efficiently. The need to select both rapidly given a fixed computing budget has arisen from many applications. This work develops a new OCBA-based approach for selecting both best and worst designs at the same time. Its theoretical foundation is laid. Our numerical results show that it can well outperform all the existing methods in terms of probability of correct selection and computational efficiency.

Markov decision process (MDP) models are widely used for modeling sequential decision-making problems that arise in engineering, economics, computer science, and the social sciences. Many real-world problems modeled by MDPs have huge state and/or action spaces, giving an opening to the curse of dimensionality and so making practical solution of the resulting models intractable. In other cases, the system of interest is too complex to allow explicit specification of some of the MDP model parameters, but simulation samples are readily available (e.g., for random transitions and costs). For these settings, various sampling and population-based algorithms have been developed to overcome the difficulties of computing an optimal solution in terms of a policy and/or value function. Specific approaches include adaptive sampling, evolutionary policy iteration, evolutionary random policy search, and model reference adaptive search. This substantially enlarged new edition reflects the latest developments in novel algorithms and their underpinning theories, and presents an updated account of the topics that have emerged since the publication of the first edition. Includes: innovative material on MDPs, both in constrained settings and with uncertain transition properties; game-theoretic method for solving MDPs; theories for developing roll-out based algorithms; and details of approximation stochastic annealing, a population-based on-line simulation-based algorithm. The self-contained approach of this book will appeal not only to researchers in MDPs, stochastic modeling, and control, and simulation but will be a valuable source of tuition and reference for students of control and operations research.

Convergence rate analysis for optimal computing budget allocation algorithms

The main objective of this paper is to synthesize optimal decision-making policies for a finite-horizon Markov Decision Process (MDP) while satisfying a safety constraint that imposes an upper bound on the state probability density function (pdf) of the underlying Markov Chain (MC) for all time steps. The classical approach based on state-action frequencies for constrained MDPs yields decision policies that provide safety constraint satisfaction for stationary distributions (i.e., asymptotically), but not necessarily providing safety during the transient regime. This paper introduces a new synthesis method for randomized Markovian policies for finite-horizon MDPs, where the safety constraint satisfaction is guaranteed for both the transient and the stationary distributions independent from initial state (i.e., providing safe policies for the worst-case analysis). An efficient Linear Programming (LP) based synthesis algorithm is proposed, which produces a convex set of feasible policies and ensures that the expected total reward is above a computable lower-bound. A simulation example of a swarm of autonomous agents is also presented to demonstrate the practical importance of having safe policies.

This study is concerned with Markov Decision Processes with uncertain transition matrices. In the discounted case, the Bayesian analysis of this model is studied.We define an adaptive policy and a learning policy and show that there exists, for any ∈> 0 an ∈-optimal and learning policy. In the average case, the non-Bayesian analysis of this model is studied and an optimal adaptive policy is constructed.

Q-learning is a very popular reinforcement learning algorithm being proven to converge to optimal policies in Markov decision processes. However, Q-learning shows artifacts if the optimal action is played with a low probability, a situation that may arise due to wrong initialization of Q-values or due to convergence to an almost pure policy after which a change in the environment makes another action optimal. These artifacts was resolved in literature by the variant Frequency Adjusted Q-learning (FAQL). However, FAQL also suffered from practical concerns that limited the policy subspace for which the behavior was improved. Here, we introduce the Repeated Update Q-learning (RUQL), a variant of Q-learning that resolves the undesirable artifacts of Q-learning without the practical concerns of FAQL. We show (both theoretically and experimentally) the similarities and differences between RUQL and FAQL (the closest state-of-the-art). Experimental results verify the theoretical insights and show how RUQL outperforms FAQL and QL in non-stationary environments.

Stochastic point location (SPL) is to search for a target point on the line in stochastic environment. An SPL solver can be described as a Learning Machine (LM) attempting to locate a target point on a line. By using the prompts from stochastic environment, possibly erroneous, the LM moves along the line yielding updated estimates to approximate the target point. This paper proposes an SPL algorithm based on Optimal Computing Budget Allocation (OCBA), named as SPL-OCBA, which employs OCBA and the historical sample information to guide to the location of a target point in stationary and stochastic environment. The proposed algorithm partitions or combines the subintervals of the target line adaptively. Then, OCBA considers such subintervals as its designs and allocates the sample budget for them based on the historical data, thereby resulting in a new method. Extensive experiments show that the newly proposed algorithm is significantly more efficient than the existing ones.