Optimization of Cybercafe Internet Service and Costs under Stochastic Demand

Feature Reinforcement Learning: Part I. Unstructured MDPsGeneral-purpose, intelligent, learning agents cycle through sequences of observations, actions, and rewards that are complex, uncertain, unknown, and non-Markovian. On the other hand, reinforcement learning is well-developed for small finite state Markov decision processes (MDPs). Up to now, extracting the right state representations out of bare observations, that is, reducing the general agent setup to the MDP framework, is an art that involves significant effort by designers. The primary goal of this work is to automate the reduction process and thereby significantly expand the scope of many existing reinforcement learning algorithms and the agents that employ them. Before we can think of mechanizing this search for suitable MDPs, we need a formal objective criterion. The main contribution of this article is to develop such a criterion. I also integrate the various parts into one learning algorithm. Extensions to more realistic dynamic Bayesian networks are developed in Part II (Hutter, 2009c). The role of POMDPs is also considered there.

A new self-contained approach based on the Drazin generalized inverse is used to derive many basic results in discrete time, finite state Markov decision processes. A product form representation for the transition matrix of a stationary policy gives new derivations of the average reward evaluation equations, Laurent series expansions, as well as the finite test for Blackwell optimality. This representation also suggests new computational methods.

This paper extends finite state and action space Markov Decision Process (MDP) models by introducing a new type of measurement for the outcomes of actions. The new measurement allows to sequentially observe the next-state transition for taking an action, i.e., the actions are ordered and the next action outcome in the sequence is observed only if the current action is not chosen. The sequentially-observed MDP (SO-MDP) shares some properties with a standard MDP: among history dependent policies, Markovian ones are still optimal. SO-MDP policies have the advantage of producing better rewards than standard optimal MDP policies due to additional measurements. Computing these policies, on the other hand, is more complex and we present a linear programming based synthesis of the optimal decision policies for the finite horizon SO-MDPs. A simulation example of multiple autonomous agents is also provided to demonstrate the SO-MDP model and the proposed policy synthesis method.

In today’s fast-paced and competitive market place, organizations need every edge available to them to ensure success in planning and managing inventory of items with demand uncertainty. In such an effort, cost effective methods in determining optimal replenishment policies are paramount. In this paper, a mathematical model is proposed that optimize inventory replenishment policies of a periodic review inventory system under stochastic demand; with particular focus on malaria drugs in Ugandan pharmacies. Adopting a Markov decision process approach, the states of a Markov chain represent possible states of demand for drugs that treat malaria. Using weekly equal intervals, the decisions of whether or not to replenish additional units of drugs were made using discrete time Markov chains and dynamic programming over a finite period planning horizon. Empirical data was collected from two pharmacies in Uganda. The customer transactions of drugs were taken on a weekly basis; where data collected was analyzed and tested to establish the optimal replenishment policy and inventory costs of drugs. Results from the study indicated the existence of an optimal state-dependent replenishment policy and inventory costs of drugs at the respective pharmacies.

Standardized cargo network revenue management with dual channels under stochastic and time-dependent demand

Reinforcement Learning (RL) Markov Decision Processes is studied with an emphasis on the well-studied exploration problem. We first formulate and discuss a definition of “efficient” algorithms that is termed Probably Approximately Correct (PAC) in RL. Next we provide general sufficient conditions for such an algorithm that applies to several different modeling assumptions. The conditions can be used to demonstrate that efficient learning is possible in finite MDPs, with either a model-based or model-free approach, in factored MDPs, and in continuous MDPs with linear dynamics. In the reinforcement-learning (RL) problem (Sutton & Barto 1998), an agent acts in an unknown or incompletely known environment with the goal of maximizing an external reward signal. In the most standard mathematical formulation of the problem, the environment is modeled as a Markov Decision Process (MDP) and the goal of the agent is to obtain near-optimal discounted return. Over the years, many algorithms have been proposed for this problem, but analyses of their performances have been relatively scarce. In fact, until recently, most theoretical guarantees have been that certain algorithms will discover an optimal policy in the limit, after an infinite amount of experience. In contrast, several attempts have been made to study “Probably Approximately Correct” or PAC-MDP algorithms, which exhibit near-optimal behavior in polynomial time and experience. This paper discusses several extensions of those results. We present a theorem that provides sufficient conditions for an algorithm to be PAC-MDP. We examine these conditions and show how they can be applied to prove that efficient learning is possible in three interesting scenerios: finite MDPs (i.e. the “Tabular case”),

As one of the most popular supply chain management problems, the Vehicle Routing Problem (VRP) has been thoroughly studied in the last decades, most of these studies focus on deterministic problem where the customer demands are known in advance. But the Vehicle Routing Problem with Stochastic Demands (VRPSD) has not received enough consideration. In the VRPSD, the vehicle does not know the customer demands until the vehicle arrive to them. This paper use a hybrid algorithm for solving VRPSD, the hybrid algorithm based on Particle Swarm Optimization (PSO) Algorithm, combines a Greedy Randomized Adaptive Search Procedure (GRASP) algorithm, and Variable Neighborhood Search (VNS) algorithm. A real number encoding method is designed to build a suitable mapping between solutions of problem and particles in PSO. A number of computational studies, along with comparisons with other existing algorithms, showed that the proposed hybrid algorithm is a feasible and effective approach for Vehicle Routing Problem with Stochastic Demands.

An important component of the cyber-defense mechanism is the adequate staffing levels of its cybersecurity analyst workforce and their optimal assignment to sensors for investigating the dynamic alert traffic. The ever-increasing cybersecurity threats faced by today’s digital systems require a strong cyber-defense mechanism that is both reactive in its response to mitigate the known risk and proactive in being prepared for handling the unknown risks. In order to be proactive for handling the unknown risks, the above workforce must be scheduled dynamically so the system is adaptive to meet the day-to-day stochastic demands on its workforce (both size and expertise mix). The stochastic demands on the workforce stem from the varying alert generation and their significance rate, which causes an uncertainty for the cybersecurity analyst scheduler that is attempting to schedule analysts for work and allocate sensors to analysts. Sensor data are analyzed by automatic processing systems, and alerts are generated. A portion of these alerts is categorized to be significant , which requires thorough examination by a cybersecurity analyst. Risk, in this article, is defined as the percentage of significant alerts that are not thoroughly analyzed by analysts. In order to minimize risk, it is imperative that the cyber-defense system accurately estimates the future significant alert generation rate and dynamically schedules its workforce to meet the stochastic workload demand to analyze them. The article presents a reinforcement learning-based stochastic dynamic programming optimization model that incorporates the above estimates of future alert rates and responds by dynamically scheduling cybersecurity analysts to minimize risk (i.e., maximize significant alert coverage by analysts) and maintain the risk under a pre-determined upper bound. The article tests the dynamic optimization model and compares the results to an integer programming model that optimizes the static staffing needs based on a daily-average alert generation rate with no estimation of future alert rates (static workforce model). Results indicate that over a finite planning horizon, the learning-based optimization model, through a dynamic (on-call) workforce in addition to the static workforce, (a) is capable of balancing risk between days and reducing overall risk better than the static model, (b) is scalable and capable of identifying the quantity and the right mix of analyst expertise in an organization, and (c) is able to determine their dynamic (on-call) schedule and their sensor-to-analyst allocation in order to maintain risk below a given upper bound. Several meta-principles are presented, which are derived from the optimization model, and they further serve as guiding principles for hiring and scheduling cybersecurity analysts. Days-off scheduling was performed to determine analyst weekly work schedules that met the cybersecurity system’s workforce constraints and requirements.

Addresses the control design problem for discrete-time, finite-state Markov decision processes, when both risk-neutral and minimax objectives are of interest. We introduce the mixed risk-neutral/minimax objective and utilize results from risk-neutral and minimax control to derive an information state process and dynamic programming equations for the value function. We synthesize optimal control laws both on the finite and the infinite horizons. We study the effectiveness of both the mixed risk-neutral/minimax family and the risk-sensitive family of controllers as tools to tradeoff risk-neutral and minimax objectives. We conclude that the mixed risk-neutral/minimax family is more effective, at the cost of increased controller complexity.

We consider an inventory routing problem with stochastic demand where the demand is expressed in a probabilistic sense and it is dynamic in which the demand is gradually revealed at the end of each period. The problem is known as dynamic and stochastic inventory routing problem (DSIRP). DSIRP distribution network consists of a supplier and a set of retailers is considered. A limited number of single product is assumed to be produced by the supplier and a holding cost per unit product is incurred at both the supplier. The demand is assumed to follow the binomial probability distribution. We adopt an order-up-to level inventory policy and each unit of positive inventory at the retailer is incurred a holding cost, while the unmet demand (stock-out) is penalized. We assumed that the transportation of the product from supplier to the retailers is handled by a third party. The aim of DSIRP is to minimize the total expected cost which includes the inventory cost at supplier, inventory cost at retailers, stockout cost at retailers and transportation cost. The DSIRP is modeled as a Markov decision process (MDP) where the state in period t is represented as the inventory level for all retailers. A hybrid rollout algorithm where a Mixed Integer Linear Programming of the problem is used to obtain the control in each state. We also proposed an Artificial Bee Colony (ABC) algorithm to obtain a feasible and better controls. Computational experiments is conducted using an existing benchmark problem.

Contributors

In this paper, the Vehicle Routing Problem with Stochastic Demands (VRPSD) is considered where customer demands are normally distributed. We propose a new model for computing the expected length of a tour. Monte Carlo simulation is used to demonstrate the accuracy of the model on randomly generated test problems. It is assumed that the service policy is non-divisible, meaning that the entire demand at each customer must be served in a single visit by a unique vehicle.

Integrating millimeter Wave (mmWave) and long-term evolution (LTE) sub-6 GHz bands to provide wireless services for high-speed rails (HSRs) is a promising solution to balance the ever-rising service demand and operational cost. In this paper, we optimize the network interface selection of the mmWave and LTE interfaces for HSR communications, to minimize the operational cost while satisfying the service requests of passengers. The network interface selection problem is formulated as a finite-horizon discrete-time Markov decision process (MDP) problem. Based on the file requests and associated deadlines of a passenger and the train schedule, we solve the MDP problem ahead-of-time by using dynamic programming (DP) and achieve optimal interface selection for each waypoint along the railways. Further, by exploiting the subadditivity and superadditivity of the considered problem, we reveal that an optimal switch of interfaces is only triggered at the moment when the remaining undelivered data or the elapsed time reaches their specific thresholds. Based on the thresholds, the optimal interface selection can be directly achieved for every waypoint, hence eliminating the need for DP and substantially reducing the computational complexity. The proposed algorithms are evaluated with practical train schedules. Simulation results show that the proposed algorithms can substantially reduce the operational cost by 35% in our considered simulation setting, as compared to existing alternatives.

This chapter presents an overview of simulation-based techniques useful for solving Markov decision processes (MDPs). MDPs model problems of sequential decision-making under uncertainty, in which decisions made in each state collectively affect the trajectory of the states visited by the system over a time horizon of interest. Traditionally, MDPs have been solved via dynamic programming (DP), which requires the transition probability model that is difficult to derive in many realistic settings. The use of simulation for solving MDPs allows us to bypass the transition probability model and solve large-scale MDPs considered intractable to solve by traditional DP. The simulation-based methodology for solving MDPs, which like DP is also rooted in the Bellman equations, goes by names such as reinforcement learning, neuro-DP, and approximate or adaptive DP. We begin with a description of algorithms for infinite-horizon discounted reward MDPs, followed by the same for infinite-horizon average reward MDPs. Then we present a discussion on finite-horizon MDPs. For each problem considered, we present a step-by-step description of a selected group of algorithms. In making this selection, we have attempted to blend the old and the classical with more recent developments. Finally, after touching upon extensions and convergence theory, we conclude with a brief summary of some applications and directions for future research.

The inventory routing problem (IRP) combines inventory management and delivery route‐planning decisions. This work presents a simheuristic approach that integrates Monte Carlo simulation within a variable neighborhood search (VNS) framework to solve the multiperiod IRP with stochastic customer demands. In this realistic variant of the problem, our goal is to establish the optimal refill policies for each customer–period combination, that is, those individual refill policies that minimize the total expected cost over the periods. This cost is the aggregation of both expected inventory and routing costs. Our simheuristic algorithm allows to consider the inventory changes between periods generated by the realization of the random demands in each period, which have an impact on the quantities to be delivered in the next period and, therefore, on the associated routing plans. A range of computational experiments are carried out in order to illustrate the potential of our simulation–optimization approach.