Assessing the appropriateness of using markov decision processes for RF spectrum management

Towards enhanced threat modelling and analysis using a Markov Decision Process

Modeling and analyzing a physician clinic environment using discrete-event (visual) simulation

Faustmann's formula gives the land value, or the forest value of land with trees, under deterministic assumptions regarding future stand growth and prices, over an infinite horizon. Markov decision process (MDP) models generalize Faustmann's approach by recognizing that future stand states and prices are known only as probabilistic distributions. The objective function is then the expected discounted value of returns, over an infinite horizon. It gives the land or the forest value in a stochastic environment. In MDP models, the laws of motion between stand-price states are Markov chains. Faustmann's formula is a special case where the probability of movement from one state to another is equal to unity. MDP models apply whether the stand state is bare land, or any state with trees, be it even- or uneven-aged. Decisions change the transition probabilities between stand states through silvicultural interventions. Decisions that maximize land or forest value depend only on the stand-price state, independently of how it was reached. Furthermore, to each stand-price state corresponds one single best decision. The solution of the MDP gives simultaneously the best decision for each state, and the forest value (land plus trees), given the stand state and following the best policy. Numerical solutions use either successive approximation, or linear programming. Examples with deterministic and stochastic cases show in particular the convergence of the MDP model to Faustmann's formula when the future is assumed known with certainty. In this deterministic environment, Faustmann's rule is independent of the distribution of stands in the forest. FOR. SCI. 47(4):466–474.

Women with BRCA1/2 mutations have higher risk for breast and ovarian cancers. Available intervention actions include prophylactic surgeries and breast screening, which vary significantly in cost, cancer prevention, and in resulting death from other causes. We present a model designed to yield optimal intervention strategies for mutation carriers between the ages of 30 and 65 and any prior intervention history. A Markov decision process (MDP) model is developed that considers yearly state transitions for the mutation carriers and state dependent intervention actions. State is defined as a vector comprising mutation type, health states, prior intervention actions, and age. A discounted value iteration algorithm is used to obtain optimal strategies from the MDP model using both cost and quality-adjusted life years (QALYs) as rewards. The results from MDP model show that for 30-year-old women with BRCA1 mutation and no prior intervention history, the cost-optimal strategy is a combination of prophylactic mastectomy (PM) and prophylactic oophorectomy (PO) at age 30 with no screening afterwards. Whereas, the QALYs-optimal strategy suggests PO at age 30 and PM at age 50 with screening afterwards. For BRCA2 mutation carriers at age 30, the cost-optimal strategy is PO at age 30, PM at age 40, and yearly screening only after age 56. Corresponding QALYs-optimal strategy is PM at age 40 with screening. Strategies for all other ages (31 to 65) are obtained and presented. It is also demonstrated that the cost-optimal strategies offer near maximum survival rate and near minimum cancer incidence rates by age 70, when compared to other ad hoc strategies.

As a typical scenario in future generation communication network applications, UAV-assisted communication can perform autonomous data delivery for massive machine type communication (mMTC), where the data generated from Internet of Things (IoT) devices can be carried and delivered to the corresponding locations with no direct communication channels to the IoT devices. Wireless energy transfer technique can recharge the UAV when the system is in operation, assisting the UAV to continuously collect and deliver data. In this work, we formulate a Markov decision process (MDP) model to describe the energy and data transfer optimization problem for the UAV. To maximize the long-term utility of the UAV, the MDP model is solved by value iteration algorithm to obtain the optimal strategies of the UAV to collect data, deliver data, and receive transferred energy to replenish on-device battery energy storage. Furthermore, to tackle the issues of system state uncertainties, partially observable states, and large state space in UAV-assisted communication systems, we extend the MDP model and solve it by using a Q -learning and a deep reinforcement learning (DRL) schemes. Simulations and numerical results validate that, compared with baseline schemes, the proposed MDP model with DRL based scheme can achieve better wireless energy and data transfer strategies in terms of the higher long-term utility of the UAV.

Invisible stock loss often occurs in inventory management systems, which is usually caused by quality deterioration, theft, and misplacement and makes inventory record inaccurate. Inaccurate inventory record will bring significant negative impact on the performance of inventory control and leads to severe stock-out. To eliminate this negative impact, in this paper, we develop a Markov decision process (MDP) model for an inventory control system with invisible stock loss, inaccurate record, and replenishment lead-time. Based on the MDP model, a robust inventory control policy is constructed by using the policy iteration technique. From numerical experiments, we find that the robust policy still has a reorder point/order-up-to level structure i.e., an (s, S) structure, and can significantly reduce the impact of inventory inaccuracy.

The past decade has witnessed the shift from calling taxis within sight to using applications to send location information which enables e-hailing platforms to match passengers to drivers out of sight. After receiving orders, e-hailing platforms search for a suitable driver to accept the order to complete the matching. At present, platforms mainly set a fixed matching distance according to passengers’ locations for finding available drivers in this distance, but they do not consider if there are enough drivers available for their passengers. In order to find the properest driver which enables the decrease of passengers’ waiting time (i.e., from the time the passenger places the order to the time the driver accepts the order), we propose a model to dynamically change matching distance. The purpose is to find the best matching distance in different supply (i.e., the number of drivers) and demand (i.e., the number of passengers) circumstances. We obtain the required probability information by extracting data from multiple datasets in different supply and demand regions. Later, a Markov Decision Process (MDP) model is established to solve the optimal dynamically changing matching distance policy. At the same time, we propose a reward function to measure the waiting time of passengers and the distance between the driver and the passenger boarding point. Finally, we compared MDP model with three benchmark models, the model of fixed matching distance, dynamic programming matching distance, and linear matching distance. Through comparison, it is concluded that the policy of dynamically changing matching distance based on the MDP model we proposed is best applied. The policy we proposed balance the waiting time of passengers and the distance between the driver and the passenger boarding point, improve platform efficiency.KeywordsMarkov decision processDynamic matching distanceOrder dispatchUrban computingE-hailing

Dynamic selling of quality-graded products under demand uncertainties

Linear programming formulation for non-stationary, finite-horizon Markov decision process models

Although Thrombolysis has been licensed in the UK since 2003, it is still administered only to a small percentage of eligible patients. We consider the impact of investing the impact of thrombolysis on important acute stroke services, and the effect on quality of life. The concept is illustrated using data from the Northern Ireland Stroke Service. Retrospective study. We first present results of survival analysis utilizing length of stay (LOS) for discharge destinations, based on data from the Belfast City Hospital (BCH). None of these patients actually received thrombolysis but from those who would have been eligible, we created two initial groups, the first representing a scenario where they received thrombolysis and the second comprising those who do not receive thrombolysis. On the basis of the survival analysis, we created several subgroups based on discharge destination. We then developed a discrete event simulation (DES) model, where each group is a patient pathway within the simulation. Coxian phase type distributions were used to model the group LOS. Various scenarios were explored focusing on cost-effectiveness across hospital, community and social services had thrombolysis been administered to these patients, and the possible improvement in quality of life, should the proportion of patients who are administered thrombolysis be increased. Our aim in simulating various scenarios for this historical group of patients is to assess what the cost-effectiveness of thrombolysis would have been under different scenarios; from this we can infer the likely cost-effectiveness of future policies. The cost of thrombolysis is offset by reduction in hospital, community rehabilitation and institutional care costs, with a corresponding improvement in quality of life. Our model suggests that provision of thrombolysis would produce moderate overall improvement to the service assuming current levels of funding.

MO3 COMPARING METHODS OF DATA SYNTHESIS: UPDATING PARAMETERS OF AN EXISTING PROBABILISTIC COST-EFFECTIVENESS MODEL

MO1 ADVANTAGES AND DISADVANTAGES OF MARKOV MODELS VERSUS DISCRETE EVENT MODELS FOR SIMULATING PATIENT FLOWS

Geographic Service Delivery for Endovascular Clot Retrieval: Using Discrete Event Simulation to Optimize Resources

One of the basic components of Industry 4.0 is the design of a flexible manufacturing system (FMS), which involves the choice of parameters to optimize its performance. Discrete event simulation (DES) models allow the user to understand the operation of dynamic and stochastic system performance and to support FMS diagnostics and design. In combination with DES models, optimization methods are often used to search for the optimal designs, which, above all, involve more than one objective function to be optimized simultaneously. These methods are called the multi-objective simulation–optimization (MOSO) method. Numerous MOSO methods have been developed in the literature, which spawned many proposed MOSO methods classifications. However, the performance of these methods is not guaranteed because there is an absence of comparative studies. Moreover, previous classifications have been focused on general MOSO methods and rarely related to the specific area of manufacturing design. For this reason, a new conceptual classification of MOSO used in FMS design is proposed. After that, four MOSO methods are selected, according to this classification, and compared through a detailed case study related to the FMS design problem. All of these methods studied are based on Design of Experiments (DoE). Two of them are metamodel-based approaches that integrate Goal Programming (GP) and Desirability Function (DF), respectively. The other two methods are not metamodel-based approaches, which integrate Gray Relational Analysis (GRA) and the VIKOR method, respectively. The comparative results show that the GP and VIKOR methods can result in better optimization than DF and GRA methods. Thus, the use of the simulation metamodel cannot prove its superiority in all situations.

In manufacturing plants, part of assembly is often performed by human workers and therefore its performance largely depends on humans (or human performances) rather than machines. However, most studies on human centred assembly systems using the modelling simulation methods do not consider or incorporate the effects of human performance that may also impact the overall system performance of such as an assembly line. The prediction of human performance or behaviour in a manufacturing system evaluation is often overlooked by researchers or system designers partially due to a lack of proper versions of existing simulation tools that can incorporate human attributes into an established simulation model. This paper presents a study by incorporating human attributes of learning and ageing into a discrete event simulation (DES) model based on a human centred assembly system. The effects of worker performance due to aging and learning were investigated and identified through a literature review [1]. The simulation result demonstrates that the worker performance may approach his/her full capacity at the age of 38 years old and this may decrease by an average of 1% from 38 to 40 years old, and 6% at the age of 45 years old. After the age of 45 years old, the decline rate of worker performance may slow down as it can be offset by accumulating worker experience through learning.