Linear programming formulation for non-stationary, finite-horizon Markov decision process models

Faustmann's formula gives the land value, or the forest value of land with trees, under deterministic assumptions regarding future stand growth and prices, over an infinite horizon. Markov decision process (MDP) models generalize Faustmann's approach by recognizing that future stand states and prices are known only as probabilistic distributions. The objective function is then the expected discounted value of returns, over an infinite horizon. It gives the land or the forest value in a stochastic environment. In MDP models, the laws of motion between stand-price states are Markov chains. Faustmann's formula is a special case where the probability of movement from one state to another is equal to unity. MDP models apply whether the stand state is bare land, or any state with trees, be it even- or uneven-aged. Decisions change the transition probabilities between stand states through silvicultural interventions. Decisions that maximize land or forest value depend only on the stand-price state, independently of how it was reached. Furthermore, to each stand-price state corresponds one single best decision. The solution of the MDP gives simultaneously the best decision for each state, and the forest value (land plus trees), given the stand state and following the best policy. Numerical solutions use either successive approximation, or linear programming. Examples with deterministic and stochastic cases show in particular the convergence of the MDP model to Faustmann's formula when the future is assumed known with certainty. In this deterministic environment, Faustmann's rule is independent of the distribution of stands in the forest. FOR. SCI. 47(4):466–474.

Towards enhanced threat modelling and analysis using a Markov Decision Process

Invisible stock loss often occurs in inventory management systems, which is usually caused by quality deterioration, theft, and misplacement and makes inventory record inaccurate. Inaccurate inventory record will bring significant negative impact on the performance of inventory control and leads to severe stock-out. To eliminate this negative impact, in this paper, we develop a Markov decision process (MDP) model for an inventory control system with invisible stock loss, inaccurate record, and replenishment lead-time. Based on the MDP model, a robust inventory control policy is constructed by using the policy iteration technique. From numerical experiments, we find that the robust policy still has a reorder point/order-up-to level structure i.e., an (s, S) structure, and can significantly reduce the impact of inventory inaccuracy.

Women with BRCA1/2 mutations have higher risk for breast and ovarian cancers. Available intervention actions include prophylactic surgeries and breast screening, which vary significantly in cost, cancer prevention, and in resulting death from other causes. We present a model designed to yield optimal intervention strategies for mutation carriers between the ages of 30 and 65 and any prior intervention history. A Markov decision process (MDP) model is developed that considers yearly state transitions for the mutation carriers and state dependent intervention actions. State is defined as a vector comprising mutation type, health states, prior intervention actions, and age. A discounted value iteration algorithm is used to obtain optimal strategies from the MDP model using both cost and quality-adjusted life years (QALYs) as rewards. The results from MDP model show that for 30-year-old women with BRCA1 mutation and no prior intervention history, the cost-optimal strategy is a combination of prophylactic mastectomy (PM) and prophylactic oophorectomy (PO) at age 30 with no screening afterwards. Whereas, the QALYs-optimal strategy suggests PO at age 30 and PM at age 50 with screening afterwards. For BRCA2 mutation carriers at age 30, the cost-optimal strategy is PO at age 30, PM at age 40, and yearly screening only after age 56. Corresponding QALYs-optimal strategy is PM at age 40 with screening. Strategies for all other ages (31 to 65) are obtained and presented. It is also demonstrated that the cost-optimal strategies offer near maximum survival rate and near minimum cancer incidence rates by age 70, when compared to other ad hoc strategies.

As a typical scenario in future generation communication network applications, UAV-assisted communication can perform autonomous data delivery for massive machine type communication (mMTC), where the data generated from Internet of Things (IoT) devices can be carried and delivered to the corresponding locations with no direct communication channels to the IoT devices. Wireless energy transfer technique can recharge the UAV when the system is in operation, assisting the UAV to continuously collect and deliver data. In this work, we formulate a Markov decision process (MDP) model to describe the energy and data transfer optimization problem for the UAV. To maximize the long-term utility of the UAV, the MDP model is solved by value iteration algorithm to obtain the optimal strategies of the UAV to collect data, deliver data, and receive transferred energy to replenish on-device battery energy storage. Furthermore, to tackle the issues of system state uncertainties, partially observable states, and large state space in UAV-assisted communication systems, we extend the MDP model and solve it by using a Q -learning and a deep reinforcement learning (DRL) schemes. Simulations and numerical results validate that, compared with baseline schemes, the proposed MDP model with DRL based scheme can achieve better wireless energy and data transfer strategies in terms of the higher long-term utility of the UAV.

In the coronavirus epidemic, many Chinese hospitals have established buffer zones to prevent the spread and transmission of the virus. The buffer zone is a monitored and separate area where the patients who need hospitalizations after the quick treatments in the emergency department can temporarily wait for the Covid-19 test and receive some healthcare services to stabilize their conditions. Because the beds in the buffer zones are limited, the managers face the patient admission control problem for the buffer zone. This management and control problem is challenging since the patient arrivals are uncertain, and the patients’ conditions are different. In this paper, we build the infinite- and finite-horizon Markov decision process (MDP) models for this problem. We use the uniformization method to discretize the patient flow. We propose various iteration algorithms to solve the MDP models and obtain the optimal and threshold policies. Numerical experiments validate the advantages of the policies obtained by the algorithms in this paper over the current policies of hospitals. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —The ongoing COVID-19 pandemic has been causing enormous damage to people’s health, jobs, and well-being. COVID-19 has affected almost all countries globally and has changed the operation mode of the healthcare system, especially the hospitals. The hospitals are the frontlines of healthcare service and the battle with the COVID-19 pandemic. This article is motivated by our collaborations with hospitals in Shanghai, China. In China, many hospitals establish buffer zones: a monitored area where the patients who need hospitalizations after the quick treatments in the emergency department can temporarily wait for the Covid-19 test and receive some healthcare services to stabilize their conditions. Because the zone’s capacity is limited, the managers must make dynamic patient admission control decisions according to multiple factors, such as patients’ health conditions and the usage of beds in the zone. We propose two MDP models to solve this complex problem. Several iteration algorithms are designed to solve the MDP models and obtain the optimal and threshold policies. Based on hospitals’ real-life data, we show the methods presented in this paper can help hospital managers make more reasonable decisions. Although we focus on the hospital’s buffer zone in China, the methodology and approach for this problem can be extended to other practical hospital management scenarios in the coronavirus pandemic. For example, For example, some hospitals have admission control problems for coronavirus patients due to hospital capacity limitations. The hospital has to decide if a patient is accepted as an inpatient or suggested to home quarantine. In such a case, the admission control problem can also be solved by the methodologies in the paper.

The past decade has witnessed the shift from calling taxis within sight to using applications to send location information which enables e-hailing platforms to match passengers to drivers out of sight. After receiving orders, e-hailing platforms search for a suitable driver to accept the order to complete the matching. At present, platforms mainly set a fixed matching distance according to passengers’ locations for finding available drivers in this distance, but they do not consider if there are enough drivers available for their passengers. In order to find the properest driver which enables the decrease of passengers’ waiting time (i.e., from the time the passenger places the order to the time the driver accepts the order), we propose a model to dynamically change matching distance. The purpose is to find the best matching distance in different supply (i.e., the number of drivers) and demand (i.e., the number of passengers) circumstances. We obtain the required probability information by extracting data from multiple datasets in different supply and demand regions. Later, a Markov Decision Process (MDP) model is established to solve the optimal dynamically changing matching distance policy. At the same time, we propose a reward function to measure the waiting time of passengers and the distance between the driver and the passenger boarding point. Finally, we compared MDP model with three benchmark models, the model of fixed matching distance, dynamic programming matching distance, and linear matching distance. Through comparison, it is concluded that the policy of dynamically changing matching distance based on the MDP model we proposed is best applied. The policy we proposed balance the waiting time of passengers and the distance between the driver and the passenger boarding point, improve platform efficiency.KeywordsMarkov decision processDynamic matching distanceOrder dispatchUrban computingE-hailing

Dynamic selling of quality-graded products under demand uncertainties

Vehicle Edge Computing (VEC) leverages compact cloud computing at the mobile network edge to meet the processing and latency needs of vehicles. By bringing computation closer to the vehicles, VEC reduces data transmission, minimizes latency, and boosts performance for compute-intensive applications. However, during peak hours of urban road traffic, the scarce computational resources available at edge servers could pose challenges in fulfilling the processing needs of vehicles. Introducing Unmanned Aerial Vehicles (UAVs) as supplementary edge computing nodes could significantly mitigate the aforementioned issue. In this paper, we propose a flexible edge computing framework in which a fleet of UAVs function as mobile computational service providers, offering computation offloading services to multiple vehicles. We design and optimize a computation offloading model for the UAV-enabled vehicle edge computing environment. The proposed model tackles the task offloading challenge, aiming to optimize UAV revenue and task processing efficiency while considering the constraints of UAVs’ restricted computational power and energy resources. Towards this end, our model jointly considers two key factors: task partitioning and computational resource allocation. To tackle the challenges posed by the aforementioned non-convex optimization problem, we construct a Markov Decision Process (MDP) model for the multi-UAV-enabled mobile edge computing system and introduce an innovative Multi-Agent Deep Reinforcement Learning (MADRL) framework addressing the decision-making challenge represented by MDP model. Comprehensive simulation outcomes illustrate that our devised task offloading technique outperforms other optimization methods.

This paper analyzes how triage classification errors and blood transfusion kits impact military medical evacuation (MEDEVAC) system performance with regards to dispatching policies. A discounted, infinite-horizon Markov decision process (MDP) model is formulated to analyze the MEDEVAC dispatching problem. A notional, representational scenario based in Azerbaijan is utilized to compare the MDP-generated policies to current practices. Results reveal the MDP-generated dispatching policy outperforms the currently practiced dispatching policy by 2.18%. Triage classification errors negatively impact system performance between 0.6% and 2.5% for the scenarios analyzed. Moreover, the inclusion of blood transfusion kits on board aircraft increase MEDEVAC system performance between 2.83% and 4.37%, depending on which units are equipped.

This paper extends finite state and action space Markov Decision Process (MDP) models by introducing a new type of measurement for the outcomes of actions. The new measurement allows to sequentially observe the next-state transition for taking an action, i.e., the actions are ordered and the next action outcome in the sequence is observed only if the current action is not chosen. The sequentially-observed MDP (SO-MDP) shares some properties with a standard MDP: among history dependent policies, Markovian ones are still optimal. SO-MDP policies have the advantage of producing better rewards than standard optimal MDP policies due to additional measurements. Computing these policies, on the other hand, is more complex and we present a linear programming based synthesis of the optimal decision policies for the finite horizon SO-MDPs. A simulation example of multiple autonomous agents is also provided to demonstrate the SO-MDP model and the proposed policy synthesis method.

The study proposes a Markov Decision Process framework-based method for ensuring AM features are accurate in real time. The capability to make decisions in stochastic circumstances is facilitated by the Markov Decision Process framework, a mathematical model. Recasting the issue of AM quality assurance as a Markov decision process enables the objective of real-time optimization of process parameters and material attributes to ensure high-quality printing. In adjusting printing process parameters and material properties, the Markov Decision Process model considers the state of space, action space, transition probabilities, and rewards associated with such modifications. The objective is to identify the optimal policy that maximizes quality output while minimizing errors and rejections. In particular, the proposed approach accomplishes this through the utilization of machine learning and sensor data analysis. Temperature and pressure are among the process parameters and material qualities that are monitored in real time by sensors integrated into the additive manufacturing system. To ascertain any inconsistencies in quality among the data gathered by the sensors, statistical methodologies and machine learning algorithms are implemented. Regression analysis and control charts are two examples. Optimization is an additional component of the strategy. The Markov Decision Process framework incorporates optimization algorithms, such as value iteration, to determine the optimal approach for decision-making. By continuously updating the value function and policy, the technique learns and adapts to the dynamic nature of the printing process, ensuring a steady improvement in quality.

With the rapid development of electric vehicles (EVs), the dramatic rise in the demand for electricity is creating heavy pressure on local grids. The combination of renewable energy and EV charging stations (EVCSs) provides a promising solution for alleviating the scarcity of electricity. In this paper, a finite-horizon Markov decision process (MDP) model is proposed for the optimal control of a photovoltaic (PV)-assisted EVCS in a university campus. The proposed model employs the vehicle-to-grid (V2G) technology to provide ancillary services and takes dynamic electricity price and the uncertainty of the EV owners’ parking behaviors into consideration. To guarantee computational efficiency, an adapted bounded real-time dynamic programming (BRTDP) algorithm is developed as the solution technique of the MDP model. Numerical simulations based on the dynamic electricity price of France, real PV data, and the vehicle parking patterns in a university campus are conducted to demonstrate the effectiveness of the proposed energy management system (EMS). Simulation results show that the EMS can reduce the total costs by more than 55% and 29% in summer and winter compared to the conventional charging policy, while guaranteeing the satisfaction rate of the demand for EV-charging without knowing the departure times of EVs a priori.

Markov decision process (MDP) models have been used to evaluate the performance of policies in various domains, such as treatment planning in medical decision making. However, in practice, decision makers may prefer other strategies that are not statistically different from the actions in their initial policy of interest. To allow for decision makers' expertise and provide flexibility in implementing policies, this paper introduces a new framework for identifying sets of similar-performing actions in finite MDP models. These sets are obtained by combining a simulation-based dynamic programming algorithm for policy evaluation with a simulation-based statistical multiple comparisons procedure. The framework in this paper is applied in a medical decision-making setting to find sets of similar-performing antihypertensive treatment choices for a set of clinically representative patient profiles.

Effects of stochastic interest rates in decision making under risk: A Markov decision process model for forest management