Markov Models of Policy Support for Technology Transitions

Towards enhanced threat modelling and analysis using a Markov Decision Process

Faustmann's formula gives the land value, or the forest value of land with trees, under deterministic assumptions regarding future stand growth and prices, over an infinite horizon. Markov decision process (MDP) models generalize Faustmann's approach by recognizing that future stand states and prices are known only as probabilistic distributions. The objective function is then the expected discounted value of returns, over an infinite horizon. It gives the land or the forest value in a stochastic environment. In MDP models, the laws of motion between stand-price states are Markov chains. Faustmann's formula is a special case where the probability of movement from one state to another is equal to unity. MDP models apply whether the stand state is bare land, or any state with trees, be it even- or uneven-aged. Decisions change the transition probabilities between stand states through silvicultural interventions. Decisions that maximize land or forest value depend only on the stand-price state, independently of how it was reached. Furthermore, to each stand-price state corresponds one single best decision. The solution of the MDP gives simultaneously the best decision for each state, and the forest value (land plus trees), given the stand state and following the best policy. Numerical solutions use either successive approximation, or linear programming. Examples with deterministic and stochastic cases show in particular the convergence of the MDP model to Faustmann's formula when the future is assumed known with certainty. In this deterministic environment, Faustmann's rule is independent of the distribution of stands in the forest. FOR. SCI. 47(4):466–474.

Women with BRCA1/2 mutations have higher risk for breast and ovarian cancers. Available intervention actions include prophylactic surgeries and breast screening, which vary significantly in cost, cancer prevention, and in resulting death from other causes. We present a model designed to yield optimal intervention strategies for mutation carriers between the ages of 30 and 65 and any prior intervention history. A Markov decision process (MDP) model is developed that considers yearly state transitions for the mutation carriers and state dependent intervention actions. State is defined as a vector comprising mutation type, health states, prior intervention actions, and age. A discounted value iteration algorithm is used to obtain optimal strategies from the MDP model using both cost and quality-adjusted life years (QALYs) as rewards. The results from MDP model show that for 30-year-old women with BRCA1 mutation and no prior intervention history, the cost-optimal strategy is a combination of prophylactic mastectomy (PM) and prophylactic oophorectomy (PO) at age 30 with no screening afterwards. Whereas, the QALYs-optimal strategy suggests PO at age 30 and PM at age 50 with screening afterwards. For BRCA2 mutation carriers at age 30, the cost-optimal strategy is PO at age 30, PM at age 40, and yearly screening only after age 56. Corresponding QALYs-optimal strategy is PM at age 40 with screening. Strategies for all other ages (31 to 65) are obtained and presented. It is also demonstrated that the cost-optimal strategies offer near maximum survival rate and near minimum cancer incidence rates by age 70, when compared to other ad hoc strategies.

The past decade has witnessed the shift from calling taxis within sight to using applications to send location information which enables e-hailing platforms to match passengers to drivers out of sight. After receiving orders, e-hailing platforms search for a suitable driver to accept the order to complete the matching. At present, platforms mainly set a fixed matching distance according to passengers’ locations for finding available drivers in this distance, but they do not consider if there are enough drivers available for their passengers. In order to find the properest driver which enables the decrease of passengers’ waiting time (i.e., from the time the passenger places the order to the time the driver accepts the order), we propose a model to dynamically change matching distance. The purpose is to find the best matching distance in different supply (i.e., the number of drivers) and demand (i.e., the number of passengers) circumstances. We obtain the required probability information by extracting data from multiple datasets in different supply and demand regions. Later, a Markov Decision Process (MDP) model is established to solve the optimal dynamically changing matching distance policy. At the same time, we propose a reward function to measure the waiting time of passengers and the distance between the driver and the passenger boarding point. Finally, we compared MDP model with three benchmark models, the model of fixed matching distance, dynamic programming matching distance, and linear matching distance. Through comparison, it is concluded that the policy of dynamically changing matching distance based on the MDP model we proposed is best applied. The policy we proposed balance the waiting time of passengers and the distance between the driver and the passenger boarding point, improve platform efficiency.KeywordsMarkov decision processDynamic matching distanceOrder dispatchUrban computingE-hailing

As a typical scenario in future generation communication network applications, UAV-assisted communication can perform autonomous data delivery for massive machine type communication (mMTC), where the data generated from Internet of Things (IoT) devices can be carried and delivered to the corresponding locations with no direct communication channels to the IoT devices. Wireless energy transfer technique can recharge the UAV when the system is in operation, assisting the UAV to continuously collect and deliver data. In this work, we formulate a Markov decision process (MDP) model to describe the energy and data transfer optimization problem for the UAV. To maximize the long-term utility of the UAV, the MDP model is solved by value iteration algorithm to obtain the optimal strategies of the UAV to collect data, deliver data, and receive transferred energy to replenish on-device battery energy storage. Furthermore, to tackle the issues of system state uncertainties, partially observable states, and large state space in UAV-assisted communication systems, we extend the MDP model and solve it by using a Q -learning and a deep reinforcement learning (DRL) schemes. Simulations and numerical results validate that, compared with baseline schemes, the proposed MDP model with DRL based scheme can achieve better wireless energy and data transfer strategies in terms of the higher long-term utility of the UAV.

Invisible stock loss often occurs in inventory management systems, which is usually caused by quality deterioration, theft, and misplacement and makes inventory record inaccurate. Inaccurate inventory record will bring significant negative impact on the performance of inventory control and leads to severe stock-out. To eliminate this negative impact, in this paper, we develop a Markov decision process (MDP) model for an inventory control system with invisible stock loss, inaccurate record, and replenishment lead-time. Based on the MDP model, a robust inventory control policy is constructed by using the policy iteration technique. From numerical experiments, we find that the robust policy still has a reorder point/order-up-to level structure i.e., an (s, S) structure, and can significantly reduce the impact of inventory inaccuracy.

Dynamic selling of quality-graded products under demand uncertainties

Linear programming formulation for non-stationary, finite-horizon Markov decision process models

State-of-health monitoring of lithium-ion batteries in electric vehicles by on-board internal resistance estimation

<div class="section abstract"><div class="htmlview paragraph">In this study, the duty-cycle of a commercial lithium-ion battery (LIB) for a typical passenger airplane (Bombardier CRJ200) is obtained and compared to the duty- cycle of the same LIB for electric vehicles. For this purpose, the velocity and altitude of the airplane is monitored during a typical flight and the instantaneous mechanical power of the airplane is obtained by modeling. Based on the airplane required power and the characteristics of the LIB, a battery pack is designed for the airplane. Then, the duty-cycle of a LIB cell in the battery pack is yielded. The duty-cycle of the same LIB for a typical electric vehicle is also obtained from modeling based on the Highway Fuel Economy Test (HWFET), New York City Cycle (NYCC) and United States 2006 (US06) drive-cycles. Finally, the duty-cycle of the LIB for two different applications of electric airplane and electric vehicle is compared. The duty-cycles obtained in this study can be employed to study the lifespan of LIBs.</div></div>

The stochastic nature of wireless communication suggests a Markov Decision Process (MDP) as a formalism for identifying and evaluating spectrum control policies. However, in practice numerous factors influence the success or failure of a transmission, so that the applicability of particular MDP models to real spectrum management problems must itself be examined. This paper presents a series of model validation studies in which correspondence between an MDP model and a discrete-event simulation (DES) model is evaluated. We test several hypotheses that together provide a foundation and an exemplar for the idea of using MDPs to guide management of shared spectrum. We conclude that there is sufficient similarity between the performance predictions made by the MDP model and the DES model that MDPs can be used effectively to determine spectrum control policies.

Recently, there has been growing recognition of the significance of energy and environmental challenges. Utilization of lithium-ion batteries in electric vehicles has shown considerable potential and benefits for tackling these issues. The effective management of battery temperature has become a crucial factor in the advancement and widespread adoption of lithium-ion batteries in electric vehicles. In this study, a thermo-coupled pseudo-two-dimensional (P2D) electrochemical model is employed to simulate the heat generation of the NCM811-21700 cylindrical battery cell at various discharge rates at an ambient temperature of 25 °C, and is validated by experimental data. The validation results demonstrate that the thermo-coupled P2D model can effectively predict the battery voltage curve during the discharge process with less than 4% errors. Although there is a slightly larger error in the temperature prediction during the battery 2C and 3C discharge processes, the maximum error approaches 10%, which is still generally within an acceptable range. In addition, the battery’s electrochemical and thermal characteristics during discharge are presented. The suggested thermo-coupled electrochemical model can be used for applications in the thermal management system of the NCM811-21700 battery.

For safe and reliable operation of lithium-ion batteries in electric vehicles, the real-time monitoring of their internal states is important. The purpose of our study is to find an easily implementable, online identification method for lithium-ion batteries in electric vehicles. In this article, we propose an equivalent circuit model structure. Based on the model structure we derive the recursive mathematical description. The recursive extended least square algorithm is introduced to estimate the model parameters online. The accuracy and robustness are validated through experiments and simulations. Real-road driving cycle experiment shows that the proposed online identification method can achieve acceptable accuracy with the maximum error of less than 5.52%. In addition, it is proved that the proposed method can also be used to estimate the real-time SOH and SOC of the batteries.

In the coronavirus epidemic, many Chinese hospitals have established buffer zones to prevent the spread and transmission of the virus. The buffer zone is a monitored and separate area where the patients who need hospitalizations after the quick treatments in the emergency department can temporarily wait for the Covid-19 test and receive some healthcare services to stabilize their conditions. Because the beds in the buffer zones are limited, the managers face the patient admission control problem for the buffer zone. This management and control problem is challenging since the patient arrivals are uncertain, and the patients’ conditions are different. In this paper, we build the infinite- and finite-horizon Markov decision process (MDP) models for this problem. We use the uniformization method to discretize the patient flow. We propose various iteration algorithms to solve the MDP models and obtain the optimal and threshold policies. Numerical experiments validate the advantages of the policies obtained by the algorithms in this paper over the current policies of hospitals. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —The ongoing COVID-19 pandemic has been causing enormous damage to people’s health, jobs, and well-being. COVID-19 has affected almost all countries globally and has changed the operation mode of the healthcare system, especially the hospitals. The hospitals are the frontlines of healthcare service and the battle with the COVID-19 pandemic. This article is motivated by our collaborations with hospitals in Shanghai, China. In China, many hospitals establish buffer zones: a monitored area where the patients who need hospitalizations after the quick treatments in the emergency department can temporarily wait for the Covid-19 test and receive some healthcare services to stabilize their conditions. Because the zone’s capacity is limited, the managers must make dynamic patient admission control decisions according to multiple factors, such as patients’ health conditions and the usage of beds in the zone. We propose two MDP models to solve this complex problem. Several iteration algorithms are designed to solve the MDP models and obtain the optimal and threshold policies. Based on hospitals’ real-life data, we show the methods presented in this paper can help hospital managers make more reasonable decisions. Although we focus on the hospital’s buffer zone in China, the methodology and approach for this problem can be extended to other practical hospital management scenarios in the coronavirus pandemic. For example, For example, some hospitals have admission control problems for coronavirus patients due to hospital capacity limitations. The hospital has to decide if a patient is accepted as an inpatient or suggested to home quarantine. In such a case, the admission control problem can also be solved by the methodologies in the paper.

Vehicle Edge Computing (VEC) leverages compact cloud computing at the mobile network edge to meet the processing and latency needs of vehicles. By bringing computation closer to the vehicles, VEC reduces data transmission, minimizes latency, and boosts performance for compute-intensive applications. However, during peak hours of urban road traffic, the scarce computational resources available at edge servers could pose challenges in fulfilling the processing needs of vehicles. Introducing Unmanned Aerial Vehicles (UAVs) as supplementary edge computing nodes could significantly mitigate the aforementioned issue. In this paper, we propose a flexible edge computing framework in which a fleet of UAVs function as mobile computational service providers, offering computation offloading services to multiple vehicles. We design and optimize a computation offloading model for the UAV-enabled vehicle edge computing environment. The proposed model tackles the task offloading challenge, aiming to optimize UAV revenue and task processing efficiency while considering the constraints of UAVs’ restricted computational power and energy resources. Towards this end, our model jointly considers two key factors: task partitioning and computational resource allocation. To tackle the challenges posed by the aforementioned non-convex optimization problem, we construct a Markov Decision Process (MDP) model for the multi-UAV-enabled mobile edge computing system and introduce an innovative Multi-Agent Deep Reinforcement Learning (MADRL) framework addressing the decision-making challenge represented by MDP model. Comprehensive simulation outcomes illustrate that our devised task offloading technique outperforms other optimization methods.