Online learning adaptation strategy for DASH clients

In this study, the authors focus on segment characteristic analysis and joint optimisation of modulation and coding scheme (MCS) selection, resource block (RB) assignment and the block error ratio (BLER) determination and segment adaptation scheme to satisfy dynamic adaptive streaming over HTTP (DASH) clients over wireless networks. First, the authors define a utility function as the continuous playback time that the packets scheduled with the allocated resource can support. Then, the authors formulate the MCS selection, RB assignment and BLER determination into a mathematical model. The relationship among the above three factors can be explored instead of performing MCS selection and RB assignment with the fixed BLER. By decomposing the original problem into some sub‐problems, the authors can get a solution to the original problem with low complexity. At the client level, the authors develop an adaptive segment request strategy based on the playback information, the segments' characteristics and the estimated transmission rate. To decrease the influence of the inaccurate estimations, an adaptive guard time interval based on the real playback information and transmission information of the previous segments is introduced. Simulation results show that the proposed algorithm can efficiently improve playback continuity and video quality over the existing algorithms.

Advanced planning for autonomous vehicles using reinforcement learning and deep inverse reinforcement learning

Reinforcement Learning with Temporal Logic Constraints

Towards enhanced threat modelling and analysis using a Markov Decision Process

Faustmann's formula gives the land value, or the forest value of land with trees, under deterministic assumptions regarding future stand growth and prices, over an infinite horizon. Markov decision process (MDP) models generalize Faustmann's approach by recognizing that future stand states and prices are known only as probabilistic distributions. The objective function is then the expected discounted value of returns, over an infinite horizon. It gives the land or the forest value in a stochastic environment. In MDP models, the laws of motion between stand-price states are Markov chains. Faustmann's formula is a special case where the probability of movement from one state to another is equal to unity. MDP models apply whether the stand state is bare land, or any state with trees, be it even- or uneven-aged. Decisions change the transition probabilities between stand states through silvicultural interventions. Decisions that maximize land or forest value depend only on the stand-price state, independently of how it was reached. Furthermore, to each stand-price state corresponds one single best decision. The solution of the MDP gives simultaneously the best decision for each state, and the forest value (land plus trees), given the stand state and following the best policy. Numerical solutions use either successive approximation, or linear programming. Examples with deterministic and stochastic cases show in particular the convergence of the MDP model to Faustmann's formula when the future is assumed known with certainty. In this deterministic environment, Faustmann's rule is independent of the distribution of stands in the forest. FOR. SCI. 47(4):466–474.

As a typical scenario in future generation communication network applications, UAV-assisted communication can perform autonomous data delivery for massive machine type communication (mMTC), where the data generated from Internet of Things (IoT) devices can be carried and delivered to the corresponding locations with no direct communication channels to the IoT devices. Wireless energy transfer technique can recharge the UAV when the system is in operation, assisting the UAV to continuously collect and deliver data. In this work, we formulate a Markov decision process (MDP) model to describe the energy and data transfer optimization problem for the UAV. To maximize the long-term utility of the UAV, the MDP model is solved by value iteration algorithm to obtain the optimal strategies of the UAV to collect data, deliver data, and receive transferred energy to replenish on-device battery energy storage. Furthermore, to tackle the issues of system state uncertainties, partially observable states, and large state space in UAV-assisted communication systems, we extend the MDP model and solve it by using a Q -learning and a deep reinforcement learning (DRL) schemes. Simulations and numerical results validate that, compared with baseline schemes, the proposed MDP model with DRL based scheme can achieve better wireless energy and data transfer strategies in terms of the higher long-term utility of the UAV.

Since video traffic has become the major source of generated mobile traffic worldwide, there has been a strong interest in new techniques to improve the Quality of Experience (QoE) of users generating this traffic. Several efforts have been made to propose adaptive streaming algorithms on the client side to be able to adjust the quality of the downloaded video segments to the current network conditions, taking advantage of the Dynamic Adaptive Streaming over HTTP (DASH) standard. This study uses this knowledge to propose a pre-fetching proxy cache, to be placed at the network's edge, which will predict the quality that the client will request for the following segment. The proxy predicts the future network conditions and models the system as a Markov Decision Process (MDP), in order to find the optimal decision for the proxy, given the current network conditions. This Just-in-Time caching technique pre-fetches the segment just before the client requests it, aiming to decrease the total time spent by the client downloading the segments, and indirectly increasing the user's QoE, as the DASH client will perceive better network conditions. The study concludes that the predictive technique improves QoE by increasing the video quality and decreasing the number of stalling events, in comparison to solutions that pre-fetch the previously requested quality, and solutions which do not use any pre-fetching technique.

The past decade has witnessed the shift from calling taxis within sight to using applications to send location information which enables e-hailing platforms to match passengers to drivers out of sight. After receiving orders, e-hailing platforms search for a suitable driver to accept the order to complete the matching. At present, platforms mainly set a fixed matching distance according to passengers’ locations for finding available drivers in this distance, but they do not consider if there are enough drivers available for their passengers. In order to find the properest driver which enables the decrease of passengers’ waiting time (i.e., from the time the passenger places the order to the time the driver accepts the order), we propose a model to dynamically change matching distance. The purpose is to find the best matching distance in different supply (i.e., the number of drivers) and demand (i.e., the number of passengers) circumstances. We obtain the required probability information by extracting data from multiple datasets in different supply and demand regions. Later, a Markov Decision Process (MDP) model is established to solve the optimal dynamically changing matching distance policy. At the same time, we propose a reward function to measure the waiting time of passengers and the distance between the driver and the passenger boarding point. Finally, we compared MDP model with three benchmark models, the model of fixed matching distance, dynamic programming matching distance, and linear matching distance. Through comparison, it is concluded that the policy of dynamically changing matching distance based on the MDP model we proposed is best applied. The policy we proposed balance the waiting time of passengers and the distance between the driver and the passenger boarding point, improve platform efficiency.KeywordsMarkov decision processDynamic matching distanceOrder dispatchUrban computingE-hailing

Women with BRCA1/2 mutations have higher risk for breast and ovarian cancers. Available intervention actions include prophylactic surgeries and breast screening, which vary significantly in cost, cancer prevention, and in resulting death from other causes. We present a model designed to yield optimal intervention strategies for mutation carriers between the ages of 30 and 65 and any prior intervention history. A Markov decision process (MDP) model is developed that considers yearly state transitions for the mutation carriers and state dependent intervention actions. State is defined as a vector comprising mutation type, health states, prior intervention actions, and age. A discounted value iteration algorithm is used to obtain optimal strategies from the MDP model using both cost and quality-adjusted life years (QALYs) as rewards. The results from MDP model show that for 30-year-old women with BRCA1 mutation and no prior intervention history, the cost-optimal strategy is a combination of prophylactic mastectomy (PM) and prophylactic oophorectomy (PO) at age 30 with no screening afterwards. Whereas, the QALYs-optimal strategy suggests PO at age 30 and PM at age 50 with screening afterwards. For BRCA2 mutation carriers at age 30, the cost-optimal strategy is PO at age 30, PM at age 40, and yearly screening only after age 56. Corresponding QALYs-optimal strategy is PM at age 40 with screening. Strategies for all other ages (31 to 65) are obtained and presented. It is also demonstrated that the cost-optimal strategies offer near maximum survival rate and near minimum cancer incidence rates by age 70, when compared to other ad hoc strategies.

Invisible stock loss often occurs in inventory management systems, which is usually caused by quality deterioration, theft, and misplacement and makes inventory record inaccurate. Inaccurate inventory record will bring significant negative impact on the performance of inventory control and leads to severe stock-out. To eliminate this negative impact, in this paper, we develop a Markov decision process (MDP) model for an inventory control system with invisible stock loss, inaccurate record, and replenishment lead-time. Based on the MDP model, a robust inventory control policy is constructed by using the policy iteration technique. From numerical experiments, we find that the robust policy still has a reorder point/order-up-to level structure i.e., an (s, S) structure, and can significantly reduce the impact of inventory inaccuracy.

Dynamic selling of quality-graded products under demand uncertainties

Linear programming formulation for non-stationary, finite-horizon Markov decision process models

Vehicle Edge Computing (VEC) leverages compact cloud computing at the mobile network edge to meet the processing and latency needs of vehicles. By bringing computation closer to the vehicles, VEC reduces data transmission, minimizes latency, and boosts performance for compute-intensive applications. However, during peak hours of urban road traffic, the scarce computational resources available at edge servers could pose challenges in fulfilling the processing needs of vehicles. Introducing Unmanned Aerial Vehicles (UAVs) as supplementary edge computing nodes could significantly mitigate the aforementioned issue. In this paper, we propose a flexible edge computing framework in which a fleet of UAVs function as mobile computational service providers, offering computation offloading services to multiple vehicles. We design and optimize a computation offloading model for the UAV-enabled vehicle edge computing environment. The proposed model tackles the task offloading challenge, aiming to optimize UAV revenue and task processing efficiency while considering the constraints of UAVs’ restricted computational power and energy resources. Towards this end, our model jointly considers two key factors: task partitioning and computational resource allocation. To tackle the challenges posed by the aforementioned non-convex optimization problem, we construct a Markov Decision Process (MDP) model for the multi-UAV-enabled mobile edge computing system and introduce an innovative Multi-Agent Deep Reinforcement Learning (MADRL) framework addressing the decision-making challenge represented by MDP model. Comprehensive simulation outcomes illustrate that our devised task offloading technique outperforms other optimization methods.

With the growing demand for multimedia services, Dynamic Adaptive Streaming over HTTP (DASH) has become a key solution for delivering high-quality video content. In this work, we consider an Edge-DASH scenario and formulate a joint optimization problem that involves four critical aspects: bitrate allocation, user-to-server assignment, caching, and bandwidth allocation. Due to the complexity of the joint problem, we decompose it into sub-problems and address them separately. To solve the resulting sub-problems, we employ deep reinforcement learning, specifically the Deep Deterministic Policy Gradient (DDPG) method, for three of them, and develop a heuristic solution for the fourth. Simulation results demonstrate that our approach enhances performance across multiple metrics, including improved video delivery, reduced buffer underflow and overflow, and more efficient caching, which collectively enable greater utilization of edge resources for streaming. Moreover, we evaluated inference latency across edge and cloud hardware, confirming sub- to few-millisecond performance suitable for real-time deployment. This showcases the benefits of combining learning-based and heuristic techniques to meet the growing demand for adaptive video streaming in edge computing environments.

In the coronavirus epidemic, many Chinese hospitals have established buffer zones to prevent the spread and transmission of the virus. The buffer zone is a monitored and separate area where the patients who need hospitalizations after the quick treatments in the emergency department can temporarily wait for the Covid-19 test and receive some healthcare services to stabilize their conditions. Because the beds in the buffer zones are limited, the managers face the patient admission control problem for the buffer zone. This management and control problem is challenging since the patient arrivals are uncertain, and the patients’ conditions are different. In this paper, we build the infinite- and finite-horizon Markov decision process (MDP) models for this problem. We use the uniformization method to discretize the patient flow. We propose various iteration algorithms to solve the MDP models and obtain the optimal and threshold policies. Numerical experiments validate the advantages of the policies obtained by the algorithms in this paper over the current policies of hospitals. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —The ongoing COVID-19 pandemic has been causing enormous damage to people’s health, jobs, and well-being. COVID-19 has affected almost all countries globally and has changed the operation mode of the healthcare system, especially the hospitals. The hospitals are the frontlines of healthcare service and the battle with the COVID-19 pandemic. This article is motivated by our collaborations with hospitals in Shanghai, China. In China, many hospitals establish buffer zones: a monitored area where the patients who need hospitalizations after the quick treatments in the emergency department can temporarily wait for the Covid-19 test and receive some healthcare services to stabilize their conditions. Because the zone’s capacity is limited, the managers must make dynamic patient admission control decisions according to multiple factors, such as patients’ health conditions and the usage of beds in the zone. We propose two MDP models to solve this complex problem. Several iteration algorithms are designed to solve the MDP models and obtain the optimal and threshold policies. Based on hospitals’ real-life data, we show the methods presented in this paper can help hospital managers make more reasonable decisions. Although we focus on the hospital’s buffer zone in China, the methodology and approach for this problem can be extended to other practical hospital management scenarios in the coronavirus pandemic. For example, For example, some hospitals have admission control problems for coronavirus patients due to hospital capacity limitations. The hospital has to decide if a patient is accepted as an inpatient or suggested to home quarantine. In such a case, the admission control problem can also be solved by the methodologies in the paper.