Cost Minimization for Admission Control in Bandwidth Asymmetry Wireless Networks

This paper investigates the optimal call admission control (CAC) policy for non-priority scheme (NPS) and reserved channel scheme (RCS) in wireless networks, respectively. Both new call and handoff call arrival processes are assumed to be Poisson processes, and the call holding times are exponentially distributed with different rate for new call and handoff calls. Admitting each call would bring a reward to the network provider but holding each call in the system would also incur some expenses (or cost) to the provider. We concentrate on the optimization problems of when to admit or reject a call in order to achieve the maximum total expected discounted reward. By establishing a discounted semi Markov decision process (SMDP) model, we verify that the optimal policies are state-related control limit policies for both NPS and RCS. Our numerical results explained in both tables and diagrams are consistent with our theoretic results.

In the coronavirus epidemic, many Chinese hospitals have established buffer zones to prevent the spread and transmission of the virus. The buffer zone is a monitored and separate area where the patients who need hospitalizations after the quick treatments in the emergency department can temporarily wait for the Covid-19 test and receive some healthcare services to stabilize their conditions. Because the beds in the buffer zones are limited, the managers face the patient admission control problem for the buffer zone. This management and control problem is challenging since the patient arrivals are uncertain, and the patients’ conditions are different. In this paper, we build the infinite- and finite-horizon Markov decision process (MDP) models for this problem. We use the uniformization method to discretize the patient flow. We propose various iteration algorithms to solve the MDP models and obtain the optimal and threshold policies. Numerical experiments validate the advantages of the policies obtained by the algorithms in this paper over the current policies of hospitals. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —The ongoing COVID-19 pandemic has been causing enormous damage to people’s health, jobs, and well-being. COVID-19 has affected almost all countries globally and has changed the operation mode of the healthcare system, especially the hospitals. The hospitals are the frontlines of healthcare service and the battle with the COVID-19 pandemic. This article is motivated by our collaborations with hospitals in Shanghai, China. In China, many hospitals establish buffer zones: a monitored area where the patients who need hospitalizations after the quick treatments in the emergency department can temporarily wait for the Covid-19 test and receive some healthcare services to stabilize their conditions. Because the zone’s capacity is limited, the managers must make dynamic patient admission control decisions according to multiple factors, such as patients’ health conditions and the usage of beds in the zone. We propose two MDP models to solve this complex problem. Several iteration algorithms are designed to solve the MDP models and obtain the optimal and threshold policies. Based on hospitals’ real-life data, we show the methods presented in this paper can help hospital managers make more reasonable decisions. Although we focus on the hospital’s buffer zone in China, the methodology and approach for this problem can be extended to other practical hospital management scenarios in the coronavirus pandemic. For example, For example, some hospitals have admission control problems for coronavirus patients due to hospital capacity limitations. The hospital has to decide if a patient is accepted as an inpatient or suggested to home quarantine. In such a case, the admission control problem can also be solved by the methodologies in the paper.

Dynamic selling of quality-graded products under demand uncertainties

Minimizing Risk Models in Markov Decision Processes with Policies Depending on Target Values

Next generation wireless networks will support a diverse set of access technologies. And the demanding breed of multimedia applications will even more considerably require Quality of Service (QoS) support. This paper proposes an optimal prority-based Call Admission Control (CAC) scheme in heterogeneous wireless networks, i.e., the generalized semi-Markov Decision Process based CAC (GCAC). Considering the limitations of the traditional Markov decision process for optimal CAC policy, we characterize the CAC problem through a GSMDP formulation that characterize s the network behaviors in the heterogeneous system more exactly. And the proposed scheme can guarantee the QoS constrains both in the physical and network layer. Numerical examples illustrate that the performance in the GCAC scheme is significantly better than that in the conventioinal GC scheme and an optimal CAC schme based on MDP.

In this paper, we consider a class of continuous-time, continuous-space stochastic optimal control problems. Using the Markov chain approximation method and recent advances in sampling-based algorithms for deterministic path planning, we propose a novel algorithm called the incremental Markov Decision Process to incrementally compute control policies that approximate arbitrarily well an optimal policy in terms of the expected cost. The main idea behind the algorithm is to generate a sequence of finite discretizations of the original problem through random sampling of the state space. At each iteration, the discretized problem is a Markov Decision Process that serves as an incrementally refined model of the original problem. We show that with probability one, (i) the sequence of the optimal value functions for each of the discretized problems converges uniformly to the optimal value function of the original stochastic optimal control problem, and (ii) the original optimal value function can be computed efficiently in an incremental manner using asynchronous value iterations. Thus, the proposed algorithm provides an anytime approach to the computation of optimal control policies of the continuous problem. The effectiveness of the proposed approach is demonstrated on motion planning and control problems in cluttered environments in the presence of process noise.

As a typical scenario in future generation communication network applications, UAV-assisted communication can perform autonomous data delivery for massive machine type communication (mMTC), where the data generated from Internet of Things (IoT) devices can be carried and delivered to the corresponding locations with no direct communication channels to the IoT devices. Wireless energy transfer technique can recharge the UAV when the system is in operation, assisting the UAV to continuously collect and deliver data. In this work, we formulate a Markov decision process (MDP) model to describe the energy and data transfer optimization problem for the UAV. To maximize the long-term utility of the UAV, the MDP model is solved by value iteration algorithm to obtain the optimal strategies of the UAV to collect data, deliver data, and receive transferred energy to replenish on-device battery energy storage. Furthermore, to tackle the issues of system state uncertainties, partially observable states, and large state space in UAV-assisted communication systems, we extend the MDP model and solve it by using a Q -learning and a deep reinforcement learning (DRL) schemes. Simulations and numerical results validate that, compared with baseline schemes, the proposed MDP model with DRL based scheme can achieve better wireless energy and data transfer strategies in terms of the higher long-term utility of the UAV.

This paper presents a hierarchical approach for controlling the spread of an epidemic disease. The approach consists of a three-layer architecture where a set of two-layer multiple social networks is governed by a (third) top-layer consisting of an optimal control policy. Each of the two-layer social networks is modeled by a microscopic Markov chain. On top of all the two-layer networks is an optimal control policy that has been developed by using an underlying Markov Decision Process (MDP) model. Mathematical models pertaining to the top-level MDP as well as two-layer microscopic Markov chains have been presented. Practical implementation methodology using the proposed models has also been discussed along with a numerical example. The results in the numerical example illustrate the control of an epidemic using the optimal policy. Directions for further research and characterization of the optimal policy have also been discussed with the help of the same numerical example.Article HighlightsAn optimal approach for controlling the spread of an epidemic infection.The approach is able to model the uncertainties involved in the problem.The approach is able to cater for the underlying social network.

BackgroundWe propose a mathematical model formulated as a finite-horizon Markov Decision Process (MDP) to allocate capacity in a radiology department that serves different types of patients. To the best of our knowledge, this is the first attempt at considering radiology resources with different capacities and individual no-show probabilities of ambulatory patients in an MDP model. To mitigate the negative impacts of no-show, overbooking rules are also investigated.MethodsThe model’s main objective is to identify an optimal policy for allocating the available capacity such that waiting, overtime, and penalty costs are minimized. Optimization is carried out using traditional dynamic programming (DP). The model was applied to real data from a radiology department of a large Brazilian public hospital. The optimal policy is compared with five alternative policies, one of which resembles the one currently used by the department. We identify among alternative policies the one that performs closest to the optimal.ResultsThe optimal policy presented the best performance (smallest total daily cost) in the majority of analyzed scenarios (212 out of 216). Numerical analyses allowed us to recommend the use of the optimal policy for capacity allocation with a double overbooking rule and two resources available in overtime periods. An alternative policy in which outpatients are prioritized for service (rather than inpatients) displayed results closest to the optimal policy, being also recommended due to its easy implementation.ConclusionsBased on such recommendation and observing the state of the system at any given period (representing the number of patients waiting for service), radiology department managers should be able to make a decision (i.e., define number and type of patients) that should be selected for service such that the system’s cost is minimized.

This article addresses the admission control of a loss queueing network, or shortly loss network, of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$N$ </tex-math></inline-formula> parallel multiserver stations with no waiting rooms. <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$M$ </tex-math></inline-formula> classes of customers arrive at a Poisson rate and require an exponential service time with common mean. A customer can be served by any free server of a station in the set of stations determined by the class of the customer. Admission of a customer to a station brings different rewards that depend on customer class and the preference order of that station. Our objective is to find the optimal admission control policy that maximizes the average reward. We adopt a research strategy that evolves from simpler networks to more complicated ones. First, we consider a one-station <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$M$ </tex-math></inline-formula> -class loss network whose results form a fundamental basis for the analysis of more complicated networks. Second, we consider a two-station <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$M$ </tex-math></inline-formula> -class loss network for which we establish the existence of an optimal threshold admission policy. Finally, we consider a general <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$N$ </tex-math></inline-formula> -station <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$M$ </tex-math></inline-formula> -class loss network and propose an iterative approximation policy (IAP) and a mixed-integer linear programming model that is proven to compute an upper bound. We demonstrate the near-optimal performance of the proposed IAP and the tightness of the upper bound on several numerical instances motivated by real-life networks, such as healthcare emergency service networks and emergency call centers. Note to Practitioners—This article is mainly motivated by real-time scheduling of emergency service networks, such as emergency healthcare and telecommunication networks. In such networks, emergent demands need to be served promptly and are lost otherwise. They can be served by multiple hospitals/operator groups but have their own preferences. The decision maker has to decide dynamically whether to keep the capacity of a hospital/operator for its own customers to meet customer preference or serve the incoming demand to improve the resource utilization. We set the problem as the admission control of a multiclass multistation loss network, study the properties of the optimal policy, and propose an iterative approximation policy that is proven to be near-optimal for large-size loss networks. The proposed policy is found to improve by up to 16% and 52% the no-control policy admitting the customer to the most preferred available station and the no-overflow policy with admission to only the most preferred station, respectively. Managerial insights and extensions are also discussed.

This paper studies the optimal policy for joint control of admission, routing, service, and jockeying in a queueing system consisting of two exponential servers in parallel. Jobs arrive according to a Poisson process. Upon each arrival, an admission/routing decision is made, and the accepted job is routed to one of the two servers with each being associated with a queue. After each service completion, the servers have an option of serving a job from its own queue, serving a jockeying job from another queue, or staying idle. The system performance is inclusive of the revenues from accepted jobs, the costs of holding jobs in queues, the service costs and the job jockeying costs. To maximize the total expected discounted return, we formulate a Markov decision process (MDP) model for this system. The value iteration method is employed to characterize the optimal policy as a hedging point policy. Numerical studies verify the structure of the hedging point policy which is convenient for implementing control actions in practice.

This paper studies the admission control and resource allocation in a cell-based wireless system that supports single- media and multirate services. Utilizing the idea of adaptive and dynamic reallocation, we propose three control policies, AO, ADRF, and ADRA. The Markov decision process (MDP) theory is used to build the model of the system. We solve the MDP model and find the optimal policy by linear programming (LP). The proposed policies not only make the best use of precious wireless bandwidth, but also satisfy the requirement of quality of services such as handover blocking probability.

This paper considers a manufacturer with a two-station make-to-stock (MTS) and make-to-order (MTO) serial production system. The MTS facility processes a single type of raw material and provides these materials for the MTO facility. The facility in the second stage produces end items using the material from the order of firms. The materials processed by the MTS facility are either stocked for production in the second stage or consumed to satisfy exogenous demand from the material market. Revenues are generated in each stage by satisfying the material demands and the end item demands, respectively. The manufacturer can either accept or reject some of the exogenous material demands according to the status of the system and the rejected exogenous material demands incur no penalty. The orders of firms can be rejected with penalty. Holding costs are incurred for the materials and the end items in inventory at each stage. Demand processes and production times of the component and the end item are stochastic. The set of decision epochs in our model corresponds to the epochs of customer order and material demand arrivals, MTO production completion, and material process completion. At each decision epoch, the manufacturer determines whether or not to process materials. An the epochs corresponding to a customer order arrival or a material demand arrival, a decision must be made whether to accept or reject it. This paper studies a joint order admission, demand control, and material process rate control policy which maximizes the manufacturer’s profit. Using the Markov decision process model, we investigate the structural properties of the optimal policy using a numerical example. The numerical investigation indicates that the optimal policy can be defined by the monotone stitching curves. Specifically, the optimal policy will accept more customer orders as material inventory increases and accepts fewer material demands as the number of outstanding customer orders increases. Furthermore, it is more profitable to increase the material production rate as the number of outstanding customer orders increases. We also examine the impact of the material process rate control option on the manufacturer’s profit. The study confirms that the material processing policy with a controllable production rate has a great percentage improvement in the average profit over the one with a fixed production rate. The study also show that the performance improvement due to using the controllable production rate rather than the fixed one is more prominent for the cases with a lower MTO customer arrival rate, a lower unit MTO sales revenue, a lower MTO production rate, a lower unit material sales revenue or a lower material holding cost. It is also observed that the impact of the material demand rate on the performance difference between the policies with a controllable production rate and a fixed one can vary depending on the value of other parameters.

This paper extends finite state and action space Markov Decision Process (MDP) models by introducing a new type of measurement for the outcomes of actions. The new measurement allows to sequentially observe the next-state transition for taking an action, i.e., the actions are ordered and the next action outcome in the sequence is observed only if the current action is not chosen. The sequentially-observed MDP (SO-MDP) shares some properties with a standard MDP: among history dependent policies, Markovian ones are still optimal. SO-MDP policies have the advantage of producing better rewards than standard optimal MDP policies due to additional measurements. Computing these policies, on the other hand, is more complex and we present a linear programming based synthesis of the optimal decision policies for the finite horizon SO-MDPs. A simulation example of multiple autonomous agents is also provided to demonstrate the SO-MDP model and the proposed policy synthesis method.

Towards enhanced threat modelling and analysis using a Markov Decision Process