Optimal Power and Retransmission Control Policies for Random Access Systems

In this paper, we propose an optimal video streaming framework using retransmissions based on the theory of Markov decision processes (MDP). Using MDPs, we calculate average reward optimal policies so as to maximize the on time delivery probability of video frames in the transmission window while conforming to network-imposed available bandwidth constraints. The effectiveness of the proposed approach is validated by comparing the results with theoretically obtained rate-distortion curves corresponding to the optimal policies by using the distortion model in K. Stuhlmuller et al., (2000)

The purpose of this section is to introduce the notation that will be used in the subsequent parts and the most essential facts that we will need from the theory of Markov Decision Processes (MDPs) in the rest of the book. Readers familiar with MDPs should skim through this section to familiarize themselves with the notation. Readers unfamiliar with MDPs are suggested to spend enough time with this section to understand the details. Proofs of most of the results (with some simplifications) are included in Appendix A. The reader who is interested in learning more about MDPs is suggested to consult one of the many excellent books on the subject, such as the books of Bertsekas and Shreve (1978), Puterman (1994), or the two-volume book by Bertsekas (2007a,b).

In this paper, We consider an uncertain Markov decision process (MDP) with a control cost and a linear temporal logic (LTL) control specification. We propose a reinforcement learning (RL) based method for design of an optimal control policy by which the controlled MDP satisfies the control specification with probability 1 and minimizes an expected discounted sum of the control costs. First, we construct a deterministic Rabin automaton (DRA) that accepts all and only infinite words satisfying the LTL control specification. Second, we construct a product MDP of the MDP and the DRA to represent a dynamic control policy that satisfies the control specification. Third, we modify the product MDP in order to apply RL to the design of an optimal control policy. The control action of the modified product MDP is a pair of a pattern and an action, where the pattern is a set of actions. Moreover, we introduce a reward that represents both the satisfaction of the control specification and the minimally restrictiveness of the pattern. Finally, we proposed an algorithm for design of an optimal control policy that consists of a sequential decision making of two steps. At the first decision making, we select a pattern that maximizes a discounted sum of the reward. At the second one, we select an action from the pattern selected at the first one such that it minimizes the expected discounted sum of the costs. Moreover, we consider an illustrative example to show that the proposed algorithm can obtain an optimal control policy.

This paper studies optimal stochastic power control problem for a time-varying communication link, where the transmitter randomly harvests renewable energies from the environment. The harvested energies are stored in an energy buffer (or battery). Packets arrive at the transmitter data buffer with a constant rate μ. The objective is to maximize μ under the statistical delay and energy harvesting (EH) constraints. In order to study the optimal power control policy, we reformulate the problem as an infinite-horizon Markov decision process (MDP) using asymptotic delay analysis. The optimal policy and its structural properties are studied by employing the post-decision framework approach. We then propose an online power control algorithm, which converges to the optimal solution without requiring the statistical knowledge of the channel fading and EH processes. Numerical results demonstrate the effectiveness of the online algorithm for different delay constraints and EH settings.

This paper studies the admission control and resource allocation in a cell-based wireless system that supports single- media and multirate services. Utilizing the idea of adaptive and dynamic reallocation, we propose three control policies, AO, ADRF, and ADRA. The Markov decision process (MDP) theory is used to build the model of the system. We solve the MDP model and find the optimal policy by linear programming (LP). The proposed policies not only make the best use of precious wireless bandwidth, but also satisfy the requirement of quality of services such as handover blocking probability.

We consider a Markov decision process with a Borel state space, a countable action space, finite action sets, bounded rewards and a bounded transition density satisfying a simultaneous Doeblin condition. The existence of stationary strong 0-discount optimal polices is proved.

For the design of call admission control (CAC) policy in wireless networks, how to decrease the average system cost is a key issue. In this paper, we study the optimal admission policy for minimizing the system cost. By modeling the admission control problem into a Markov decision process (MDP) and analyzing the corresponding value function, we obtain some monotonicity properties of the optimal policy. These properties suggest that the optimal admission control policy for the bandwidth asymmetry wireless networks should have a threshold structure and the threshold specified for a class of calls may change with the system state. Due to the prohibitively high complexity for computing the thresholds in a system with large state space, we propose a heuristic CAC policy called call-rate- based dynamic threshold (CRDT) policy to approximate the theoretical optimal policy based on the insights we obtain from the modeling and the analytical study on the properties of the optimal policy. The CRDT policy is efficient and can be easily implemented. The numerical results show that the performance of average system cost of the proposed CRDT policy is close to that of the optimal policy from the MDP model and is better than that of some known existing CAC schemes, including those performing well in bandwidth asymmetry wireless networks.

In this paper, secure communication over a buffer-aided two-hop communication link, in which a source sends data to a destination with the aid of a trusted half-duplex relay node in the presence of a passive eavesdropper, is studied. It is assumed that there is no direct link between the source and the destination, and the buffer-aided relay forwards the information to the destination by employing the decode-and-forward (DF) scheme. Both the source and the relay transmissions are assumed to be vulnerable to the eavesdropper. The perfect channel side information (CSI) of the network is assumed to be available at the source and the relay. Initially, conventional relaying protocols where equal partition of the time for the reception and transmission of the relay is considered. Then, the optimal time fraction with the optimal power control policy that maximizes the secrecy throughput is identified. Subsequently, the optimal joint link selection and power control policy that maximizes the secrecy throughput is derived. Numerical results demonstrate that the joint link selection and power control scheme can significantly improve the achievable secrecy throughput of the buffer-aided two-hop communication system.

The average cost optimality equation for Markov control processes on Borel spaces

The optimal resource allocation scheme in a full-duplex Wireless Powered Communication Network (WPCN) composed of one Access Point (AP) and two wireless devices is analyzed and derived. AP operates in a full-duplex mode and is able to broadcast wireless energy signals in downlink and receive information data in uplink simultaneously. On the other hand, each wireless device is assumed to be equipped with Radio-Frequency (RF) energy harvesting circuitry which gathers the energy sent by AP and stores it in a finite capacity battery. The harvested energy is then used for performing uplink data transmission tasks. In the literature, the main focus so far has been on slot-oriented optimization. In this context, all the harvested RF energy in a given slot is also consumed in the same slot. However, this approach leads to sub-optimal solutions because it does not take into account the Channel State Information (CSI) variations over future slots. Differently from most of the prior works, in this paper we focus on the long-term weighted throughput maximization problem. This approach significantly increases the complexity of the optimization problem since it requires to consider both CSI variations over future slots and the evolution of the batteries when deciding the optimal resource allocation. We formulate the problem using the Markov Decision Process (MDP) theory and show how to solve it. Our numerical results emphasize the superiority of our proposed full-duplex WPCN compared to the half-duplex WPCN and reveal interesting insights about the effects of perfect as well as imperfect self-interference cancellation techniques on the network performance.

We consider the problem of <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">quickest</i> <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">detection</i> <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">of</i> <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">an</i> <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">intrusion</i> using a sensor network, keeping only a minimal number of sensors <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">active</i> . By using a minimal number of sensor devices, we ensure that the energy expenditure for <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">sensin</i> g, <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">computation</i> and <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">communication</i> is minimized (and the lifetime of the network is maximized). We model the intrusion detection (or change detection) problem as a <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Markov</i> <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">decision</i> <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">process</i> (MDP). Based on the theory of MDP, we develop the following closed loop <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">sleep/wake</i> scheduling algorithms: (1) optimal control of M <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k+1</sub> , the number of sensors in the wake state in time slot k + 1, (2) optimal control of q <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k+1</sub> , the probability of a sensor in the wake state in time slot k + 1, and an open loop <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">sleep/wake</i> scheduling algorithm which (3) computes q, the optimal probability of a sensor in the wake state (which does not vary with time), based on the sensor observations obtained until time slot k. Our results show that an optimum closed loop control on M <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k+1</sub> significantly decreases the cost compared to keeping any number of sensors active all the time. Also, among the three algorithms described, we observe that the total cost is minimum for the optimum control on M <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k+1</i> and is maximum for the optimum open loop control on q.

This paper presents an optimization model for selection of pavement maintenance intervention using a theory of Markov Decision Processes (MDP). There are some particular characteristics of the MDP developed in this paper which distinguish it from other similar studies or optimization models intended for pavement maintenance policy development. These unique characteristics include a direct inclusion of constraints into the formulation of MDP, the use of an average cost method of MDP, and the policy development process based on the dual linear programming solution. The limited information or discussions that are available on these matters in terms of stochastic based optimization model in road network management motivates this study. This paper uses a data set acquired from road authorities of state of Victoria, Australia, to test the model and recommends steps in the computation of MDP based stochastic optimization model, leading to the development of optimum pavement maintenance policy.

This volume deals with the theory of Markov Decision Processes (MDPs) and their applications. Each chapter was written by a leading expert in the respective area. The papers cover major research areas and methodologies, and discuss open questions and future research directions. The papers can be read independently, with the basic notation and concepts of Section 1.2. Most chap- ters should be accessible by graduate or advanced undergraduate students in fields of operations research, electrical engineering, and computer science.KeywordsMarkov Decision ProcessReward FunctionStochastic GameOptimality EquationAverage RewardThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Queueing models are frequently helpful in the analysis and control of communication, manufacturing, and transportation systems. The theory of Markov decision processes and the inductive techniques of dynamic programming have been used to develop normative models for optimal control of admission, servicing, routing, and scheduling of jobs in queues and networks of queues. We review some of these models, beginning with single-facility models and then progressing to models for networks of queues. We emphasize the use of induction on a sequence of successive approximations of the optimal value function (value iteration) to establish the form of optimal control policies.

Planning under time constraints in stochastic domains