Optimal Sleep-Wake Scheduling for Quickest Intrusion Detection Using Wireless Sensor Networks

The purpose of this section is to introduce the notation that will be used in the subsequent parts and the most essential facts that we will need from the theory of Markov Decision Processes (MDPs) in the rest of the book. Readers familiar with MDPs should skim through this section to familiarize themselves with the notation. Readers unfamiliar with MDPs are suggested to spend enough time with this section to understand the details. Proofs of most of the results (with some simplifications) are included in Appendix A. The reader who is interested in learning more about MDPs is suggested to consult one of the many excellent books on the subject, such as the books of Bertsekas and Shreve (1978), Puterman (1994), or the two-volume book by Bertsekas (2007a,b).

In this paper, we propose an optimal video streaming framework using retransmissions based on the theory of Markov decision processes (MDP). Using MDPs, we calculate average reward optimal policies so as to maximize the on time delivery probability of video frames in the transmission window while conforming to network-imposed available bandwidth constraints. The effectiveness of the proposed approach is validated by comparing the results with theoretically obtained rate-distortion curves corresponding to the optimal policies by using the distortion model in K. Stuhlmuller et al., (2000)

We consider in this study dynamic control policies for slotted Aloha random access systems. New performance bounds are derived when random access is combined with power control for system optimization, and we establish the existence of optimal control approaches for such systems. We analyze throughput and delay when the number of backlogged users is known, where we can explicitly obtain optimal policies and analyze their corresponding performance using Markov Decision Process (MDP) theory with average cost criterion. For the realistic unknown-backlog case, we establish the existence of optimal backlog-minimizing policies for the same range of arrival rates as the ideal known-backlog case by using the theory of MDPs with Borel state space and unbounded costs. We also propose suboptimal control policies with performance close to the optimal without sacrificing stability. These policies perform substantially better than existing Certainty Equivalence controllers.

This paper presents an optimization model for selection of pavement maintenance intervention using a theory of Markov Decision Processes (MDP). There are some particular characteristics of the MDP developed in this paper which distinguish it from other similar studies or optimization models intended for pavement maintenance policy development. These unique characteristics include a direct inclusion of constraints into the formulation of MDP, the use of an average cost method of MDP, and the policy development process based on the dual linear programming solution. The limited information or discussions that are available on these matters in terms of stochastic based optimization model in road network management motivates this study. This paper uses a data set acquired from road authorities of state of Victoria, Australia, to test the model and recommends steps in the computation of MDP based stochastic optimization model, leading to the development of optimum pavement maintenance policy.

This volume deals with the theory of Markov Decision Processes (MDPs) and their applications. Each chapter was written by a leading expert in the respective area. The papers cover major research areas and methodologies, and discuss open questions and future research directions. The papers can be read independently, with the basic notation and concepts of Section 1.2. Most chap- ters should be accessible by graduate or advanced undergraduate students in fields of operations research, electrical engineering, and computer science.KeywordsMarkov Decision ProcessReward FunctionStochastic GameOptimality EquationAverage RewardThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

The paper is devoted to Time Division Multiple Access Link Scheduling Protocols in wireless sensor networks for full duplex (two-way) communication, where each sensor is scheduled on an incident link as a transmitter and as a receiver in two different time slots. We formulate the full duplex link scheduling problem (FDLSP) as distance-2 edge coloring in bidirected graphs. We proves that there exists a Δ-approximation algorithm for FDLSP (Δ being the maximum node degree in the network). Then, we present two distributed algorithms. The first is a synchronous algorithm based on finding maximal independent sets. The second is an asynchronous depth first search (DFS) algorithm. The maximal independent set based algorithm requires only O(Δlog*n) communication rounds (where n is the number of processors in the network) for growth bounded graphs, which is a realistic geometric model of sensor networks. For general graphs, the maximal independent set based algorithm requires O(Δ <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">4</sup> + Δ <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> log*n) communication rounds, improving upon the previous best algorithm with O(nΔ <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> +n <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> m) communication rounds (where m is the number of links in the network). The asynchronous DFS based algorithm requires only O(n) communication rounds for both general and growth bounded graphs. The simulations show that the proposed algorithms assign on average equal or fewer number of time slots compared to the best known distributed algorithm while being significantly faster.

This paper studies the admission control and resource allocation in a cell-based wireless system that supports single- media and multirate services. Utilizing the idea of adaptive and dynamic reallocation, we propose three control policies, AO, ADRF, and ADRA. The Markov decision process (MDP) theory is used to build the model of the system. We solve the MDP model and find the optimal policy by linear programming (LP). The proposed policies not only make the best use of precious wireless bandwidth, but also satisfy the requirement of quality of services such as handover blocking probability.

In this paper, a Long-Term Evolution-Advanced (LTE-A) Modified Best Channel Quality Indicator (M-BCQI) Scheduler is proposed for enhanced cell edge throughput performance. The scheduling algorithm utilizes two time slots, each of 0.5 ms in length, with the First Time Slot (FTS) implemented with the Best Channel Quality Indicator (BCQI) algorithm has a CQI-focused Resource Block (RB) allocation mechanism, while the Robin (RR) algorithm is the mechanism used in the Second Time Slot (STS) for RB allocation. Vienna LTE system level simulator was used for the proposed scheduling algorithm assessment. The results obtained showed significant improvement, indicating 44% superior cell edge performance and 56 % better fairness index in comparison to BCQI. However, M-BCQI slightly underperformed by 4.2% in terms of average throughput in comparison to BCQI and by 7% in comparison to RR in terms of fairness index.

With the development of Internet of Things (IoT) technology, wireless sensor networks (WSNs) composed of multiple sensor nodes (SNs) have been able to provide a large number of high-quality services, such as continuous environmental monitoring, automatic control, and transmission of data for intelligent decision-making. SNs in WSNs are typically energy limited and often exist in large numbers, which are widely distributed in hard-to-reach areas. Hence, it is difficult to collect data from SNs without infrastructure support such as base stations (BSs). The low cost and high maneuverability of unmanned aerial vehicles (UAVs) provide an efficient solution for data acquisition in WSNs but with the challenge of control system of the UAV. In this paper, we proposed a deep reinforcement learning based UAV control system to intelligently solve the data harvesting problem in WSNs. Our optimization target is to jointly optimize the trajectory of the UAV and the bandwidth resources of the airborne base station to maximize the data acquisition success rate when the UAV is limited in energy. The objective function and constraints of this problem are highly non-convex. To solve this problem, we first modeled the movement of the UAV and the bandwidth allocation of the airborne base station in each time slot as a Markov decision process (MDP). Then we designed a deep reinforcement learning architecture to solve the above MDP. Compared with the existing data harvesting algorithms, the proposed method has a great improvement in data acquisition success rate.

Opportunistic scheduling algorithms are effective in exploiting channel variations and maximizing system throughput in multirate wireless networks. However, most scheduling algorithms ignore the per-user quality-of-service (QoS) requirements and try to allocate resources (for example, the time slots) among multiple users. This leads to a phenomenon commonly referred to as the exposure problem, wherein the algorithms fail to satisfy the minimum slot requirements of the users due to substitutability and complementarity requirements of user slots. To eliminate this exposure problem, we propose a novel scheduling algorithm based on two-phase combinatorial reverse auction, with the primary objective of maximizing the number of satisfied users in the system. We also consider maximizing the system throughput as a secondary objective. In the proposed scheme, multiple users bid for the required number of time slots and the allocations are done to satisfy the two objectives in a sequential manner. We provide an approximate solution to the proposed scheduling problem, which is NP-complete. The proposed algorithm has an approximation ratio of (1 + log m) with respect to the optimal solution, where m is the number of slots in a schedule cycle. Simulation results are provided to compare the proposed scheduling algorithm with other competitive schemes.

Input Queued (IQ) switch architecture has been of interest due to its low memory bandwidth requirement. A scheduling algorithm is required to schedule the transfer of packets through cross-bar switch fabric at every time slot. The performance, that is throughput and delay, of a switch depends on the scheduling algorithm. The maximum weight matching (MWM) algorithm is known to deliver 100% throughput under any admissible traffic. Leonardi et. al. (2001) obtained a nontrivial bound on the delay for the MWM algorithm under admissible Bernoulli i.i.d. traffic. There has been a lot of interesting work done over time to analyze throughput of scheduling algorithms. But apart from the work of Leonardi et al. there has not been any work done to obtain bounds on delay of scheduling algorithms. The MWM algorithm is perceived to be a very good scheduling algorithm in general and simulations have suggested that it performs better than most of the known algorithms in terms of delay. But it is very complex to implement. Hence many simple to implement approximations to MWM have been proposed. In this paper, we study a class of algorithms to MWM, which always obtain a schedule whose weight W differs from the weight of MWM schedule W* by at most f(W*), where f(.) is a sub-linear function. We call this difference in weight as approximation distance of algorithm from MWM. We denote this class of algorithms by 1-APRX. We prove that any 1-APRX algorithm is stable, that is, it delivers upto 100% of throughput under any admissible Bernoulli i.i.d. input traffic. Under any admissible Bernoulli i.i.d. traffic, we obtain bounds on the average queue length(equivalently delay) of the 1-APRX algorithms using a Lyapunov function technique, which was motivated in Leonardi et al. The delay bounds obtained for the 1-APRX algorithm are linearly related with the approximation distance, which matches the intuition that the better the weight of the schedule, the better the algorithm will perform. Interestingly, simulations show a linear relationship between the average queue length (equivalently delay) and the approximation distance. Thus, the approximation distance of a scheduling algorithm can serve as a metric to differentiate between the performance of different stable algorithms, even though throughput may be same for these algorithms. We also obtain a novel heuristic tighter bound on the average queue length (equivalently delay) under uniform Bernoulli i.i.d. traffic for MWM using a very simple argument.

Long-Range Wide Area Network (LoRaWAN), as Low Power Wide Area Network that connects battery-powered devices, has gained much attention lately. The LoRaWAN technology uses ALOHA as the medium access control where the End Devices (EDs) transmit data randomly and re-transmit up to eight times if collisions occur, which is not feasible in large networks. Several techniques, including synchronization and scheduling schemes, to address the challenge have been reported. However, the existing synchronization and scheduling algorithms transmit synchronization messages randomly using one super frame with fixed time slots that accommodate devices using different Spreading Factors (SFs). This phenomenon can result in collisions, idle slots, and inefficient energy use, hence limiting the LoRaWAN network scalability. To alleviate the aforementioned problems, this work proposes a dynamic Multi-Frame Multi-Spreading Factor (MFMSF) scheduling algorithm with slotted synchronization approach. In the proposed algorithm, the time slots are assigned dynamically to EDs on first to synchronize first to be assigned basis. It was revealed that the proposed dynamic MFMSF TDMA algorithm is more energy-efficient than the existing algorithm in a denser network with devices arranged in a circular disk with different radii. The packet delivery rate for the dynamic MFMSF TDMA is about 99% signifying reduced collisions during data transmission. This implies that the use of the proposed dynamic MFMSF scheduling algorithm in LoRaWAN allows large number of EDs to communicate with reduced collisions and enhanced energy efficiency of the battery-powered devices.

This paper addresses the problem of scheduling real-time wireless flows under dynamic network conditions and general traffic patterns. The objective is to maximize the fraction of packets of each flow to be delivered within their deadlines, referred to as timely-throughput. The scheduling problem under restrictive frame-based traffic models or greedy maximal scheduling schemes like LDF has been extensively studied so far, but scheduling algorithms to provide deadline guarantees on packet delivery for general traffic under dynamic network conditions are very limited. We propose two scheduling algorithms using deep reinforcement learning approach to optimize timely-throughput for general traffic in dynamic wireless networks: RL-Centralized scheduling algorithm and RL-Decentralized scheduling algo-rithm. Specifically, we formulate the centralized scheduling problem as a Markov Decision Process (MDP) and a multi-environments double deep Q-network (ME-DDQN) structure is proposed to adapt to the dynamic network conditions. The decentralized scheduling problem is formulated as a Partially Observable Markov Decision Process (POMDP) and an expert-apprentice centralized training and decentralized execution (EA-CTDE) structure is designed to accelerate the training speed and achieve the optimal timely-throughput. The extensive results show that the proposed scheduling algorithms converge fast and adapt well to network dynamics with superior performance compared to baseline policies. Finally, experimental tests confirm simulation results and also show that the proposed algorithms are feasible in practice on resource limited platforms.

Markov programming with policy constraints

This paper is the first of two papers that present and evaluate an approach for determining suboptimal policies for large-scale Markov decision processes (MDP). Part 1 is devoted to the determination of bounds that motivate the development and indicate the quality of the suboptimal design approach; Part 2 is concerned with the implementation and evaluation of the suboptimal design approach. The specific MDP considered is the infinite-horizon, expected total discounted cost MDP with finite state and action spaces. The approach can be described as follows. First, the original MDP is approximated by a specially structured MDP. The special structure suggests how to construct associated smaller, more computationally tractable MDP's. The suboptimal policy for the original MDP is then constructed from the solutions of these smaller MDP's. The key feature of this approach is that the state and action space cardinalities of the smaller MDP's are exponential reductions of the state and action space cardinalities of the original MDP.