A Time-Varying Opportunistic Multiple Access for Delay-Sensitive Inference in Wireless Sensor Networks

This chapter presents an overview of simulation-based techniques useful for solving Markov decision processes (MDPs). MDPs model problems of sequential decision-making under uncertainty, in which decisions made in each state collectively affect the trajectory of the states visited by the system over a time horizon of interest. Traditionally, MDPs have been solved via dynamic programming (DP), which requires the transition probability model that is difficult to derive in many realistic settings. The use of simulation for solving MDPs allows us to bypass the transition probability model and solve large-scale MDPs considered intractable to solve by traditional DP. The simulation-based methodology for solving MDPs, which like DP is also rooted in the Bellman equations, goes by names such as reinforcement learning, neuro-DP, and approximate or adaptive DP. We begin with a description of algorithms for infinite-horizon discounted reward MDPs, followed by the same for infinite-horizon average reward MDPs. Then we present a discussion on finite-horizon MDPs. For each problem considered, we present a step-by-step description of a selected group of algorithms. In making this selection, we have attempted to blend the old and the classical with more recent developments. Finally, after touching upon extensions and convergence theory, we conclude with a brief summary of some applications and directions for future research.

We consider a single-hop, random-access, wireless sensor network (WSN) performing sequential distributed detection under a bandwidth constraint. At each time slot, the sensor nodes will make their decisions on whether the observed event is happening. The sensor censoring strategy is applied such that only the local decisions equal to one (i.e., the event happens) will be sent to the fusion center (FC). Because only one transmission channel between the FC and the sensor nodes is assumed, if two or more local decisions are sent at a time slot, a packet collision happens. We design a sequential probability ratio test (SPRT) at the FC that is aware of the collisions. The performance measures of the collision-aware SPRT are analyzed based on a Markov chain. In addition, we derive the asymptotic relative efficiency of the collision-aware SPRT relative to another SPRT. Numerical results show some interesting characteristics of the collision-aware SPRT. For example, the collision-aware SPRT prefers the scenario that many local decisions are likely to be sent per time slot.

In this paper, we consider the problem of sensor management for target tracking in a wireless sensor network (WSN). To determine the set of sensors with the most informative data, we develop a probabilistic sensor management scheme based on the concepts developed in compressive sensing. In the proposed scheme where each sensor transmits its observation with a certain probability via a coherent multiple access channel (MAC), the observation vector received at the fusion center becomes a compressed version of the original observations. In this framework, the sensor management problem can be cast as the problem of finding the probability of transmission at each node so that a given performance metric is optimized. Our goal is to determine the optimal values of the probabilities of transmission so that the trace of the Fisher information matrix (FIM) is maximized at any given time instant with a constraint on the available energy. We consider two cases, where the fusion center has i) complete information and ii) only partial information, regarding the sensor transmissions. The expression for FIM is derived for both cases and the optimal values of the probabilities of transmission are found accordingly. With nonidentical probabilities, we obtain the results numerically while under the assumption that each sensor transmits with equal probability, we obtain the optimal values analytically. We provide numerical results to illustrate the performance of the proposed probabilistic sensor management scheme.

In this paper, downlink transmission scheduling of popular files is optimized with the assistance of wireless cache nodes. Specifically, the requests of each file, which is further divided into a number of segments, are modeled as a Poisson point process within its finite lifetime. Two downlink transmission modes are considered: 1) the base station reactively multicasts the file segments to the requesting users and selected cache nodes and (2) the base station proactively multicasts some file segments to the selected cache nodes without requests. The cache nodes with decoded file segments can help to offload the traffic via other spectrum. Without the proactive multicast, we formulate the downlink transmission resource minimization as a dynamic programming problem with random stage number, which can be approximated via a finite-horizon Markov decision process (MDP) with fixed stage number. To address the prohibitively huge state space, we propose a low-complexity scheduling policy by linearly approximating the value functions of the MDP, where the bound on the approximation error is derived. Moreover, we propose a learning-based algorithm to evaluate the approximated value functions for unknown geographical distribution of requesting users. Finally, given the above reactive multicast policy, a proactive multicast policy is introduced to exploit the temporal diversity of shadowing effect. It is shown by simulation that the proposed low-complexity reactive multicast policy can significantly reduce the resource consumption at the base station, and the proactive multicast policy can further improve the performance.

Energy-efficient sampling of networked control systems over IEEE 802.15.4 wireless networks

BackgroundWe propose a mathematical model formulated as a finite-horizon Markov Decision Process (MDP) to allocate capacity in a radiology department that serves different types of patients. To the best of our knowledge, this is the first attempt at considering radiology resources with different capacities and individual no-show probabilities of ambulatory patients in an MDP model. To mitigate the negative impacts of no-show, overbooking rules are also investigated.MethodsThe model’s main objective is to identify an optimal policy for allocating the available capacity such that waiting, overtime, and penalty costs are minimized. Optimization is carried out using traditional dynamic programming (DP). The model was applied to real data from a radiology department of a large Brazilian public hospital. The optimal policy is compared with five alternative policies, one of which resembles the one currently used by the department. We identify among alternative policies the one that performs closest to the optimal.ResultsThe optimal policy presented the best performance (smallest total daily cost) in the majority of analyzed scenarios (212 out of 216). Numerical analyses allowed us to recommend the use of the optimal policy for capacity allocation with a double overbooking rule and two resources available in overtime periods. An alternative policy in which outpatients are prioritized for service (rather than inpatients) displayed results closest to the optimal policy, being also recommended due to its easy implementation.ConclusionsBased on such recommendation and observing the state of the system at any given period (representing the number of patients waiting for service), radiology department managers should be able to make a decision (i.e., define number and type of patients) that should be selected for service such that the system’s cost is minimized.

This paper proposes and analyzes impulse radio ultra-wideband (IR-UWB) system for wireless sensor networks (WSNs) using massive antenna arrays (MAAs) at fusion center (FC) for distributed detection. The coherent- and energy-based fusion rules are analyzed for the proposed WSN over multiple access channels. The tradeoff between the performance and the implementation complexity of the coherent- and energy-based fusion is studied. Furthermore, we show that MAAs at FC with various levels of channel knowledge can enhance the performance of energy-based detector in a UWB sensor network with simple system implementation and signal processing. Performance of the proposed UWB sensor network with reference to probability of detection, false alarm, and error are analyzed over the standard IEEE 802.15.4a multipath channels and results are validated using simulations. We have also analyzed the impact of various design parameters, such as the number of sensors, receiver antennas, sensor quality, and integration interval, on the system performance.

We consider the distributed estimation of an unknown vector signal in a bandwidth constrained sensor network with a fusion center (FC). Due to power and bandwidth limitations, each sensor compresses its data in order to minimize the amount of information that needs to be communicated to the FC. In this context, we design a linear decentralized estimation scheme (DES), where each sensor linearly encodes its observations before the transmission to the FC, which performs a minimum mean squared error (MMSE) estimation for the unknown vector signal based on the received messages. When the channels between sensors and the FC are orthogonal, it has been shown previously that the complexity of designing the optimal encoding matrices is NP-hard in general. In this paper, we study the optimal design of linear DES for the case of non- orthogonal multiple access channel (MAC) under both bandwidth and power constraints. We show that when the MAC between sensors and the FC is noiseless, the resulting problem has a closed-form solution, while in the noisy MAC case, the problem can be efficiently solved by semi-definite programming (SDP).

With the development of Internet of Things (IoT) technology, wireless sensor networks (WSNs) composed of multiple sensor nodes (SNs) have been able to provide a large number of high-quality services, such as continuous environmental monitoring, automatic control, and transmission of data for intelligent decision-making. SNs in WSNs are typically energy limited and often exist in large numbers, which are widely distributed in hard-to-reach areas. Hence, it is difficult to collect data from SNs without infrastructure support such as base stations (BSs). The low cost and high maneuverability of unmanned aerial vehicles (UAVs) provide an efficient solution for data acquisition in WSNs but with the challenge of control system of the UAV. In this paper, we proposed a deep reinforcement learning based UAV control system to intelligently solve the data harvesting problem in WSNs. Our optimization target is to jointly optimize the trajectory of the UAV and the bandwidth resources of the airborne base station to maximize the data acquisition success rate when the UAV is limited in energy. The objective function and constraints of this problem are highly non-convex. To solve this problem, we first modeled the movement of the UAV and the bandwidth allocation of the airborne base station in each time slot as a Markov decision process (MDP). Then we designed a deep reinforcement learning architecture to solve the above MDP. Compared with the existing data harvesting algorithms, the proposed method has a great improvement in data acquisition success rate.

Appropriate planning and optimization strategies for day-ahead power management play important roles in efficient operation of Microgrids (MGs). Due to the uncertainties in electricity demand and renewable generations, and the multi-objective (MO) nature of MG power management, conventional optimization techniques have not been as effective in giving satisfactory results. This paper aims at solving the day-ahead power management problem as a MO optimization problem, with a focus on increasing the system's resiliency using an agent-based Dynamic Programming (DP) approach named Value Iteration (VI) and a model-free Q-learning (QL) algorithm. The two objectives of the MO problem are: maximizing load serviceability and minimizing operational cost. Both the approaches are data-driven, and the behavior of the agent of each component of a MG is formulated as a finite-horizon Markov Decision Process (MDP). VI guarantees an optimal solution to the MO problem given the MDP model, and QL has the ability to work under uncertainty and incomplete information. The effectiveness of the two algorithms have been evaluated using a benchmark MG test system.

We consider data confidentiality in a distributed detection scenario with a type-based multiple-access (TBMA) protocol where a large set of sensors sends local measurements to an ally fusion center (FC) over an insecure wireless medium called the main channel. Then, the ally FC makes a final decision to the physical environment. Although many wireless sensor networks are mission-specific and need data confidentiality due to the broadcast nature of wireless transmission, it can be easily wiretapped by unauthorized enemy FCs through eavesdropping channels. We propose a novel TBMA protocol called secure TBMA which provides data confidentiality by taking advantage of inherent properties of wireless channels, namely randomness and independence of the main and eavesdropping channels. In particular, the secure TBMA activates sensors having strong and weak main channel gains and makes the sensors follow different reporting rules based on the magnitudes of their channel gains. The reporting rules are carefully designed to confuse the enemy FC. The proposed secure TBMA delivers unconditional/perfect secrecy and does not assume any superiority of the ally FC over the enemy FC in terms of computational capability, secret key, and so on. For Rayleigh fading channels, we analyze the performance of the secure TBMA at both enemy and ally FCs by investigating conditions for perfect secrecy and an error exponent of detection error probability, respectively. On the one hand, the analysis at the enemy FC provides a design criterion of the reporting rules to achieve perfect secrecy. On the other hand, the analysis of the error exponent carried out with a Gaussian approximation shows that perfect secrecy is achievable at a marginal cost in detection error performance. All our claims are also verified with simulation results which have good matches with the analysis.

We consider the uplink of an energy harvesting (EH) wireless sensor network (WSN) with two users communicating with one common fusion center (FC), proposing resource allocation algorithms aiming to minimize the average age of information (AoI) for two different scenarios, i.e., frequency-division multiple access (FDMA) and time-division multiple access (TDMA). Furthermore, the convexity of a lower bound of AoI is shown in this manuscript, facilitating the proposed allocation algorithms to find the optimal point with low complexity. Numerical results show the equivalency between the brute-force search and proposed algorithms and that FDMA can achieve the smaller AoI than TDMA over fading channels.

In this paper, we tackle decision fusion for distributed detection in a randomly-deployed clustered Wireless Sensor Networks (WSNs) operating over a non-ideal multiple access channels (MACs), i.e. considering Rayleigh fading, path loss and additive noise. To mitigate fading, we propose the distributed equal gain transmit combining (dEGTC) and distributed maximum ratio transit combining (dMRTC). The first and second order statistics of the received signals were analytically computed via stochastic geometry tools. Then the distribution of the received signal over the MAC are approximated by Gaussian and log-normal distributions via moment matching. This enabled the derivation of moment matching optimal fusion rules (MOR) for both distributions. Moreover, suboptimal simpler fusion rules were also proposed, in which all the CHs data are equally weighed, which is termed moment matching equal gain fusion rule (MER). It is shown by simulations that increasing the number of clusters improve the performance. Moreover, MOR-Gaussian based algorithms are better under free-space propagation whereas their lognormal counterparts are more suited in the ground-reflection case. Also, the latter algorithms show better results in low SNR and SN numbers conditions. We have proved that the received power at the CH in MAC is proportional <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mathcal {O}(\lambda ^{2} R^{2})$</tex-math></inline-formula> and to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mathcal {O}(\lambda ^{2} \ln ^{2} R)$</tex-math></inline-formula> in the free-space propagation and the ground-reflection cases respectively, where <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\lambda$</tex-math></inline-formula> is SN deployment intensity and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$R$</tex-math></inline-formula> is the cluster radius. This implies that having more clusters decreases the required transmission power for a given SNR at the receiver.

Many sensor nodes have been widely deployed in the physical world to gather various environmental information, such as water quality, earthquake, and huge dam safety. Due to the limitation in the batter power, memory, and computational capacity, missing data can occur at arbitrary sensor nodes and time slots. In extreme situations, some sensors may lose readings at consecutive time slots. The successive missing data takes the side effects on the accuracy of real-time monitoring as well as the performance on the data analysis in the wireless sensor networks. Unfortunately, existing solutions to the missing data filling cannot well uncover the complex non-linear spatial and temporal relations. To address these problems, a DNN (Deep Neural Network) multi-view learning method (DNN-MVL) is proposed to fill the successive missing readings. DNN-MVL mainly considers five views: global spatial view, global temporal view, local spatial view, local temporal view, and semantic view. These five views are modeled with inverse distance of weight interpolation, bidirectional simple exponential smoothing, user-based collaborative filtering, mass diffusion-based collaborative filtering with the bipartite graph, and structural embedding, respectively. The results of the five views are aggregated to a final value in a multi-view learning algorithm with DNN model to obtain the final filling readings. Experiments on large-scale real dam deformation data demonstrate that DNN-MVL has a mean absolute error about 6.5%, and mean relative error 21.4%, and mean square error 8.17% for dam deformation data, outperforming all of the baseline methods.

In this paper, an iterative distributed approach is studied for the signal amplitude optimization when the distributed detection is carried out with a symmetric signaling constraint in a wireless sensor network (WSN) consisting of a fusion center (FC) and multiple sensors. In the conventional distributed detection, if sensors'' characteristics change due to aging problems or varying operating conditions, they have to be available at the FC for a best combining result. On the other hand, in the proposed approach, each sensor can adjust its signal amplitude using a type-based multiple access (TBMA) technique to provide a best performance at the FC through an iterative approach without sending their individual characteristics.