Three-Tier Computing Platform Optimization: A Deep Reinforcement Learning Approach

Millimeter-wave (mmWave) communication is a key technology of fifth generation wireless systems to achieve the expected 1000x data rate. With large bandwidth at the mmWave band, the link capacity between users and base stations (BSs) can be much higher compared to sub-6 GHz wireless systems. Meanwhile, due to the high cost of infrastructure upgrade, it would be difficult for operators to drastically enhance the capacity of backhaul links between mmWave BSs and the core network. As a result, the data rate provided by backhaul may not be sufficient to support all mmWave links; hence, the backhaul connection becomes the new bottleneck. On the other hand, as mmWave channels are subject to random blockage, the data rates of mmWave users significantly vary over time. With limited backhaul capacity and highly dynamic data rates of users, how to allocate backhaul resource to each user remains a challenge for mmWave systems. In this article, we present a deep reinforcement learning (DRL) approach to address this challenge. By learning the blockage pattern, the system dynamics can be captured and predicted, resulting in efficient utilization of backhaul resource. We begin with a discussion on DRL and its application in wireless systems. We then investigate the problem of backhaul resource allocation and present the DRL-based solution. Finally, we discuss open problems for future research and conclude this article.

This paper investigates the covert communications via spectrum allocations in a cellular-enabled unmanned aerial vehicle (UAV) network consisting of a base station (BS), UAVs, ground users (GUs), and a warden, where warden attempts to detect the transmission from a target GU to a UAV receiver. We formulate the spectrum allocation as an optimization problem with the constraints of covertness performance requirement and the qualities of service (QoS) of cellular communications. This is a nonlinear and nonconvex problem, which is generally challenging to be solved. Thus, we propose a deep reinforcement learning (DRL) approach to solve it. Under such an approach, we first model the multi-agent DRL environment in such networks. Then we define the state, action, reward and interaction mechanism of the DRL environment. Finally, a DRL algorithm is presented for learning the optimal policy of spectrum allocation.

Energy efficient task scheduling based on deep reinforcement learning in cloud environment: A specialized review

This paper investigates multicast beamforming for scalable videos in cache-enabled heterogeneous networks, where a macro base station (MBS) and multiple small base stations (SBSs) serve multicast group users on-demand video services. Inspired by the main idea of the scalable video coding, the extension technology of the H.264#x002F;advanced video coding, each video is divided into a base layer (BL) and an enhancement layer (EL). The BL can provide the fundamental viewing quality and adding EL to the received BL can guarantee superior perceptual experience. The MBS and multiple SBSs cache the BLs and ELs of the most popular videos, respectively. Employing the transmission schemes in coordinated multi-point, i.e., parallel transmission and joint transmission, the MBS and SBSs parallelly transmit BLs and ELs while the SBSs cooperatively transmit the ELs if higher video qualities are required. To improve the quality of the received videos for users whose requirements can be satisfied locally, our aim is to maximize the weighted sum rate of them and this problem can be converted into an iterative second-order cone programming problem by successive convex optimization with low complexity. Numerical results demonstrate the advantage of our proposed scheme compared to the benchmark scheme, i.e., only MBS serving the end users, even when the transmit power of each SBS is kept low.

Multi-access edge computing (MEC) is the key enabling technology that supports compute-intensive applications in 5G networks. By deploying powerful servers at the edge of wireless networks, MEC can extend the computational capacity of the mobile devices by migrating compute-intensive tasks to the MEC servers. In this paper, we consider a multi-user MEC wireless network in which multiple mobile devices can associate and perform computation offloading via wireless channels to MEC servers attached to the base stations (BSs). The decision whether the computation task is executed locally at the user device or to be offloaded for MEC server execution should be adaptive to the time-varying network dynamics. Taking into account the dynamic of the environment, we propose a deep reinforcement learning (DRL) based approach to solve the formulated nonconvex problem of minimizing computation cost in terms of total delay. However, real-world networks tend to have a large number of users and MEC servers involving large numbers of different actions (continuous and discrete), where evaluating the combination of every possible action becomes impractical. Therefore, conventional DRL methods may be difficult or even impossible to directly apply to the proposed model. Based on the recursive decomposition of the action space available to each state, we propose a DRL-based algorithm for joint server selection, cooperative offloading, and handover in a multi-access edge wireless network. Numerical results show that the proposed DRL based algorithm significantly outperforms the traditional Q-learning method and local computation in terms of task success rate and total delay.

In this work we investigate optimal geographical caching in heterogeneous cellular networks where different types of base stations (BSs) have different cache capacities. Users request files from a content library according to a known probability distribution. The performance metric is the total hit probability, which is the probability that a user at an arbitrary location in the plane will find the content that it requires in one of the BSs that it is covered by. We consider the problem of optimally placing content in all BSs jointly. As this problem is not convex, we provide a heuristic scheme by finding the optimal placement policy for one type of base station conditioned on the placement in all other types. We demonstrate that these individual optimization problems are convex and we provide an analytical solution. As an illustration, we find the optimal placement policy of the small base stations (SBSs) depending on the placement policy of the macro base stations (MBSs). We show how the hit probability evolves as the deployment density of the SBSs varies. We show that the heuristic of placing the most popular content in the MBSs is almost optimal after deploying the SBSs with optimal placement policies. Also, for the SBSs no such heuristic can be used; the optimal placement is significantly better than storing the most popular content. Finally, we show that solving the individual problems to find the optimal placement policies for different types of BSs iteratively, namely repeatedly updating the placement policies, does not improve the performance.

To meet the subscribers ever increasing traffic demand, micro and macro base stations are being deployed excessively. As the traffic pattern varies according to the user's behavior, the deployment of micro and macro base stations plays a vital role in saving energy while maintaining the traffic demand of the subscribers. A macro base station consumes more than double the energy than a micro base station. Due to the space and time characteristics of the traffic, the BS cannot allocate resources effectively, which results in wasting energy consumption and low energy efficiency. Therefore, energy saving through deployment of base stations play a significant role to increase the energy efficiency. In this paper, the user traffic pattern is determined and the resources needed to fulfill the traffic are analyzed and finally the deployment strategies for the base stations are formulated.
 Since the base stations are fully loaded only for few hours a day, energy saving on the stations during low traffic will be significant. The energy saving schemes saved up to 18.8 % of energy in macro and 26.9 % of energy in micro BS. So, it would be more efficient to implement a heterogeneous network with more micro cells with energy saving schemes than just macro base stations.

Fair resource allocation and SBS selection for MC-NOMA-based HetNets supporting D2D communications

We introduce a sleep mechanism for energy efficiency in heterogeneous cellular networks, where macro base stations and small base stations coexist in the same frequency band. The strategy is to put high-power macro base stations into sleep mode and offload the users to low-power small base stations or neighboring macro base stations. To do so, we first formulate a joint optimization problem to minimize the total energy consumption while maintaining the QoS of users. Then we decouple it into two subproblems: user association and resource allocation, and macro base station sleep mechanism. We propose a modified many-to-one matching algorithm to solve the first one and a voting-based dynamic sleep mechanism for the second one. Simulation results show that our scheme has both lower energy consumption and lower user blocking ratio compared with existing sleep mechanisms.

Future mobile networks are converging toward heterogeneous multi-tier networks, where various classes of base stations (BS) are deployed based on user demand.So it is quite necessary to utilize the BSs resources rationally when BSs are sufficient.In this paper, we develop a more realistic model that fully considering the inter-tier dependence and the dependence between users and BSs, where the macro base stations (MBSs) are distributed according to a homogeneous Poisson point process (PPP) and the small base stations (SBSs) follows a Matern cluster process (MCP) whose parent points are located in the positions of the MBSs in order to offload the users from the over-loaded MBSs.We also assume the users are just randomly located in the circles centered at the MBSs.Under this model, we derive the association probability and the average ergodic rate by stochastic geometry.An interesting result that the density of MBS and the radius of the clusters jointly affect the association probabilities in a joint form is obtained.We also observe that using the clustered SBSs results in aggressive offloading compared with previous cellular networks.

Optimal scheduling of island integrated energy systems considering multi-uncertainties and hydrothermal simultaneous transmission: A deep reinforcement learning approach

The modern network is equipped with many service functions to acquire high-quality service. The emergence of network function virtualization (NFV) provides a convenient way to accomplish the network services in the form of virtual network function (VNF) and also makes the scheduling solution of VNFs flexible. The VNFs can be deployed on commodity servers as software processes. Besides, multiple VNFs are chained in a specified order as a service function chain (SFC) to serve a given flow, increasing the scheduling difficulty to minimize the average flow completion time. In this paper, we study the problem of scheduling multiple SFCs with the constraint of different resource limitations in various commodity servers. This problem is typically formulated as an Integer Linear Programming (ILP) problem, which is NP-hard. To well tackle this problem, we propose a deep reinforcement learning (DRL) approach. It involves multi-step decision making, which can be naturally transformed into a DRL problem. We design specific reward and state representations for such a multi-resource scheduling problem. We also consider how to use DRL to handle online requests of SFCs. The experiment results demonstrate that the DRL approach can significantly reduce the average completion time of a set of SFC and achieves a cost saving of 39.94% against the benchmark method.

SIR analysis for non-uniform HetNets with joint decoupled association and interference management

With the growth of mobile data traffic in wireless networks, caches are used to bring data closer to mobile users and to minimize the traffic load on macro base station (MBS). Storing data in caches on user terminals (UTs) and small base stations (SBSs) faces challenges on which data to cache and where to cache these data. The process of deciding the cache contents involves multiple objectives regarding the content popularity, contact duration between UT and SBSs, communication ranges between UT and SBSs caches, and contact probability between UT and SBSs. In this paper, we propose a new strategy on cache placement decisions for mobile edge networks based on binary classification technique. The aim is to formulate the cache placement as a classification problem that is solved using machine learning techniques in order to define an optimal decision boundary on cache or not cache decisions. Simulation results show that the performance of cache placement algorithms using classifier based learning techniques can achieve higher hit rate than other algorithms.

In this paper, secure transmission in a two-tier heterogeneous network, consisting of a macrocell and a set of small cells, is considered, where an eavesdropper attempts to wiretap legitimate macrocell users. A game-theoretic framework is proposed to enhance the security of the macrocell while guaranteeing user satisfaction of the small cells. Specifically, a two-level Stackelberg game is formulated, where the macro base station as a follower employs multiple small base stations (SBSs) to jam the eavesdropper at the top level, and each employed SBS as a leader requires offloading service from multiple helping SBSs in its cluster at the bottom level if needed. Two types of objectives are investigated at the bottom level, one with the priority given to user satisfaction over the leader's profit, and the other to balance between user satisfaction and the leader's profit, respectively. The Stackelberg equilibrium of the proposed game is also derived. Numerical results verify the analysis and show that employing the existing SBSs is a promising approach to enhancing security while satisfying the user demands of the small cells.