Deep Reinforcement Learning Aided Computation Offloading and Resource Allocation for IoT

Sparse code multiple access (SCMA) is a kind of code-domain non-orthogonal multiple access (NOMA) scheme, which can support the increasing requirements for high spectral efficiency and massive connections. Meanwhile, multi-access edge computing (MEC) is a promising technology for providing resource-constrained users with computing resources. The integration of these two technologies can efficiently improve the computation service. In this paper, we propose a novel optimization scheme in the SCMA-based MEC network from the perspective of energy and latency for the Internet of Things (IoT) devices. To minimize the total overhead of devices, the communication and computation resources allocation, as well as computation offloading are jointly considered. Primarily, a joint SCMA codebook assignment based on the bidirectional matching principle and optimal power allocation algorithm is proposed to maximize the uplink transmit rate. Moreover, we put forward CPU frequency scheduling strategies utilizing convex optimization to optimize the computing resources allocation of local devices and the MEC server. Finally, a low-complexity task offloading policy based on simulated annealing is presented. Numerical results show that our proposed joint optimization algorithm for resource allocation and computation offloading can achieve a good compromise between time delay and energy consumption for IoT devices.

Computation offloading services provide required computing resources for vehicles with computation-intensive tasks. Past computation offloading research mainly focused on mobile edge computing (MEC) or cloud computing, separately. This paper presents a collaborative approach based on MEC and cloud computing that offloads services to automobiles in vehicular networks. A cloud-MEC collaborative computation offloading problem is formulated through jointly optimizing computation offloading decision and computation resource allocation. Since the problem is non-convex and NP-hard, we propose a collaborative computation offloading and resource allocation optimization (CCORAO) scheme, and design a distributed computation offloading and resource allocation algorithm for CCORAO scheme that achieves the optimal solution. The simulation results show that the proposed algorithm can effectively improve the system utility and computation time, especially for the scenario where the MEC servers fail to meet demands due to insufficient computation resources.

In the current fifth-generation (5G) and Beyond 5G (B5G) era, the Unmanned Aerial Vehicles (UAVs) have been playing a vital role and attracting interest in different application areas in the military, and civil applications such as communications, disaster management, search and rescue, security, control, agriculture, Internet of things (IoT), etc. In these networks, ultra-heterogeneous IoT devices generate time-sensitive traffic. However, those devices have limited resources to compute tasks. Recently, Mobile Edge Computation Offloading (MECO) has been considered as an encouraging model to enable the computation tasks of IoT devices to be performed by MEC servers and support ultra-low latency IoT applications to ensure Quality of services (QoS). However, terrestrial network failure due to natural and human-made disasters has been increasing, and difficult to provide reliable computation offloading and resource allocation services to IoT networks. Nowadays, UAVs have been promising technology to quickly deploy and recover the system to provide efficient services to edge nodes. The offloading and resource allocation problems in current network technology are complex, and offloading task to edge server is vulnerable to security risks. Hence, we utilize a deep reinforcement learning method to handle a complex problem for computation offloading and resource allocations in a dynamic environment. And also, we explore a blockchain-based multi-UAV-assisted MEC architecture in securing and optimizing the offloading problems.

5G beyond is an end-edge-cloud orchestrated network that can exploit heterogeneous capabilities of the end devices, edge servers, and the cloud and thus has the potential to enable computation-intensive and delay-sensitive applications via computation offloading. However, in multi user wireless networks, diverse application requirements and the possibility of various radio access modes for communication among devices make it challenging to design an optimal computation offloading scheme. In addition, having access to complete network information that includes variables such as wireless channel state, available bandwidth and computation resources, is a major issue. Deep Reinforcement Learning (DRL) is an emerging technique to address such an issue with limited and less accurate network information. In this paper, we utilize DRL to design an optimal computation offloading and resource allocation strategy for minimizing system energy consumption. We first present a multi-user end-edge-cloud orchestrated network where all devices and base stations have computation capabilities. Then, we formulate the joint computation offloading and resource allocation problem as a Markov Decision Process (MDP) and propose a new DRL algorithm to minimize system energy consumption. Numerical results based on a real-world dataset demonstrate that the proposed DRL-based algorithm significantly outperforms the benchmark policies in terms of system energy consumption. Extensive simulations show that learning rate, discount factor, and number of devices have considerable influence on the performance of the proposed algorithm.

Since the Internet of Things (IoT) will be entwined with everything we use in our daily life, the consequence of security flaws escalates. Smart objects will govern most of the home appliances and car engines yielding potential disaster scenarios. In this context, successful attacks could lead to chaos and scary scenarios (www.darkreading.com). Unprotected personal information may expose sensitive and embarrassing data to the public and attacks may threaten not only our computers and smart devices, but our intimacy and perhaps our lives too. Because persons and objects will be bonded with each other, user consent becomes critical. Therefore, thing, object, and user Identity will be the focus of future IoT security solutions, yielding a Trust, Security, and Privacy (TSP) paradigm, which may constitute the Achilles' heel of IoT. While security issues are quite straightforward, mainly from background knowledge, privacy issues are far more complex. Privacy constitutes a rather challenging task, even for the m...

For future Internet of Things (IoT) systems, data-driven and dynamic spectrum-sharing schemes can significantly improve the spectrum utilization and efficiency. However, conventional centralized architecture of such dynamic IoT spectrum-sharing systems is often considered to be nontransparent, costly, and vulnerable to potential attacks and single-point failures. To address the aforementioned issues, a blockchain-based dynamic spectrum-sharing scheme has been proposed and investigated in this work, which aims at enhancing the system by providing desirable features, such as decentralization, transparency, immutability, and auditability. By considering the privacy and transaction dynamics issues when blockchain is integrated into spectrum-sharing systems, a privacy-preserving double auction mechanism based on differential privacy is developed for incentivizing spectrum sharing, where the time-varying valuations of the spectrum resources are also taken into consideration. In the proposed auction, a winner determination problem (WDP) is formulated to decide the winning bidders and spectrum allocation. A deep reinforcement learning (DRL)-based method is then proposed for efficiently solving the WDP. The proposed auction mechanism can be integrated with smart contracts on blockchain platforms. Furthermore, the computation of the DRL-based method for solving the WDP is designed as part of the consensus mechanism in the blockchain. Theoretical analysis show that the proposed privacy-aware double auction mechanism satisfies the properties of differential privacy, individual rationality, and truthfulness. Finally, simulation results are provided to validate the performance of the spectrum-sharing approach.

Internet of Things (IoT) is a promising technology to connect tremendous devices together, where the major challenges are that the available energy is limited and computing capability is low. In this paper, we propose an efficient IoT computing tasks offloading mechanism based on cooperative communication and mobile cloud computing (MCC) system. The problem will be formulated as a joint optimization problem of computation and radio resource allocation aiming to minimize the system energy consumption, under the constraints of latency and transmission power. We will first propose a joint iterative computation offloading and resource allocation algorithm to solve the non-convex optimization problem. To further reduce the computation complexity, we also propose a matching based sub- optimal algorithm to solve this problem. Simulation results demonstrate that the proposed iterative algorithm achieves the goal to substantially reduce energy consumption by offloading computation. Moreover, the sub-optimal algorithm significantly reduces the computation complexity with only a small portion of performance loss.

Herein, an Internet of Things (IoT) system with remote reconfigurable wireless sensor network (WSN) nodes is presented and its application to measure group work activity in a classroom is discussed. This IoT system consists of wiki pages, bots, (defined as remotely controlled devices), gateways, and remotely reconfigurable WSN Nodes. The bots/gateways and WSN nodes of the IoT system are controlled by scripts written on wiki pages that are read by the bot/gateways, which then forward the commands contained in the scripts to the WSN nodes. Each WSN node then executes the commands, obtains the resulting sensor values, and then returns those values to the bot/gateways. The bot/gateways relay the values and write them onto the wiki page. Each WSN node is flexible and reconfigurable because its behavior can be changed by the script commands written onto the wiki page. Using this IoT system, we created a measurement system to evaluate activity levels of group work in a classroom. After developing the measurement system, we found it was possible to adjust the sampling rate for obtaining sensor values simply by rewriting the script on the wiki page without directly manipulating the WSN.

Applying blockchain technology in the Internet of Things (IoT) systems to realize massive data storage is a promising technology, which can eliminate the dependence of IoT on central servers and protect data security effectively. However, in legacy IoT systems, it is hardly to store a complete blockchain at all nodes due to the limited storage space. In order to securely store massive data in the resource-constrained IoT systems, we design a blockchain-assisted distributed storage mechanism, by which multiple nodes collaboratively store a complete blockchain. Meanwhile, we design a deep reinforcement learning (DRL) based algorithm to obtain the optimal block allocation strategy for the collaborative storage. The optimal strategy can maximize the number of storage blocks so as to maximize the utilization of the limited storage space under the premise of meeting the requirements of user query delay and reliability. The simulation results show that compared with the random search algorithm and the policy gradients algorithm, the proposed Deep Q Network algorithm can achieve the maximum number of storage blocks in the same storage space, and the fastest storage speed.

Cloud radio access networks (CRANs) have become a key enabling technique for the next generation wireless communications. Resource allocation in CRANs still needs to be further improved to reach the objective of minimizing power consumption and meeting demands of wireless users over a long period. Inspired by the success of Deep Reinforcement Learning (DRL) on solving complicated control problems, we present a novel framework, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ReCARL</i> , for power-efficient resource allocation in CRANs with deep reinforcement learning. Specifically, we define the state space, action space and reward function for the DRL agent, apply a deep neural network (DNN) to approximating the action-value function, and formally formulate the resource allocation problem (in each decision epoch) as a convex optimization problem. Under ReCARL, we propose two different DRL agents: one has a regular DNN structure trained with the basic deep Q-learning method ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ReCARL-Basic</i> ); while the other has a context-aware DNN structure trained with a hybrid deep Q-learning method ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ReCARL-Hybrid</i> ). We evaluated the performance of ReCARL along with the two DRL agents by comparing them with two widely-used baselines via extensive simulation. The simulation results show that ReCARL achieves significant power savings while meeting user demands, and it can well handle highly dynamic cases.

LoRaWAN based internet of things (IoT) system for precision irrigation in plasticulture fresh-market tomato

By pushing computation to the mobile network edge, Multi-access Edge Computing (MEC) has been an enabler for the stringent latency and energy requirements of the new Internet of Things (IoT) services. On the other hand, ultra-dense heterogeneous networks with wireless backhaul have been proposed as a low-cost solution, allowing Network Operators (NOs) to extend the network capability, by deploying densified close-proximity small-cells and hence supporting a large number of low-latency low-energy IoT devices. In this paper, we study the problem of IoT task offloading in a MEC-enabled heterogeneous network, which to the best of our knowledge, is the first attempt to thoroughly explore the task offloading problem in a heterogeneous network with MEC support and wireless backhaul. We jointly optimize the offloading decision, transmission power, and the allocation of radio and computational resources, with the objective of minimizing the devices energy consumption, while respecting their latency deadline. We mathematically formulate our problem as a non-convex mixed-integer program, and due to its complexity, we propose an iterative algorithm based on the Successive Convex Approximation (SCA) method for providing an approximate solution on the original problem. Through numerical analysis, we perform simulations based on multiple scenarios, and find out how NOs can respond to the requested load and help in minimizing the total devices energy consumption.

Privacy preservation in blockchain based IoT systems: Integration issues, prospects, challenges, and future research directions

Deep reinforcement learning (DRL) has been extensively used to address portfolio optimization problems. DRL agents acquire knowledge and make decisions through unsupervised interactions with their environment without requiring explicit knowledge of the joint dynamics of portfolio assets. Among these DRL algorithms, the combination of actor-critic algorithms and deep function approximators is the most widely used DRL algorithm. Here, we find that training the DRL agent using the actor-critic algorithm and deep function approximators may lead to scenarios where the improvement in the DRL agent's risk-adjusted profitability is insignificant. We argue that such situations primarily arise from the following two problems: sparsity in positive reward and the curse of dimensionality. These limitations prevent DRL agents from comprehensively learning asset price change patterns in the training environment. As a result, the DRL agents cannot effectively explore the dynamic portfolio optimization policy to improve the risk-adjusted profitability in the training process. To address these problems, we propose a novel multi-agent learning system based on the hierarchical deep reinforcement learning (HDRL) algorithmic framework in this research. Under this framework, the agents work together as a learning system for portfolio optimization. Specifically, by designing an auxiliary agent that works together with the executive agent for optimal policy exploration, the learning system can focus on exploring the policy with higher risk-adjusted return in the action space with positive return and low variance. The performance of the proposed learning system is evaluated using a portfolio of 29 stocks from the Dow Jones index in four different experiments. In the training process, the objective functions of the actor and critic both ultimately achieve stable convergence in the training process. The risk-adjusted profitability of our learning system in the training environment is significantly improved. Hence, we prove that the policies executed by our learning system in out-sample experiments originate from the DRL agents' comprehensive learning of asset price change patterns in the training environment. Furthermore, we find that adopting the auxiliary agent and HDRL training algorithm can efficiently overcome the issue of the curse of dimensionality and improve the training efficiency in the positive reward sparse environment. In each back-test experiment, the proposed learning system is compared to sixteen traditional strategies and ten strategies based on machine learning algorithms in the performance of profitability and risk control ability. The empirical results in the four evaluation experiments demonstrate the efficacy of our learning system, which outperforms all other strategies by at least 8.2% in terms of Sharpe ratio, Sorino ratio, and Calmar ratio. This indicates that the policies learned in the training environment can exhibit excellent generalization ability in the back-testing experiments.

A survey of trust computation models for service management in internet of things systems