Intelligent User Association for Symbiotic Radio Networks Using Deep Reinforcement Learning

Symbiotic radio (SR) is a promising technology to support ultralow-power or even zero-power Internet of Things (IoT) devices in the sixth-generation mobile networks. In this article, we propose an energy-aware symbiotic transmission in a reconfigurable intelligent surface (RIS) enhanced SR system, in which an IoT network embeds its own data passively over cellular downlink signals by backscattering. The base station (BS) serves multiple cellular users (CUs) through time division multiple access (TDMA) and each IoT device is associated with one CU. We formulate the BS’s energy minimization problem subject to the constraints of the minimum amounts of transmission bits required by IoT devices and CUs. The user association, the active transmit beamforming at the BS, the passive reflecting beamforming at the RIS, and the frame division policy are jointly optimized. The formulated problem is a mixed integer nonlinear programming (MINLP) problem, which is NP-hard and nonconvex. We decouple the problem and solve the subproblems alternatively. First, we design a many-to-one swap-matching-based algorithm to solve the user association subproblem. Then, we develop a joint cooperative beamforming and time allocation optimization algorithm based on the alternative optimization (AO) and semidefinite relaxation (SDR) techniques. Simulation results show that the proposed joint user association and cooperative beamforming algorithm brings significant performance gain in reducing the energy consumption of the BS with fast convergence speed compared with other schemes.

UAV-assisted task offloading for IoT in smart buildings and environment via deep reinforcement learning

The online 3D bin packing problem(3D-BPP) is widely used in the logistics industry and is of great practical significance for promoting the intelligent transformation of the industry. The heuristic algorithm relies too much on manual experience to formulate more perfect packing rules. In recent years, many scholars solve 3D-BPP via deep reinforcement learning(DRL) algorithms. However, they ignore many skills used in manual packing, one of the most important skill is workers put the item aside if the item is packed improperly. Inspired by this skill, we propose a DRL algorithm with a buffer zone. Firstly, we define the wasted space and the buffer zone. And then, we integrate them into the DRL algorithm framework. Importantly, we compare the bin utilization with di erent thresholds of wasted space and di erent buffer zone sizes. Experimental results show that our algorithm outperforms existing heuristic algorithms and DRL algorithms.

Much human and computational effort has aimed to improve how deep reinforcement learning (DRL) algorithms perform on benchmarks such as the Atari Learning Environment. Comparatively less effort has focused on understanding what has been learned by such methods, and investigating and comparing the representations learned by different families of DRL algorithms. Sources of friction include the onerous computational requirements, and general logistical and architectural complications for running DRL algorithms at scale. We lessen this friction, by (1) training several algorithms at scale and releasing trained models, (2) integrating with a previous DRL model release, and (3) releasing code that makes it easy for anyone to load, visualize, and analyze such models. This paper introduces the Atari Zoo framework, which contains models trained across benchmark Atari games, in an easy-to-use format, as well as code that implements common modes of analysis and connects such models to a popular neural network visualization library. Further, to demonstrate the potential of this dataset and software package, we show initial quantitative and qualitative comparisons between the performance and representations of several DRL algorithms, highlighting interesting and previously unknown distinctions between them.

Human.computer interaction (HCI) has become an important research area for improving the user experience on Internet of Things (IoT) devices. In particular, gesture recognition and dailyactivity recognition have attracted the interest of numerous researchers. Human motions have been predicted by analyzing accelerometer data from which features were extracted to be classified into a specific activity. However, due to the memory limitations of IoT devices, it is hard to utilize all the raw data from an accelerometer sensor. This paper proposes a deep reinforcement learning algorithm to recognize human arm movements using a commercial wearable device, the Myo armband. Agents learn the patterns that are the acceleration data of human motion. In addition, using raw accelerometer sensor data without feature extraction could make an end-to-end structure. In order to demonstrate the performance of the proposed method, a deep neural network (DNN) and a deep reinforcement learning algorithm are compared. As a result, a deep reinforcement learning agent yielded accuracy similar to a DNN using less data, and the agent could learn time-series human motion acceleration data.

Maintaining adequate energy in low-powered Internet of Things (IoT) nodes is crucial for the development of several applications like smart homes, autonomous industries, etc. These IoT nodes exploit adaptive duty cycling techniques for the efficient utilization of energy resources. However, such adaptive duty cycling of IoT nodes results in their asynchronous operations thereby inducing energy holes in the network. These energy holes lead to information loss and poor quality of services of IoT networks. In this regard, energy harvesting using Mobile Energy Transmitters (MET) can improve the lifetime of an IoT network. In this work, we are introducing a metric named Age of Charging (AoC) metric to quantify the repetitive charging of power deficit IoT nodes. Energy-efficient scheduling of MET is proposed to minimize the expected average AoC such that the energy harvested by IoT nodes is maximized. In this regard, the optimization problem is first remodeled into a Markov decision process. Subsequently, a deep reinforcement learning algorithm is developed based upon the twin delayed deep deterministic policy gradient scheme for energy-efficient scheduling of MET in asynchronous IoT networks. The simulation results indicate that the proposed algorithm outperforms the conventional Deep Q-networks and soft-actor-critic algorithms. These results motivate the usage of MET-aided energy harvesting in self-sustaining IoT networks.

The growing number of complex and heterogeneous Internet of Things (IoT) applications has imposed a high demand for scarce communications and computing resources. To meet this stringent requirement, it is desirable to develop large-scale highly adaptive online resource allocation strategies to streamline existing network operations. Deep reinforcement learning (DRL), which combines the merits of reinforcement learning and deep learning, is capable of addressing complex decision-making tasks, thus enabling efficient online resource allocation. In this article, we present a DRL-based resource allocation framework. We begin a discussion on DRL basics and review its several recent applications. Then, we develop two new DRL algorithms that facilitate unlocking the potential of DRL and offer viable solutions to many more complex resource allocation problems. The first one tackles an optimization problem exposed to mixed (discrete and continuous) action spaces and bound by a number of highly non-linear quality-of-service (QoS) constraints. The second one extends the single-agent DRL to a more challenging multi-agent DRL by introducing a novel semi-distributed architecture. Finally, we discuss the challenges and future visions of applying DRL to real-world IoT networks.

Aiming at the task unloading mode in cloud computing environment, the task unloading problem for IoT devices is studied. Through theoretical analysis, we can know that in the task unloading problem, it is usually contradictory to improve the utilization of cloud resources and reduce the task delay. In order to solve this problem, a task unloading scheme for Internet of things devices using deep reinforcement learning algorithm is proposed. The deep reinforcement learning algorithm is used to model the task unloading problem. The return value with weight is introduced into the algorithm, and the utilization rate of cloud resources and the delay of unloading task are weighed by adjusting the return value of the weight. First of all, the improved k-means clustering algorithm with weighted density is used to cluster the physical machines. The physical machines of each cluster have similar bandwidth and task waiting time. Then, deep reinforcement learning is used to select the best physical machine cluster from the current unloading tasks. Finally, the improved PSO algorithm is used to select the optimal physical machine from the optimal cluster, and Pareto is used to improve the convergence speed. Experimental results show that compared with the traditional method, the proposed algorithm has a good performance, and can achieve the goal of increasing the utilization of physical machine resources and reducing task delay.

In recent years, deep reinforcement learning (DRL) has made impressive achievements in many fields. However, existing DRL algorithms usually require a large amount of exploration to obtain a good action policy. In addition, in many complex situations, the reward function cannot be well designed to meet task requirements. These two problems will make it difficult for DRL to learn a good action policy within a relatively short period. The use of expert data can provide effective guidance and avoid unnecessary exploration. This study proposes a deep imitation reinforcement learning (DIRL) algorithm that uses a certain amount of expert demonstration data to speed up the training of DRL. In the proposed method, the learning agent imitates the expert's action policy by learning from demonstration data. After imitation learning, DRL is used to optimise the action policy in a self‐learning way. By experimental comparison on a video game called the Mario racing game, it is shown that the proposed DIRL algorithm with expert demonstration data can obtain much better performance than previous DRL algorithms without expert guidance.

In a deregulated environment, the renewable energy producers will face the challenge of how to increase their revenues under uncertainties of power generation and time-varying electricity price. In traditional power network scheduling, prediction and optimization are two independent processes, which easily leads to information loss and modeling error. To deal with the uncertainty and realize an end-to-end controller, this article proposes an energy storage system control model (ESSCM) in the scene of the combined wind-solar storage system. The proposed ESSCM using deep reinforcement learning (DRL) algorithm is trained by interacting with the massive environment of a power grid without requiring the assumption on the uncertainties. It learns from scratch to realize the coordination operation of with wind power and photovoltaic power in a combined system, further maximize the benefits of the combined system in the electricity market. One state-of-the-art DRL algorithm, namely double deep Q-network, is used to formulate the proposed ESSCM. Numerical results illustrate that the proposed approach can effectively accommodate the uncertainty and bring high revenues to the combined system.

Point of Interest (POI) recommendation is one of the important means for businesses to fully understand user preferences and meet their personalized needs, laying a solid foundation for the development of e-commerce and social networks. However, traditional social network POI recommendation algorithms suffer from various problems such as low accuracy and low recall. Therefore, a social network POI recommendation algorithm using the Internet of Things (IoT) and deep reinforcement learning (DRL) is proposed. First, the overall framework of the POI recommendation algorithm is designed by integrating IoT technology and DRL algorithm. Second, under the support of this framework, IoT technology is utilized to deeply explore users' personalized preferences for POI recommendation, analyze the internal rules of user check-in behavior and integrate multiple data sources. Finally, a DRL algorithm is used to construct the recommendation model. Multiple data sources are used as input to the model, based on which the check-in probability is calculated to generate the POI recommendation list and complete the design of the social network POI recommendation algorithm. Experimental results show that the accuracy of the proposed algorithm for social network POI recommendation has a maximum value of 98%, the maximum recall is 97% and the root mean square error is low. The recommendation time is short, and the maximum recommendation quality is 0.92, indicating that the recommendation effect of the proposed algorithm is better. By applying this method to the e-commerce field, businesses can fully utilize POI recommendation to recommend products and services that are suitable for users, thus promoting the development of the social economy.

Deep Reinforcement Learning (DRL) algorithm provides an effective and powerful way for the computer learning to make decisions according to what the computer sees, the most famous application of DRL is training the computer to play computer games, such as the computer Go and Atari Games. In this paper, we present that the DRL can used to improve the classical PID controller by training an adaptive PID controller with its parameters change according to the changing state, we name our new controller as DRPID, the deep reinforcement learning algorithm that we used to train the PID controller in this paper is deep deterministic policy gradient, which is very suitable to solve continue action control problem. We test our DRPID algorithm by training it to control an inverted pendulum in the OpenAI gym simulation environment, and we found DRPID controller works very well, outperforms the common PID controller with fixed parameters, by a great margin.

Reinforcement Learning (RL) has become one of the most powerful methods in Artificial Intelligence (AI) which allows a system to learn as it is engaged in an interaction with its environment and to maximize its actions by performing a trial and error process. Although classical RL algorithms can give us a fundamental idea on RL, like the Q-Learning, its usefulness wanes in high-dimensional or continuous-state spaces. To overcome these shortcomings, Deep Reinforcement Learning (DRL) algorithms including Deep Q-Networks (DQN) build on neural networks to learn action-value functions, allowing horizontal scalability and more efficient learning. This paper reports on the experimental comparison of Q-Learning and DQN based on the environment CartPole-v1. Both of the algorithms were trained in the controlled environment and tested under various metrics that include cumulative rewards, success rate, convergence speed, safety (failure rate), and path efficiency. Findings indicate that Q-Learning moves towards moderate stability with slower convergence but DQN takes shorter learning time, is more reliable and performs better. Such results indicate the ground-breaking potential of the DRL in the progress of AI toward transcending classical RL restrictions. Future research topics encompass implementation of more challenging experimental settings, a test of other DRL algorithms (e.g., PPO, SAC), and generalization, scalability and safety in real-world environments.

As vehicular technology advances, intelligent vehicles generate numerous computation‐intensive tasks, challenging the computational resources of both the vehicles and the Internet of Vehicles (IoV). Traditional IoV struggles with fixed network structures and limited scalability, unable to meet the growing computational demands and next‐generation mobile communication technologies. In congested areas, near‐end Mobile Edge Computing (MEC) resources are often overtaxed, while far‐end MEC servers are underused, resulting in poor service quality. A novel network framework utilizing sixth‐generation mobile communication (6G) and digital twin technologies, combined with task migration, promises to alleviate these inefficiencies. To address these challenges, a task migration and re‐offloading model based on task attribute classification is introduced, employing a hybrid deep reinforcement learning (DRL) algorithm—Dueling Double Q Network DDPG (QDPG). This algorithm merges the strengths of the Deep Deterministic Policy Gradient (DDPG) and the Dueling Double Deep Q‐Network (D3QN), effectively handling continuous and discrete action domains to optimize task migration and re‐offloading in IoV. The inclusion of the Mini Batch K‐Means algorithm enhances learning efficiency and optimization in the DRL algorithm. Experimental results show that QDPG significantly boosts task efficiency and computational performance, providing a robust solution for resource allocation in IoV.

Internet of Things (IoT) enables intelligent services varying with the complex and realtime environment to achieve network benefits, where Network Function Virtualization (NFV) can dynamically provide Virtualized Network Functions (VNFs) for IoT devices. In NFV-enabled IoT architecture, a service function chain (SFC) consists of an ordered set of virtual network functions (VNFs). However, the energy consumption of the VNF migration and SFC reconfiguration is one major issue owing to the dynamic characteristic of IoT network. In this paper, we propose a new paradigm digital twin (DT) to create the virtual twin of physical objects in IoT network, then, we formalize the problem as a mathematical model, which aims to minimize the energy consumption. To this end, we prove this problem is NP-hard and propose an algorithm Bidirectional Gated Recurrent Unit (Bi-GRU) based on federated learning to predict the resource requirement. Further more, according to the prediction result, which utilizing the deep reinforcement learning (DRL) algorithm for decision making of the VNF migration. Simulation results show that our proposed method can effectively reduce the number of VNFs to be migrated and economize the energy consumption of the digital twin IoT network.