AgentGraph: Toward Universal Dialogue Management With Structured Deep Reinforcement Learning

Energy efficient task scheduling based on deep reinforcement learning in cloud environment: A specialized review

An optimal solutions-guided deep reinforcement learning approach for online energy storage control

Optimal scheduling of island integrated energy systems considering multi-uncertainties and hydrothermal simultaneous transmission: A deep reinforcement learning approach

Millimeter-wave (mmWave) communication is a key technology of fifth generation wireless systems to achieve the expected 1000x data rate. With large bandwidth at the mmWave band, the link capacity between users and base stations (BSs) can be much higher compared to sub-6 GHz wireless systems. Meanwhile, due to the high cost of infrastructure upgrade, it would be difficult for operators to drastically enhance the capacity of backhaul links between mmWave BSs and the core network. As a result, the data rate provided by backhaul may not be sufficient to support all mmWave links; hence, the backhaul connection becomes the new bottleneck. On the other hand, as mmWave channels are subject to random blockage, the data rates of mmWave users significantly vary over time. With limited backhaul capacity and highly dynamic data rates of users, how to allocate backhaul resource to each user remains a challenge for mmWave systems. In this article, we present a deep reinforcement learning (DRL) approach to address this challenge. By learning the blockage pattern, the system dynamics can be captured and predicted, resulting in efficient utilization of backhaul resource. We begin with a discussion on DRL and its application in wireless systems. We then investigate the problem of backhaul resource allocation and present the DRL-based solution. Finally, we discuss open problems for future research and conclude this article.

This paper investigates the covert communications via spectrum allocations in a cellular-enabled unmanned aerial vehicle (UAV) network consisting of a base station (BS), UAVs, ground users (GUs), and a warden, where warden attempts to detect the transmission from a target GU to a UAV receiver. We formulate the spectrum allocation as an optimization problem with the constraints of covertness performance requirement and the qualities of service (QoS) of cellular communications. This is a nonlinear and nonconvex problem, which is generally challenging to be solved. Thus, we propose a deep reinforcement learning (DRL) approach to solve it. Under such an approach, we first model the multi-agent DRL environment in such networks. Then we define the state, action, reward and interaction mechanism of the DRL environment. Finally, a DRL algorithm is presented for learning the optimal policy of spectrum allocation.

Multi-agent quantum-inspired deep reinforcement learning for real-time distributed generation control of 100% renewable energy systems

Dynamic pricing, which aims to dynamically adjust the charging price in a timely fashion to unlock the flexibility of electric vehicle (EV) customers, has been extensively studied with the rapid development of charging technologies. Many existing works on dynamic pricing focus on maximizing the social welfare of charging service providers and EV customers. Cases of high-dimensional charging environments, which are often encountered with the rapid growth of EV market penetration, have been rarely considered to date. This article proposes a new dynamic pricing framework for EV charging stations that can offer multiple charging options to customers over a finite-time horizon. The charging price can be dynamically adjusted to maximize the quality of service (QoS) with a differentiated service requirement level (SRL) whenever the arrival rates and queuing system capacities of the charging systems are given at the end of a time period. The dynamic pricing problem is formulated as a finite-discrete horizon Markov decision process (MDP) with a mixed state space. A customized deep reinforcement learning (DRL) approach is employed to solve the examined EV dynamic pricing problem. The simulation results demonstrate the effectiveness of the proposed method.

Optimal dispatch of an energy hub with compressed air energy storage: A safe reinforcement learning approach

Efficient parking distribution is crucial for urban traffic management; nevertheless, variable demand and spatial disparities raise considerable obstacles. Current research emphasizes local optimization but neglects the fundamental challenges of real-time parking allocation, resulting in inefficiencies within intricate metropolitan settings. This research delineates two key issues: (1) A dynamic imbalance between supply and demand, characterized by considerable fluctuations in parking demand over time and across different locations, rendering static allocation solutions inefficient; (2) spatial resource optimization, aimed at maximizing the efficiency of limited parking spots to improve overall system performance and user satisfaction. We present a Multi-Agent Reinforcement Learning (MARL) framework that incorporates adaptive optimization and intelligent collaboration for dynamic parking allocation to tackle these difficulties. A reinforcement learning-driven temporal decision mechanism modifies parking assignments according to real-time data, whilst a Graph Neural Network (GNN)-based spatial model elucidates inter-parking relationships to enhance allocation efficiency. Experiments utilizing actual parking data from Melbourne illustrate that Multi-Agent Reinforcement Learning (MARL) substantially surpasses conventional methods (FIFO, SIRO) in managing demand variability and optimizing resource distribution. A thorough quantitative investigation confirms the strength and flexibility of the suggested method in various urban contexts.

The modern network is equipped with many service functions to acquire high-quality service. The emergence of network function virtualization (NFV) provides a convenient way to accomplish the network services in the form of virtual network function (VNF) and also makes the scheduling solution of VNFs flexible. The VNFs can be deployed on commodity servers as software processes. Besides, multiple VNFs are chained in a specified order as a service function chain (SFC) to serve a given flow, increasing the scheduling difficulty to minimize the average flow completion time. In this paper, we study the problem of scheduling multiple SFCs with the constraint of different resource limitations in various commodity servers. This problem is typically formulated as an Integer Linear Programming (ILP) problem, which is NP-hard. To well tackle this problem, we propose a deep reinforcement learning (DRL) approach. It involves multi-step decision making, which can be naturally transformed into a DRL problem. We design specific reward and state representations for such a multi-resource scheduling problem. We also consider how to use DRL to handle online requests of SFCs. The experiment results demonstrate that the DRL approach can significantly reduce the average completion time of a set of SFC and achieves a cost saving of 39.94% against the benchmark method.

As deep space exploration missions grow in complexity, efficient data transfer in interplanetary networks (IPNs) becomes paramount. However, the vast distances, limited bandwidth, and dynamic nature of IPNs pose significant challenges for the routing and data scheduling of interplanetary data transfers (IP-DTs). To address these challenges, this work proposes a novel distributed, graph neural network (GNN) based multiagent deep reinforcement learning (DRL) approach that can jointly optimize the routing and scheduling of IP-DTs. Our proposal is based on the proximal policy optimization (PPO) framework along with the graph attention networks (GATs). We make the DRL agents for IPN nodes in each subnetwork around a celestial body learn and operate independently, for making intelligent routing and scheduling decisions to properly tradeoff between average end-to-end (E2E) latency and delivery ratio of IP-DTs while ensuring good scalability. Extensive simulations confirm that our proposal handles the routing and scheduling of IP-DTs much better than existing benchmarks. Further, by modifying the interplanetary overlay network (ION) software platform developed by NASA, we build a semi-physical IPN emulator based on Raspberry Pi boards, implement our proposal in it, and conduct experiments with real data transfers between IPN nodes. Experimental results verify that our proposal can work for practical IPNs without causing excessive overheads and prove its advantages.

With the increasing real-time demand in the internet era, especially dynamic requests for last-mile delivery, route planning is becoming more computationally expensive than ever before. Many supply chains (SCs) choose the joint distribution of multiple depots to reduce transportation costs and delivery times. However, providing real-time and high-quality solutions for such complex routing problems remains challenging. Current solution methods like mathematical programming and heuristics suffer from scalability issues and long computation times. In contrast, artificial intelligence, especially Deep Reinforcement Learning (DRL), provides a general-purpose framework for sequential decision-making that has produced good results for many challenging real-life problems. However, applying DRL to route multiple vehicles is nontrivial, as the joint distribution requires an effective method that facilitates collaboration and communication among all agents while they carry out the delivery mission. In this research, a collaborative Multi-Agent Deep Reinforcement Learning (MADRL) approach is proposed for routing multiple vehicles in the SC. The proposed MADRL model leverages the power of two frameworks, deep learning and reinforcement learning, to generate routing policies for all agents in real time. Experimental results show the ability of the proposed learning model to obtain fast and quality solutions for complex delivery problems. Furthermore, the generalization ability of MADRL is also validated by testing the well-trained model on different scale problems.

Unmanned aerial vehicles (UAVs) have emerged as a promising platform for supporting integrated sensing, computing, and communication (ISCC) functionality in Internet of Things (IoT) applications. This paper investigates a UAV-enabled ISCC network, where the UAV performs radar sensing and onboard edge computing with the computational assistance of ground access points (APs). Given the limited onboard energy, ensuring energy-efficient operation of UAVs is crucial to support the long-term sustainability of network performance. In this paper, we define computing efficiency as the ratio between the total number of successfully processed computational bits and the overall UAV energy consumption, under the constraint of a required sensing threshold. To maximize this performance metric, this paper jointly optimizes the beamforming vector, the CPU frequency, and the trajectory of the UAV. This optimization problem is modeled as a Markov decision process (MDP) and solved using a deep reinforcement learning (DRL) approach based on a memory mechanism. Specifically, a long short-term memory (LSTM) and twin delayed deep deterministic policy gradient (TD3)-based trajectory design and resource allocation (LTTDRA) algorithm is proposed. LSTM units are integrated into the actor and critic to effectively capture the temporal correlations in dynamic environments, thereby enhancing policy stability and accelerating algorithm convergence. The reward function is meticulously designed to alleviate sparse-penalty effects and learn high-performance strategies in complex environments with multiple constraints. Extensive simulations are conducted under various settings and network scenarios, and the results consistently indicate that the proposed approach substantially outperforms the baseline schemes.

Online panoramic video has recently gained enormous popularity. Tile-based adaptive streaming is a promising paradigm to deliver a panoramic video for the sake of bandwidth saving. Nevertheless, it is challenging to accurately predict user’s field of view (FoV) and deliver the optimal bitrate due to the dynamic user behavior and time-varying network conditions. In this paper, we propose a novel approach of deep reinforcement learning based predictive panoramic video delivering. Specifically, a carefully-devised long short-term memory (LSTM) model is used to predict the FoV in the next few seconds. Our quality adaptation policy is based on a deep reinforcement learning (DRL) agent, which is able to intelligently adapt its bitrate selection policy tailored to the dynamic environments. To validate the effectiveness we have implemented a prototype of this system. With the integrated DRL algorithm Rainbow, we have achieved a superior performance in terms of the quality of experience (QoE) score, which outperforms existing panoramic video streaming frameworks.

Decision-making of autonomous vehicles in interactions with jaywalkers: A risk-aware deep reinforcement learning approach