Multi-Objective Policy Gradients with Topological Constraints

An optimal solutions-guided deep reinforcement learning approach for online energy storage control

Multi-agent quantum-inspired deep reinforcement learning for real-time distributed generation control of 100% renewable energy systems

Energy efficient task scheduling based on deep reinforcement learning in cloud environment: A specialized review

The IEEE 802.11ah standard is introduced to address the growing scale of internet of things (IoT) applications. To reduce contention and enhance energy efficiency in the system, the restricted access window (RAW) mechanism is introduced in the medium access control (MAC) layer to manage the significant number of stations accessing the network. However, to achieve optimized network performance, it is necessary to appropriately determine the RAW parameters, including the number of RAW groups, the number of slots in each RAW, and the duration of each slot. In this paper, we optimize the configuration of RAW parameters in the uplink IEEE 802.11ah-based IoT network. To improve network throughput, we analyze and establish a RAW parameters optimization problem. To effectively cope with the complex and dynamic network conditions, we propose a deep reinforcement learning (DRL) approach to determine the preferable RAW parameters to optimize network throughput. To enhance learning efficiency and stability, we employ the proximal policy optimization (PPO) algorithm. We construct network environments with periodic and random traffic in an NS-3 simulator to validate the performance of the proposed PPO-based RAW parameters optimization algorithm. The simulation results reveal that using the PPO-based DRL algorithm, optimized RAW parameters can be obtained under different network conditions, and network throughput can be improved significantly.

Improving the interpretability of deep reinforcement learning in urban drainage system operation

Optimal scheduling of island integrated energy systems considering multi-uncertainties and hydrothermal simultaneous transmission: A deep reinforcement learning approach

Proton pencil beam scanning (PBS) treatment planning for head and neck (H&N) cancers is a time-consuming and experience-demanding task where a large number of potentially conflicting planning objectives are involved. Deep reinforcement learning (DRL) has recently been introduced to the planning processes of intensity-modulated radiation therapy (IMRT) and brachytherapy for prostate, lung, and cervical cancers. However, existing DRL planning models are built upon the Q-learning framework and rely on weighted linear combinations of clinical metrics for reward calculation. These approaches suffer from poor scalability and flexibility, that is, they are only capable of adjusting a limited number of planning objectives in discrete action spaces and therefore fail to generalize to more complex planning problems. Here we propose an automatic treatment planning model using the proximal policy optimization (PPO) algorithm in the policy gradient framework of DRL and a dose distribution-based reward function for proton PBS treatment planning of H&N cancers. The planning process is formulated as an optimization problem. A set of empirical rules is used to create auxiliary planning structures from target volumes and organs-at-risk (OARs), along with their associated planning objectives. Special attention is given to overlapping structures with potentially conflicting objectives. These planning objectives are fed into an in-house optimization engine to generate the spot monitor unit (MU) values. A decision-making policy network trained using PPO is developed to iteratively adjust the involved planning objective parameters. The policy network predicts actions in a continuous action space and guides the treatment planning system to refine the PBS treatment plans using a novel dose distribution-based reward function. A total of 34 H&N patients (30 for training and 4 for test) and 26 liver patients (20 for training, 6 for test) are included in this study to train and verify the effectiveness and generalizability of the proposed method. Proton H&N treatment plans generated by the model show improved OAR sparing with equal or superior target coverage when compared with human-generated plans. Moreover, additional experiments on liver cancer demonstrate that the proposed method can be successfully generalized to other treatmentsites. The automatic treatment planning model can generate complex H&N plans with quality comparable or superior to those produced by experienced human planners. Compared with existing works, our method is capable of handling more planning objectives in continuous action spaces. To the best of our knowledge, this is the first DRL-based automatic treatment planning model capable of achieving human-level performance for H&Ncancers.

Millimeter-wave (mmWave) communication is a key technology of fifth generation wireless systems to achieve the expected 1000x data rate. With large bandwidth at the mmWave band, the link capacity between users and base stations (BSs) can be much higher compared to sub-6 GHz wireless systems. Meanwhile, due to the high cost of infrastructure upgrade, it would be difficult for operators to drastically enhance the capacity of backhaul links between mmWave BSs and the core network. As a result, the data rate provided by backhaul may not be sufficient to support all mmWave links; hence, the backhaul connection becomes the new bottleneck. On the other hand, as mmWave channels are subject to random blockage, the data rates of mmWave users significantly vary over time. With limited backhaul capacity and highly dynamic data rates of users, how to allocate backhaul resource to each user remains a challenge for mmWave systems. In this article, we present a deep reinforcement learning (DRL) approach to address this challenge. By learning the blockage pattern, the system dynamics can be captured and predicted, resulting in efficient utilization of backhaul resource. We begin with a discussion on DRL and its application in wireless systems. We then investigate the problem of backhaul resource allocation and present the DRL-based solution. Finally, we discuss open problems for future research and conclude this article.

This paper investigates the covert communications via spectrum allocations in a cellular-enabled unmanned aerial vehicle (UAV) network consisting of a base station (BS), UAVs, ground users (GUs), and a warden, where warden attempts to detect the transmission from a target GU to a UAV receiver. We formulate the spectrum allocation as an optimization problem with the constraints of covertness performance requirement and the qualities of service (QoS) of cellular communications. This is a nonlinear and nonconvex problem, which is generally challenging to be solved. Thus, we propose a deep reinforcement learning (DRL) approach to solve it. Under such an approach, we first model the multi-agent DRL environment in such networks. Then we define the state, action, reward and interaction mechanism of the DRL environment. Finally, a DRL algorithm is presented for learning the optimal policy of spectrum allocation.

As deep space exploration missions grow in complexity, efficient data transfer in interplanetary networks (IPNs) becomes paramount. However, the vast distances, limited bandwidth, and dynamic nature of IPNs pose significant challenges for the routing and data scheduling of interplanetary data transfers (IP-DTs). To address these challenges, this work proposes a novel distributed, graph neural network (GNN) based multiagent deep reinforcement learning (DRL) approach that can jointly optimize the routing and scheduling of IP-DTs. Our proposal is based on the proximal policy optimization (PPO) framework along with the graph attention networks (GATs). We make the DRL agents for IPN nodes in each subnetwork around a celestial body learn and operate independently, for making intelligent routing and scheduling decisions to properly tradeoff between average end-to-end (E2E) latency and delivery ratio of IP-DTs while ensuring good scalability. Extensive simulations confirm that our proposal handles the routing and scheduling of IP-DTs much better than existing benchmarks. Further, by modifying the interplanetary overlay network (ION) software platform developed by NASA, we build a semi-physical IPN emulator based on Raspberry Pi boards, implement our proposal in it, and conduct experiments with real data transfers between IPN nodes. Experimental results verify that our proposal can work for practical IPNs without causing excessive overheads and prove its advantages.

Extracting the cerebral anterior vessel tree of patients with an intracranial large vessel occlusion(LVO) is relevant to investigate potential biomarkers that can contribute to treatment decision making. The purpose of our work is to develop a method that can achieve this from routinely acquired computed tomography angiography(CTA)and computed tomography perfusion(CTP)images. To this end, we regard the anterior vessel tree as a set of bifurcations and connected centerlines. The method consists of a proximal policy optimization(PPO) based deep reinforcement learning (DRL) approach for tracking centerlines, a convolutional neural network based bifurcation detector, and a breadth-first vessel tree construction approach taking the tracking and bifurcation detection results as input. We experimentally determine the added values of various components of the tracker. Both DRL vessel tracking and CNN bifurcation detection were assessed in a cross validation experiment using 115 subjects. The anterior vessel tree formation was evaluated on an independent test set of 25 subjects, and compared to interobserver variation on a small subset of images. The DRL tracking result achieves a median overlapping rate until the first error (1.8mm off the reference standard) of 100, [46, 100]% on 8032 vessels over 115 subjects. The bifurcation detector reaches an average recall and precision of 76% and 87% respectively during the vessel tree formation process. The final vessel tree formation achieves a median recall of 68% and precision of 70%, which is in line with the interobserver agreement.

The modern network is equipped with many service functions to acquire high-quality service. The emergence of network function virtualization (NFV) provides a convenient way to accomplish the network services in the form of virtual network function (VNF) and also makes the scheduling solution of VNFs flexible. The VNFs can be deployed on commodity servers as software processes. Besides, multiple VNFs are chained in a specified order as a service function chain (SFC) to serve a given flow, increasing the scheduling difficulty to minimize the average flow completion time. In this paper, we study the problem of scheduling multiple SFCs with the constraint of different resource limitations in various commodity servers. This problem is typically formulated as an Integer Linear Programming (ILP) problem, which is NP-hard. To well tackle this problem, we propose a deep reinforcement learning (DRL) approach. It involves multi-step decision making, which can be naturally transformed into a DRL problem. We design specific reward and state representations for such a multi-resource scheduling problem. We also consider how to use DRL to handle online requests of SFCs. The experiment results demonstrate that the DRL approach can significantly reduce the average completion time of a set of SFC and achieves a cost saving of 39.94% against the benchmark method.

Dynamic pricing, which aims to dynamically adjust the charging price in a timely fashion to unlock the flexibility of electric vehicle (EV) customers, has been extensively studied with the rapid development of charging technologies. Many existing works on dynamic pricing focus on maximizing the social welfare of charging service providers and EV customers. Cases of high-dimensional charging environments, which are often encountered with the rapid growth of EV market penetration, have been rarely considered to date. This article proposes a new dynamic pricing framework for EV charging stations that can offer multiple charging options to customers over a finite-time horizon. The charging price can be dynamically adjusted to maximize the quality of service (QoS) with a differentiated service requirement level (SRL) whenever the arrival rates and queuing system capacities of the charging systems are given at the end of a time period. The dynamic pricing problem is formulated as a finite-discrete horizon Markov decision process (MDP) with a mixed state space. A customized deep reinforcement learning (DRL) approach is employed to solve the examined EV dynamic pricing problem. The simulation results demonstrate the effectiveness of the proposed method.

Deep reinforcement learning (DRL) has many achievements in playing games and robotic control. However, transfer learning, a normal way of training a deep neural network for computer vision tasks, cannot be directly applied to DRL. This is an issue in the field of DRL. Many methods are proposed to address this problem. In this paper, a new method based on the proximal policy optimization (PPO) algorithm for learning similar tasks is proposed. The method uses a function to transform the trajectory from the previous task to the current task, both of those are similar. As shown in the result, our method can bring a more stable performance to our model at the early stage of training with compare to the model trained by using PPO without our method.

The object of the study is a mobile walking robot with two or more movable articulated limbs. The concept of a "critical condition" is introduced, in which the mechanism balances on the verge of falling (but does not fall) or there is a possibility of damage to mechanical components due to the generation of unacceptable joint angles. The subject of the study is a general algorithm for the elimination of critical conditions, which provides the possibility of training an agent based on a deep reinforcement learning algorithm directly on a real robot, without the risk of damaging its mechanisms and interrupting the process of interaction with the environment to restore a stable state. The purpose of this work is to develop a general algorithm for the elimination of critical conditions in the context of adaptive control of a walking robot based on deep learning algorithms with reinforcement. A comparison was made between the proposed and standard methods of applying deep OP on a real robot. The experiments were conducted on 6,000 episodes, with a dimension of 300 steps each. The following quality metrics were selected for evaluation: the percentage of episodes without an actual fall, the percentage of fully completed episodes, and the maximum episode length. The algorithm is based on the concept of "critical condition" and uses the following principles and methods: the "trial and error" method, the feedback principle, holding the projection of the center of gravity point in the area of the polygon formed by the points of contact of the limbs with the work surface, which ensures the balancing of the structure and allows you to determine the boundary areas in which the robot is still stable. The scientific novelty of the work lies in the proposed approach, which allows an intelligent agent to control a physical robot "directly", without pre-configuration in a simulation environment with subsequent transfer implementation. The proposed algorithm is not aimed at improving the agent's performance, but is intended to provide greater autonomy in the learning process of the robot, directly in the hardware. The basic idea is to immediately respond to a critical condition in the form of the fastest sequential return to a certain number of steps back along the decision-making trajectory, ensuring that the agent remains in a stable and safe state at all times. The method of proximal policy optimization (PPO) was used as a method of deep reinforcement learning. As a result of the comparative analysis, the proposed algorithm demonstrated a hundredfold increase in the stability of the mechanism.