Learning how to Active Learn: A Deep Reinforcement Learning Approach

An optimal solutions-guided deep reinforcement learning approach for online energy storage control

Multi-agent quantum-inspired deep reinforcement learning for real-time distributed generation control of 100% renewable energy systems

Energy efficient task scheduling based on deep reinforcement learning in cloud environment: A specialized review

In traditional active learning, one of the most well-known strategies is to select the most uncertain data for annotation. By doing that, we acquire as most as we can obtain from the labeling oracle so that the training in the next run can be much more effective than the one from this run once the informative labeled data are added to training. The strategy however, may not be suitable when deep learning become one of the dominant modeling techniques. Deep learning is notorious for its failure to achieve a certain degree of effectiveness under the adversarial environment. Often we see the sparsity in deep learning training space which gives us a result with low confidence. Moreover, to have some adversarial inputs to fool the deep learners, we should have an active learning strategy that can deal with the aforementioned difficulties. We propose a novel Active Learning strategy based on Virtual Adversarial Training (VAT) and the computation of local distributional roughness (LDR). Instead of selecting the data that are closest to the decision boundaries, we select the data that are located in a place with rough enough surface if measured by the posterior probability. The proposed strategy called Virtual Adversarial Active Learning (VAAL) can help us to find the data with rough surface, reshape the model with smooth posterior distribution output thanks to the active learning framework. Moreover, we shall prefer the labeling data that own enough confidence once they are annotated from an oracle. In VAAL, we have the VAT that can not only be used as a regularization term but also helps us effectively and actively choose the valuable samples for active learning labeling. Experiment results show that the proposed VAAL strategy can guide the convolutional networks model converging efficiently on several well-known datasets.

In Reinforcement Learning, a category of machine learning, learning is based on evaluative feedbacks without any supervised signals. The paper presents work aimed to understand the deep reinforcement learning approaches to creating such intelligent agents, by reproducing existing research and comparing their results. The project uses the Atari 2600 game called Breakout, in which the agent will learn control policies using deep reinforcement learning approaches to achieve a high score. The project explores two deep reinforcement learning approaches, Asynchronous Advantage actor-critic and Deep Q-Learning, both proposed by the DeepMind team, to train intelligent agents that can interact with an environment with automatic feature engineering thus requiring minimal domain knowledge.

Optimal scheduling of island integrated energy systems considering multi-uncertainties and hydrothermal simultaneous transmission: A deep reinforcement learning approach

Millimeter-wave (mmWave) communication is a key technology of fifth generation wireless systems to achieve the expected 1000x data rate. With large bandwidth at the mmWave band, the link capacity between users and base stations (BSs) can be much higher compared to sub-6 GHz wireless systems. Meanwhile, due to the high cost of infrastructure upgrade, it would be difficult for operators to drastically enhance the capacity of backhaul links between mmWave BSs and the core network. As a result, the data rate provided by backhaul may not be sufficient to support all mmWave links; hence, the backhaul connection becomes the new bottleneck. On the other hand, as mmWave channels are subject to random blockage, the data rates of mmWave users significantly vary over time. With limited backhaul capacity and highly dynamic data rates of users, how to allocate backhaul resource to each user remains a challenge for mmWave systems. In this article, we present a deep reinforcement learning (DRL) approach to address this challenge. By learning the blockage pattern, the system dynamics can be captured and predicted, resulting in efficient utilization of backhaul resource. We begin with a discussion on DRL and its application in wireless systems. We then investigate the problem of backhaul resource allocation and present the DRL-based solution. Finally, we discuss open problems for future research and conclude this article.

Dynamic pricing, which aims to dynamically adjust the charging price in a timely fashion to unlock the flexibility of electric vehicle (EV) customers, has been extensively studied with the rapid development of charging technologies. Many existing works on dynamic pricing focus on maximizing the social welfare of charging service providers and EV customers. Cases of high-dimensional charging environments, which are often encountered with the rapid growth of EV market penetration, have been rarely considered to date. This article proposes a new dynamic pricing framework for EV charging stations that can offer multiple charging options to customers over a finite-time horizon. The charging price can be dynamically adjusted to maximize the quality of service (QoS) with a differentiated service requirement level (SRL) whenever the arrival rates and queuing system capacities of the charging systems are given at the end of a time period. The dynamic pricing problem is formulated as a finite-discrete horizon Markov decision process (MDP) with a mixed state space. A customized deep reinforcement learning (DRL) approach is employed to solve the examined EV dynamic pricing problem. The simulation results demonstrate the effectiveness of the proposed method.

This paper investigates the covert communications via spectrum allocations in a cellular-enabled unmanned aerial vehicle (UAV) network consisting of a base station (BS), UAVs, ground users (GUs), and a warden, where warden attempts to detect the transmission from a target GU to a UAV receiver. We formulate the spectrum allocation as an optimization problem with the constraints of covertness performance requirement and the qualities of service (QoS) of cellular communications. This is a nonlinear and nonconvex problem, which is generally challenging to be solved. Thus, we propose a deep reinforcement learning (DRL) approach to solve it. Under such an approach, we first model the multi-agent DRL environment in such networks. Then we define the state, action, reward and interaction mechanism of the DRL environment. Finally, a DRL algorithm is presented for learning the optimal policy of spectrum allocation.

In recent years, machine learning has been vastly used in many scientific experiments to analyze and predict the data [1]. Since there are many types of machine learning and each of them has additional further branches, it's necessary to understand which one can predict or summarize the data with more accuracy. This paper mainly focuses on traditional machine learning and active learning. To compare the effects of these two types of machine learning, the paper uses a dataset about using metal oxide semiconductor (MOX) to measure the concentration of carbon monoxide (CO). In the first part, the paper tests the accuracy of predicting CO concentration with 3 types of traditional machine learning: Decision Tree, Random Forest and K-nearest neighbors. In the second part, the paper chooses the traditional machine learning which has the best performance in the first part and compares the accuracy of it with the active learning on predicting CO concentration. The result is that the classification accuracy of Random Forest is 0.625, which is the highest among them. And active learning is generally better than traditional machine learning when training samples are small, and they have a similar accuracy when the training sample is enough.

A deep reinforcement learning approach is used to analyze the optimal 3-dimensional interplanetary transfers of a solar sail, accounting for various sources of uncertainty. The propulsive acceleration of the sail is described using an optical thrust model, with nominal optical coefficients derived from recently published experimental measurements. Two primary sources of uncertainty in the solar sail are considered: the imprecise knowledge of the sail’s optical properties, which impacts both the magnitude and direction of the propulsive acceleration, and the presence of wrinkles on the sail due to the folding (prior to launch) and unfolding (after release on orbit) of the ultrathin membrane. The study begins with a minimum-time interplanetary trajectory obtained using an indirect optimization technique in an unperturbed scenario, serving as the reference trajectory for the sail in the presence of model uncertainties. To account for these uncertainties, a proximal policy optimization algorithm is used to train an agent that learns a control policy associating any orbital state with the corresponding sail attitude, minimizing deviations from the reference trajectory. Two distinct scenarios are analyzed, each incorporating the aforementioned sources of uncertainty. The trained control policies are then tested through Monte Carlo simulations to evaluate their effectiveness and robustness. As a case study, a 3-dimensional transfer from Earth’s orbit to Venus’ orbit is examined, demonstrating that the control policy derived from reinforcement learning is capable of guiding the sail to its target with good accuracy, providing real-time control with relatively low computational effort.

The food industry faces significant challenges in managing operational costs due to its high energy intensity and rising energy prices. Industrial food-processing facilities, with substantial thermal capacities and large demands for cooling and heating, offer promising opportunities for demand response (DR) strategies. This study explores the application of deep reinforcement learning (RL) as an innovative, data-driven approach for DR in the food industry. By leveraging the adaptive, self-learning capabilities of RL, energy costs in the investigated plant are effectively decreased. The RL algorithm was compared with the well-established optimization method Mixed Integer Linear Programming (MILP), and both were benchmarked against a reference scenario without DR. The two optimization strategies demonstrate cost savings of 17.57% and 18.65% for RL and MILP, respectively. Although RL is slightly less efficient in cost reduction, it significantly outperforms in computational speed, being approximately 20 times faster. During operation, RL only needs 2ms per optimization compared to 19s for MILP, making it a promising optimization tool for edge computing. Moreover, while MILP’s computation time increases considerably with the number of binary variables, RL efficiently learns dynamic system behavior and scales to more complex systems without significant performance degradation. These results highlight that deep RL, when applied to DR, offers substantial cost savings and computational efficiency, with broad applicability to energy management in various applications.

The modern network is equipped with many service functions to acquire high-quality service. The emergence of network function virtualization (NFV) provides a convenient way to accomplish the network services in the form of virtual network function (VNF) and also makes the scheduling solution of VNFs flexible. The VNFs can be deployed on commodity servers as software processes. Besides, multiple VNFs are chained in a specified order as a service function chain (SFC) to serve a given flow, increasing the scheduling difficulty to minimize the average flow completion time. In this paper, we study the problem of scheduling multiple SFCs with the constraint of different resource limitations in various commodity servers. This problem is typically formulated as an Integer Linear Programming (ILP) problem, which is NP-hard. To well tackle this problem, we propose a deep reinforcement learning (DRL) approach. It involves multi-step decision making, which can be naturally transformed into a DRL problem. We design specific reward and state representations for such a multi-resource scheduling problem. We also consider how to use DRL to handle online requests of SFCs. The experiment results demonstrate that the DRL approach can significantly reduce the average completion time of a set of SFC and achieves a cost saving of 39.94% against the benchmark method.

To alleviate communication pressure and terminal resource constraints in mobile edge computing (MEC) networks, this paper proposes a resource allocation optimization method for MEC systems that integrates data compression technology and non-orthogonal multiple access technology. This method considers practical constraints such as terminal device battery capacity and computational resource limitations. By jointly optimizing computational resource allocation, task offloading strategies, and data compression ratios, it constructs an optimization model aimed at minimizing the total task processing latency. Addressing the challenges stemming from the non-convex nature of the problem and the dynamic variations in channel conditions and task requirements, this paper proposes a softmax deep double deterministic policy gradient algorithm, where softmax operator function mitigates both overestimation and underestimation biases inherent in traditional reinforcement learning frameworks, enhancing convergence performance. Utilizing a deep reinforcement learning framework, the algorithm achieves joint decision-making optimization for computational resources, task offloading, and compression ratios, thereby minimizing the total task processing latency while satisfying transmit power and computational resource constraints. Simulation results demonstrate that the proposed scheme exhibits significant advantages over benchmark algorithms in terms of convergence speed and task processing latency.

Online panoramic video has recently gained enormous popularity. Tile-based adaptive streaming is a promising paradigm to deliver a panoramic video for the sake of bandwidth saving. Nevertheless, it is challenging to accurately predict user’s field of view (FoV) and deliver the optimal bitrate due to the dynamic user behavior and time-varying network conditions. In this paper, we propose a novel approach of deep reinforcement learning based predictive panoramic video delivering. Specifically, a carefully-devised long short-term memory (LSTM) model is used to predict the FoV in the next few seconds. Our quality adaptation policy is based on a deep reinforcement learning (DRL) agent, which is able to intelligently adapt its bitrate selection policy tailored to the dynamic environments. To validate the effectiveness we have implemented a prototype of this system. With the integrated DRL algorithm Rainbow, we have achieved a superior performance in terms of the quality of experience (QoE) score, which outperforms existing panoramic video streaming frameworks.