Solar Sail Transfers under Uncertainties: A Deep Reinforcement Learning Approach

An optimal solutions-guided deep reinforcement learning approach for online energy storage control

Multi-agent quantum-inspired deep reinforcement learning for real-time distributed generation control of 100% renewable energy systems

In Reinforcement Learning, a category of machine learning, learning is based on evaluative feedbacks without any supervised signals. The paper presents work aimed to understand the deep reinforcement learning approaches to creating such intelligent agents, by reproducing existing research and comparing their results. The project uses the Atari 2600 game called Breakout, in which the agent will learn control policies using deep reinforcement learning approaches to achieve a high score. The project explores two deep reinforcement learning approaches, Asynchronous Advantage actor-critic and Deep Q-Learning, both proposed by the DeepMind team, to train intelligent agents that can interact with an environment with automatic feature engineering thus requiring minimal domain knowledge.

Energy efficient task scheduling based on deep reinforcement learning in cloud environment: A specialized review

Optimal scheduling of island integrated energy systems considering multi-uncertainties and hydrothermal simultaneous transmission: A deep reinforcement learning approach

Millimeter-wave (mmWave) communication is a key technology of fifth generation wireless systems to achieve the expected 1000x data rate. With large bandwidth at the mmWave band, the link capacity between users and base stations (BSs) can be much higher compared to sub-6 GHz wireless systems. Meanwhile, due to the high cost of infrastructure upgrade, it would be difficult for operators to drastically enhance the capacity of backhaul links between mmWave BSs and the core network. As a result, the data rate provided by backhaul may not be sufficient to support all mmWave links; hence, the backhaul connection becomes the new bottleneck. On the other hand, as mmWave channels are subject to random blockage, the data rates of mmWave users significantly vary over time. With limited backhaul capacity and highly dynamic data rates of users, how to allocate backhaul resource to each user remains a challenge for mmWave systems. In this article, we present a deep reinforcement learning (DRL) approach to address this challenge. By learning the blockage pattern, the system dynamics can be captured and predicted, resulting in efficient utilization of backhaul resource. We begin with a discussion on DRL and its application in wireless systems. We then investigate the problem of backhaul resource allocation and present the DRL-based solution. Finally, we discuss open problems for future research and conclude this article.

Dynamic pricing, which aims to dynamically adjust the charging price in a timely fashion to unlock the flexibility of electric vehicle (EV) customers, has been extensively studied with the rapid development of charging technologies. Many existing works on dynamic pricing focus on maximizing the social welfare of charging service providers and EV customers. Cases of high-dimensional charging environments, which are often encountered with the rapid growth of EV market penetration, have been rarely considered to date. This article proposes a new dynamic pricing framework for EV charging stations that can offer multiple charging options to customers over a finite-time horizon. The charging price can be dynamically adjusted to maximize the quality of service (QoS) with a differentiated service requirement level (SRL) whenever the arrival rates and queuing system capacities of the charging systems are given at the end of a time period. The dynamic pricing problem is formulated as a finite-discrete horizon Markov decision process (MDP) with a mixed state space. A customized deep reinforcement learning (DRL) approach is employed to solve the examined EV dynamic pricing problem. The simulation results demonstrate the effectiveness of the proposed method.

This paper investigates the covert communications via spectrum allocations in a cellular-enabled unmanned aerial vehicle (UAV) network consisting of a base station (BS), UAVs, ground users (GUs), and a warden, where warden attempts to detect the transmission from a target GU to a UAV receiver. We formulate the spectrum allocation as an optimization problem with the constraints of covertness performance requirement and the qualities of service (QoS) of cellular communications. This is a nonlinear and nonconvex problem, which is generally challenging to be solved. Thus, we propose a deep reinforcement learning (DRL) approach to solve it. Under such an approach, we first model the multi-agent DRL environment in such networks. Then we define the state, action, reward and interaction mechanism of the DRL environment. Finally, a DRL algorithm is presented for learning the optimal policy of spectrum allocation.

The modern network is equipped with many service functions to acquire high-quality service. The emergence of network function virtualization (NFV) provides a convenient way to accomplish the network services in the form of virtual network function (VNF) and also makes the scheduling solution of VNFs flexible. The VNFs can be deployed on commodity servers as software processes. Besides, multiple VNFs are chained in a specified order as a service function chain (SFC) to serve a given flow, increasing the scheduling difficulty to minimize the average flow completion time. In this paper, we study the problem of scheduling multiple SFCs with the constraint of different resource limitations in various commodity servers. This problem is typically formulated as an Integer Linear Programming (ILP) problem, which is NP-hard. To well tackle this problem, we propose a deep reinforcement learning (DRL) approach. It involves multi-step decision making, which can be naturally transformed into a DRL problem. We design specific reward and state representations for such a multi-resource scheduling problem. We also consider how to use DRL to handle online requests of SFCs. The experiment results demonstrate that the DRL approach can significantly reduce the average completion time of a set of SFC and achieves a cost saving of 39.94% against the benchmark method.

The food industry faces significant challenges in managing operational costs due to its high energy intensity and rising energy prices. Industrial food-processing facilities, with substantial thermal capacities and large demands for cooling and heating, offer promising opportunities for demand response (DR) strategies. This study explores the application of deep reinforcement learning (RL) as an innovative, data-driven approach for DR in the food industry. By leveraging the adaptive, self-learning capabilities of RL, energy costs in the investigated plant are effectively decreased. The RL algorithm was compared with the well-established optimization method Mixed Integer Linear Programming (MILP), and both were benchmarked against a reference scenario without DR. The two optimization strategies demonstrate cost savings of 17.57% and 18.65% for RL and MILP, respectively. Although RL is slightly less efficient in cost reduction, it significantly outperforms in computational speed, being approximately 20 times faster. During operation, RL only needs 2ms per optimization compared to 19s for MILP, making it a promising optimization tool for edge computing. Moreover, while MILP’s computation time increases considerably with the number of binary variables, RL efficiently learns dynamic system behavior and scales to more complex systems without significant performance degradation. These results highlight that deep RL, when applied to DR, offers substantial cost savings and computational efficiency, with broad applicability to energy management in various applications.

Currently, helicopter towing operations mainly rely on towing systems and multi-person coordinated control, resulting in low work efficiency and posing a threat to life safety. With the development of artificial intelligence and computer science, automating helicopter towing systems is key to achieving safe and efficient helicopter operations. This paper proposes a helicopter tracking control method based on the Proximal Policy Optimization (PPO) algorithm for the helicopter towing trajectory tracking problem. This paper first improves the PPO algorithm network structure and sets the hyperparameters of the algorithm. Then, a deep learning environment is constructed based on the Markov decision process, and the state space and action space are reasonably set. A reward function is designed based on lateral deviation, longitudinal deviation, and heading deviation. Finally, the PPO algorithm is used to train the trajectory tracking control strategy, and the optimal tracking control strategy is obtained and verified by simulation. The simulation results show that the helicopter tracking control strategy based on the PPO algorithm has a fast convergence speed, and the tracking effect verifies the feasibility and generalization of the algorithm.

Autonomous decision-making in dynamic, uncertain environments (such as robotic navigation, self-driving vehicles, and smart grid management) requires integrating powerful perception with adaptive planning. We propose a hybrid framework that combines deep learning for perceptual representation and reinforcement learning (RL) for sequential control. The framework uses convolutional or recurrent neural networks to process high-dimensional sensory data (e.g., camera or sensor arrays) and outputs features or latent states, which are then fed into an RL algorithm (such as Deep Q-Network or Actor-Critic) to learn optimal policies. We detail the architecture, including neural-network-based value/policy approximation and experience replay mechanisms, and describe our simulated testbed (e.g. CARLA urban driving simulator or OpenAI Gym environments). Experiments in a simulated dynamic scenario demonstrate that our hybrid approach significantly outperforms standard tabular RL and pure deep-learning baselines. We provide training curves, comparative tables of performance metrics, and discuss the implications for real-world deployment. Key contributions include: a conceptual integration of deep representation learning with RL for dynamic control; a detailed methodology for training in simulation; empirical results illustrating learning progress and robustness.

Implementing action mask in proximal policy optimization (PPO) algorithm

A deep reinforcement learning approach is applied, for the first time, to solve the routing, modulation, spectrum and core allocation (RMSCA) problem in dynamic multicore fiber elastic optical networks (MCF-EONs). To do so, a new environment - compatible with OpenAI's Gym - was designed and implemented to emulate the operation of MCF-EONs. The new environment processes the agent actions (selection of route, core and spectrum slot) by considering the network state and physical-layer-related aspects. The latter includes the available modulation formats and their reach and the inter-core crosstalk (XT), an MCF-related impairment. If the resulting quality of the signal is acceptable, the environment allocates the resources selected by the agent. After processing the agent's action, the environment is configured to give the agent a numerical reward and information about the new network state. The blocking performance of four different agents was compared through simulation to 3 baseline heuristics used in MCF-EONs. Results obtained for the NSFNet and COST239 network topologies show that the best-performing agent achieves, on average, up to a four-times decrease in blocking probability concerning the best-performing baseline heuristic methods.

Rule-interposing deep reinforcement learning based energy management strategy for power-split hybrid electric vehicle