Scheduling Optimization of Hybrid Microgrid Generators Based on Deep Reinforcement Learning

Microgrids offer enhanced resilience and efficiency but require sophisticated energy management systems (EMS) to balance conflicting objectives like cost minimization, renewable energy utilization, and component longevity, especially under uncertainty. Traditional optimization methods often rely on precise forecasts and may struggle with real-time adaptation and complex trade-offs like battery degradation. This research aimed to develop a deep reinforcement learning (DRL) based EMS for optimizing microgrid operation considering operational cost, battery degradation, and renewable generation uncertainty. A deep Q-network (DQN) based reinforcement learning agent was trained to manage energy flows within a simulated microgrid comprising solar PV, battery storage, controllable loads, and a grid connection. The reward function incorporated operational costs, battery degradation, and renewable utilization objectives, with the agent learning control policies through environment interaction. The DRL-based EMS demonstrated effective adaptive control, achieving a 12.01% reduction in overall operational costs compared to the model predictive control benchmark. The DRL agent implicitly learned strategies that reduced battery degradation by 8.19% while increasing renewable energy utilization by 10.39%. Most notably, the approach maintained robust performance under uncertainty, with only 8.9% cost increase under severe forecast errors compared to 21.5% for conventional methods. This study demonstrates the efficacy of DRL for adaptive multi-objective microgrid energy management, successfully balancing economic operation, battery health preservation, and renewable energy integration under uncertainty.

Abstract: Online resource management in cloud computing must respond to rapidly changing demand and system conditions, often exceeding the adaptability of static or periodically recomputed optimisation plans. This study formulates cloud resource management as a Markov decision process (MDP) and investigates deep reinforcement learning (DRL) as an approximate dynamic programming (ADP) approach for deriving adaptive control policies. The system state encompasses workload, capacity utilisation, and performance indicators, while actions govern virtual machine placement, scaling, and server power-state transitions. Reward functions combine energy and operating costs with penalties associated with service-level agreement violations. DRL algorithms, including deep Q-networks, advantage actor–critic, and proximal policy optimization, are implemented in PyTorch and trained through interaction with a CloudSim-based simulation environment calibrated to realistic load and power profiles. The learned policies are evaluated against deterministic and scenario-based stochastic optimisation baselines using metrics such as average cost, energy consumption, SLA violations, and robustness to non-stationary demand. Results show that DRL policies can approach or surpass the performance of stochastic optimisation solutions in highly dynamic scenarios, while maintaining feasible control actions. The study demonstrates how DRL can be positioned within an OR/ADP framework and highlights conditions under which learning-based methods offer significant advantages for online cloud control. Keywords: deep reinforcement learning, approximate dynamic programming, cloud computing, Markov decision processes, online resource management, energy-aware control, PyTorch, simulation

Dynamic energy dispatch strategy for integrated energy system based on improved deep reinforcement learning

With the rapid development of electric vehicles (EVs) and renewable energy, the charging scheduling of EVs with renewable energy becomes a hot issue. Nevertheless, as both the renewable energy and charging demand of EVs are hard to predict, it is difficult to get the optimal charging strategy of the EVs charging problem with renewable energy. In this paper, we consider a multi-EVs charging problem with a discontinuous charging process, where the charging decision for each EV is constrained. We model the problem as Markov decision processes (MDPs) and propose a novel Deep Reinforcement Learning (DRL) method called Constrained Double Deep Q-learning Network (CDDQN) to solve the MDP problem with large state space and decision constraints. Compared with the most of existing DRL methods, the CDDQN method embeds an action constraint model to a double deep Q-learning network (DDQN), which decreases the error of Q-value estimation and improves the accuracy of the charging policy by generating more effective training data. We conduct experiments on a simulation study and compare the proposed method with other DRL methods and experience charging policies. The quantitative results show that the proposed method achieves superior performance on getting the charging policy and outperforms the other methods in getting the minimum charging cost obviously.

To train a mobile robot to navigate using end-to-end approach which maps sensors data into actions, we can use deep reinforcement learning (DRL) method by providing training environments with proper reward functions. Although some studies have shown the success of DRL in navigation task for mobile robots, the method needs appropriate hyperparameter settings such as the environment's timestep size and the robot's velocity range to produce a good navigation policy. The previous existing DRL framework has proposed the use of odometry sensor to generate dynamic timestep size in the environment to solve the mismatch problem between the timestep size and the robot's velocity. However, the framework lacks a procedure for checking terminal conditions which may occur during action executions resulting inconsistency in the environment and poor navigation policies. In the case of navigation task, terminal conditions may happen when the robot achieves the navigation goal position or collides with obstacles while performing an action in one timestep. To cope with this problem, we propose a state transition checking method in the DRL environment which is specific for navigation task that leverages odometry and laser sensor to ensure that the environment follows Markov Decision Process with dynamic timestep size. We also introduce a velocity increment scheduling to stabilize the mobile robot during training. Our experiment results show that state transition checking along with the velocity increment scheduling are able to make the robot navigate faster with higher success rate compared to other existing DRL frameworks.

The efficiency of modern drilling operations depends on the planning phase to determine possible well trajectories and the ability of the directional driller to traverse them accurately. Deviations from the planned trajectory while drilling often require updates to the original well plan, involving drilling engineers and rig personnel, which can be time-consuming due to several uncertainties, such as formation tendencies, survey measurement inaccuracy, or estimation errors. To address these challenges, this paper proposes an innovative solution that leverages artificial intelligence (AI) methods, specifically deep reinforcement learning (DRL) to dramatically reduce the need for continuous corrections to the well plan while drilling. In the DRL paradigm, the proposed approach eliminates the need for constant plan adjustments by training a drilling agent to imitate the driller's ability to dynamically adjust the well trajectory in real time based on information from previous drilling logs, well plans, and near-the-bit measurements. This research utilizes a physics-based simulation engine to model a directional drilling environment as a Markov Decision Process (MDP). The MDP is intended for an autonomous system based on geological data models and real-time measurements obtained while drilling (MWD) to train deep reinforcement learning (DRL) agents to drill directional wellbores that maintain maximum contact with the target formation. The simulator incorporates uncertainties of the real-world drilling environment, such as bit walk, formation properties, and drilling speed which, combined, help derive the actions performed by the drilling agent. Our findings reveal that using the proposed methodology in a virtual drilling environment, the drilling agent effectively navigates well paths in the presence of uncertainties and successfully tackle challenges such as avoiding excessive tortuosity and doglegs while maximizing contact with the target formation. Furthermore, the use of domain randomization during training enabled the RL agents to exhibit exceptional generalizability to a wide range of drilling scenarios through the random selection of drilling test sites from a set of unseen sites during training — demonstrating the ability of the agent to adapt and reach the target formation even when the initial well-plan is inaccurate — resulting in a 90% success rate. This self-correcting approach demonstrates the potential for automated, proactive, self-contained steering operations with minimal human involvement. The developed simulation framework is a pioneering approach to enhancing real-time adjustments of drilling well paths using reinforcement learning and optimizing drilling operations. It is the first-of-its-kind method to augment drillers’ ability to navigate through drilling uncertainties and maximize pay zone contact and pave the way for the creation of robust, scalable, and practical autonomous drilling systems.

Energy efficient task scheduling based on deep reinforcement learning in cloud environment: A specialized review

Renewable energy forecasting is crucial for pollution prevention, management, and long‐term sustainability. In response to the challenges associated with energy forecasting, the simultaneous deployment of several data‐processing approaches has been used in a variety of studies in order to improve the energy–time‐series analysis, finding that, when combined with the wavelet analysis, deep learning techniques can achieve high accuracy in energy forecasting applications. Consequently, we investigate the implementation of various wavelets within the structure of a long short‐term memory neural network (LSTM), resulting in the new LSTM wavelet (LSTMW) neural network. In addition, and as an improvement phase, we modeled the uncertainty and incorporated it into the forecast so that systemic biases and deviations could be accounted for (LSTMW with luster: LSTMWL). The models were evaluated using data from six renewable power generation plants in Chile. When compared to other approaches, experimental results show that our method provides a prediction error within an acceptable range, achieving a coefficient of determination ( R 2 ) between 0.73 and 0.98 across different test scenarios, and a consistent alignment between forecasted and observed values, particularly during the first 3 prediction steps.

With the rapid development of smart grid and renewable energy, demand response mechanism has become a key means to balance microgrid supply and demand and improve energy efficiency. As the traditional power demand response algorithm is difficult to deal with the uncertainty of power demand and the adverse effects caused by customers, this paper proposes a power demand response algorithm based on deep reinforcement learning. Firstly, we study and design a price -based layered electricity energy demand response model. Then, the dynamic optimal pricing decision of power trading market is described as Markov decision process, and the learning mechanism under the framework of deep reinforcement learning is expounded. Finally, a solution algorithm based on deep reinforcement learning is designed. The simulation results show that the proposed algorithm can adjust the electricity price adaptively according to the load demand, which can reduce the cost of users by about 17% and reduce the peak load demand.

Due to the complexity and high cost in the actual stock market, it is often difficult to achieve. Deep reinforcement learning has many advantages and can already outperform human players in many challenging video games. This paper defines the Markov decision process (MDP) model of stock portfolio trading. In order to diversify the weight of investment, on the basis of introducing entropy to improve the deep deterministic strategy gradient (DDPG) algorithm, this paper proposes a portfolio strategy based on technical indicators. Based on deep reinforcement learning, the strategy is studied from two aspects: technical indicators and covariance of portfolio stocks to further improve returns. It calculates technical indicators based on the stock’s closing, opening, high, and low prices, and then calculates the covariance of the portfolio. Input the closing price, technical indicators, covariance, stock share, and fund balance as state input, and output the portfolio value. The constituent stocks of the CSI 300 Index are selected as the stocks to be traded in the portfolio for experiments, and the results show that the strategy proposed in this paper has higher returns in terms of yield and Sharpe ratio.

With the increasing threat of global warming due to high energy consumption of wireless network infrastructure, cell activation complements the capabilities of next-generation wireless technology. In this article, we propose an energy consumption optimization strategy based on deep reinforcement learning (DRL) and transfer learning (TL) techniques. We implement an adaptive reward to autonomously adjust parameters in a reward function to balance energy consumption and quality of service (QoS) requirement of users during the learning process. We further formulate a cell activation/deactivation problem as a Markov decision process and set up our proposed relational DRL model to meet the QoS requirements of users with a minimum number of active remote radio heads under a traffic model defined to simulate a real-world scenario. A weighted TL algorithm has been developed in DRL to validate sample data from a source task. Extensive simulations reveal that the proposed scheme based on the adaptive reward has better performance in balancing the QoS requirement of users and system energy consumption. Finally, based on our simulation results, we conclude that combining DRL with TL speeds up the learning process.

Deep reinforcement learning optimized double exponentially weighted moving average controller for chemical mechanical polishing processes

Classical active flow control (AFC) methods based on solving the Navier–Stokes equations are laborious and computationally intensive even with the use of reduced-order models. Data-driven methods offer a promising alternative for AFC, and they have been applied successfully to reduce the drag of two-dimensional bluff bodies, such as a circular cylinder, using deep reinforcement-learning (DRL) paradigms. However, due to the onset of weak turbulence in the wake, the standard DRL method tends to result in large fluctuations in the unsteady forces acting on the cylinder as the Reynolds number increases. In this study, a Markov decision process (MDP) with time delays is introduced to model and quantify the action delays in the environment in a DRL process due to the time difference between control actuation and flow response along with the use of a first-order autoregressive policy (ARP). This hybrid DRL method is applied to control the vortex-shedding process from a two-dimensional circular cylinder using four synthetic jet actuators at a freestream Reynolds number of 400. This method has yielded a stable and coherent control, which results in a steadier and more elongated vortex formation zone behind the cylinder, hence, a much weaker vortex-shedding process and less fluctuating lift and drag forces. Compared to the standard DRL method, this method utilizes the historical samples without additional sampling in training, and it is capable of reducing the magnitude of drag and lift fluctuations by approximately 90% while achieving a similar level of drag reduction in the deterministic control at the same actuation frequency. This study demonstrates the necessity of including a physics-informed delay and regressive nature in the MDP and the benefits of introducing ARPs to achieve a robust and temporal-coherent control of unsteady forces in active flow control.

Due to the features of strong non-linearity and hybrid driving with multi-power sources, a novel deep reinforcement learning(DRL)-based hierarchical energy control architecture is presented for a group of connected hybrid electric vehicles (HEVs) in the platoon through cloud platform. Firstly, a higher-level model predictive control (MPC) law is designed to determine the desired acceleration of connected HEVs in a platoon. Secondly,a reward function with the change of state of charge (SOC) and instantaneous fuel consumption as independent variables is constructed, and the expert knowledge-based deep deterministic policy gradient (DDPG) method with prioritized experience replay (PER) is used for designing the lower-level energy management strategy of connected HEVs platooning. Then, to illustrate the rationality of decision-making by the presented DDPG strategy, the influence of state variables such as vehicle speed, desired acceleration and battery SOC on the agent action values of presented DDPG method is discussed. Finally, the test results show that the proposed control scheme can reasonably allocate the engine and motor powers in real time, and finally achieve the safe and energy-saving driving of connected HEVs in the platoon.

In this paper, a Reinforcement Learning (RL)-based approach to optimally dispatch PV inverters in unbalanced distribution systems is presented. The proposed approach exploits a decentralized architecture in which PV inverters are operated by agents that perform all computational processes locally; while communicating with a central agent to guarantee voltage magnitude regulation within the distribution system. The dispatch problem of PV inverters is modeled as a Markov Decision Process (MDP), enabling the use of RL algorithms. A rolling horizon strategy is used to avoid the computational burden usually associated with continuous state and action spaces, coupled with a computationally efficient learning algorithm to model the action-value function for each PV inverter. The effectiveness of the proposed decentralized RL approach is compared with the optimal solution provided by a centralized nonlinear programming (NLP) formulation. Results showed that within several executions, the proposed approach converges either to the optimal solution or to solutions with a PV curtailment excess of less than 2.5% while still enforcing voltage magnitude regulation.