Multi-microgrid Energy Management Based on Finite Horizon Twin Delayed Deep Deterministic Policy Gradient

A survey on deep learning methods for power load and renewable energy forecasting in smart microgrids

Multi-agent reinforcement learning dealing with hybrid action spaces: A case study for off-grid oriented renewable building energy system

This paper aims to introduce a solution to Sudan's inadequate electricity supply; focusing on current unconnected rural areas and the high cost of connecting these areas to Sudan's national grid. Microgrids were introduced as a viable option to create small scale distributed grids that depend solely on renewable energy to generate sufficient electricity to satisfy their loads. The paper also aims to enhance the usability of Microgrids by introducing a Machine learning technique to their secondary control that uses energy trading to ensure that all loads in islanded Microgrids are secured. The algorithm uses Reinforcement learning as control for the trading procedure. Data was extracted from a Matlab simulation and was then used to enhance the design of the Reinforcement learning environment. A generic environment for microgrids was designed and implemented which be further used in Reinforcement learning smart grids applications. A set of trading rules were implemented so that the Reinforcement Learning agents can use them across three Microgrids. The agent sees the three micro grids as one primary and two acting as trading game players. Two deep Reinforcement learning algorithms were explored as a solution; the first was an on-policy algorithm; Proximal Policy Optimization (PPO), and the second was an off-policy algorithm; Deep Deterministic Policy Gradients (DDPG). The results of applying both algorithms at three villages in Northern Kordufan State, Hamza ELsheikh, Tannah, and Um Bader were then compared. The algorithms achieved grid equilibrium without any grid loss and achieved profit from the trading process, reducing the time of return for the initial cost of the microgrid.

Renewable Energy Sources (RESs) are gaining considerable attention to reduce human dependence on fossil fuels and minimize harmful gases in our surroundings. Existing literature on energy trading focused on providing renewable energy to smart homes, smart buildings, and smart offices to fulfill their daily energy demands obtained from RESs. Besides, Electric Vehicles (EVs) use either power grid energy or a battery exchange mechanism to recharge their low EV batteries. The continuous use of power grids to recharge low EV batteries causes a significant load on power grids. Due to this, power grids are inadequate to fulfill the ever-increasing demands of EVs in the future. In this context, we propose a Blockchain Enabled Energy Trading (BEET) framework oriented EV charging. A system architecture of the BEET framework is presented to describe the functioning of each layer and its associated entities. We formulate an optimization problem that maximizes the revenue in the energy trading process using a knapsack optimization. Smart contracts are designed on the consortium blockchain network to sell and buy renewable energy to aggregators and from producers, respectively. Moreover, an EV charging mechanism is designed to intelligently allocate renewable energy to consumers at a low price. A comparative analysis is performed with state-of-the-art works in terms of charging price, revenue, throughput, and latency. The results indicate that the BEET framework outperforms compared to state-of-the-art works to address the renewable energy demand problem to realize E-mobility. It is clarified that the data considered in the experimental analysis were obtained from statistical simulations in realistic E-Mobility environment settings.

With the technical growth and the reduction of deployment cost for distributed energy resources (DERs), such as solar photovoltaic (PV), energy trading has been recently encouraged to energy consumers, which can sell energy from their own energy storage system (ESS). Meanwhile, due to the unprecedented rise of greenhouse gas (GHG) emissions, some countries (e.g., Republic of Korea and India) have mandated using a renewable energy certificate (REC) in energy trading markets. In this paper, we propose an energy broker model to boost energy trading between the existing power grid and energy consumers. In particular, to maximize the profits of energy consumers and the energy provider, the proposed energy broker is in charge of deciding the optimal demand and dynamic price of energy in an REC-based energy trading market. In this solution, the smart agents (e.g., IoT intelligent devices) of consumers exchange energy trading associated information, including the amount of energy generation, price and REC. For deciding the optimal demand and dynamic pricing, we formulate convex optimization problems using dual decomposition. Through a numerical simulation analysis, we compare the performance of the proposed dynamic pricing strategy with the conventional pricing strategies. Results show that the proposed dynamic pricing and demand control strategies can encourage energy trading by allowing RECs trading of the conventional power grid.

The transition to sustainable and intelligent energy systems has intensified the development of smart microgrids, which offer decentralized, resilient, and efficient power solutions. This review critically examines the integration of Artificial Intelligence (AI) and Deep Reinforcement Learning (DRL) into smart microgrid platforms, focusing on their role in optimizing sustainable energy management. Traditional energy management systems often struggle to adapt to the dynamic nature of modern energy demands, renewable energy intermittency, and grid complexity. AI-driven solutions, particularly DRL, provide adaptive, autonomous, and data-driven mechanisms for real-time decision-making and predictive control within microgrids. DRL, by learning optimal policies through interaction with the environment, is capable of handling multi-objective problems, including demand-response optimization, energy storage control, load forecasting, and distributed generation scheduling. This paper synthesizes recent advancements and applications of DRL algorithms such as Deep Q-Networks (DQN), Deep Deterministic Policy Gradient (DDPG), and Proximal Policy Optimization (PPO) in smart microgrids. It also explores hybrid models that combine DRL with other AI techniques, such as fuzzy logic and neural networks, to improve performance under uncertainty and nonlinearity. Furthermore, the review evaluates benchmark testbeds, simulation tools, and real-time platforms used to implement and validate these intelligent systems. Challenges such as high computational costs, model generalizability, real-time implementation issues, and cybersecurity vulnerabilities are discussed. The study also highlights recent research trends emphasizing decentralized control, edge computing, and federated learning to enhance scalability and privacy in DRL-based microgrid applications. Future directions suggest the need for explainable AI (XAI), robust training environments, and standardization to facilitate the wider adoption of DRL in real-world microgrid systems. The integration of DRL into smart microgrids represents a transformative shift toward resilient, efficient, and sustainable energy ecosystems. This review offers a comprehensive understanding of how AI and DRL are revolutionizing energy management, providing a foundation for future research and practical deployment in smart energy infrastructures.

This paper studies the real-time energy trading problem in a smart community consisting of a group of grid-connected prosumers with controllable loads, renewable generations and energy storage systems. We propose a peer-to-peer (P2P) energy trading system, which integrates energy trading with energy management, enabling each prosumer to jointly manage its energy consumption, storage scheduling and energy trading in a dynamic manner for smart communities consisting of a group of grid-connected prosumers with controllable loads, renewable generations and energy storage systems. The proposed community-based P2P energy trading system combines an online energy control and trading algorithm with a double auction mechanism. The energy control and trading algorithm is designed based on the Lyapunov theory, allowing each prosumer to independently determine its bid in each time slot only based on its current energy supply condition, while the trading price, which is determined via the double auction mechanism, reflects the collective energy supply conditions of all prosumers participating in energy trading. The integration of the Lyapunov-based energy control and trading algorithm and the double auction mechanism yields a dynamic energy trading pricing mechanism that induces the prosumers to participate in energy trading in a coordinated manner by influencing the energy consumption, energy charging/discharging and energy trading decisions of the prosumers. Numerical simulation results demonstrate that energy exchange in the proposed scalable energy trading system yields significant improvements in terms of energy cost savings and renewable energy utilization efficiency, while ensuring the fair sharing of the benefits reaped from energy trading among the prosumers.

Energy sources are critical in the industrial sector, particularly as population growth intensifies the pressure on industries to scale up production to meet increasing demands. Integrating renewable energy sources such as biomass, biogas, and photovoltaic systems in the palm oil production process can be considered a pivotal strategy for mitigating carbon emissions. However, load fluctuations due to mill size and insufficient supply of biogas and photovoltaic are significant challenges in energy management for the palm oil industry. Adopting renewable energy solutions is hindered by process disturbances and climatic variability caused by weather, ambient temperature, relative humidity, and solar radiation. These factors pose constraints on the implementation of renewable energy solutions. Therefore, this study aims to fill the gap in terms of renewable energy management within the palm oil industry by applying reinforcement learning to achieve effective energy management and reduce carbon emissions under five climatic scenarios. The result shows that reinforcement learning effectively combines renewable energy sources to generate electricity and steam while efficiently managing energy storage without exceeding predefined levels. Additionally, deploying a photovoltaic system contributes to energy savings from biomass and biogas of 2.9% and reduces average daily carbon dioxide emissions by 3137 kg. The proposed method has proved beneficial across all climate scenarios for energy savings and emission reduction.

The rapid expansion of renewable energy in buildings has been expedited by technological advancements and government policies. However, including highly permeable intermittent renewables and energy storage presents significant challenges for traditional home energy management systems (HEMSs). Deep reinforcement learning (DRL) is regarded as the most efficient approach for tackling these problems because of its robust nonlinear fitting capacity and capability to operate without a predefined model. This paper presents a DRL control method intended to lower energy expenses and elevate renewable energy usage by optimizing the actions of the battery and heat pump in HEMS. We propose four DRL algorithms and thoroughly assess their performance. In pursuit of this objective, we also devise a new reward function for multi-objective optimization and an interactive environment grounded in expert experience. The results demonstrate that the TD3 algorithm excels in cost savings and PV self-consumption. Compared to the baseline model, the TD3 model achieved a 13.79% reduction in operating costs and a 5.07% increase in PV self-consumption. Additionally, we explored the impact of the feed-in tariff (FiT) on TD3’s performance, revealing its resilience even when the FiT decreases. This comparison provides insights into algorithm selection for specific applications, promoting the development of DRL-driven energy management solutions.

Worldwide, the current energy system increasingly shows economic, social and environmental drawbacks and it does not appear to be sustainable. Among the most pressing problems are climate change due to CO2 emissions, the depletion of fossil resources, the security of energy supply, and threats to economic development. These problems can be addressed by moving towards energy systems that increasingly rely on renewable and sustainable energy sources. At the same time, the rapid progress in renewable energy harvesting and information and telecommunication technologies offer unprecedented opportunities for energy generation, distribution and management, and to more strongly involve people in the process of energy generation and management, which is needed to further enhance the sustainable energy systems. Indeed, renewable and sustainable energy sources are likely to be produced and distributed in a decentralised manner, and require energy management systems that differ greatly from the current centralised system. Hence, the integration of renewable sources has major implications for the design and management of energy systems, relying on innovative ICT solutions and stronger user involvement that facilitate supply and demand matching. Effective energy management requires the active participation and support of end users, who should accept the energy management systems, and adapt their energy use to the available demand as to enhance the efficiency and stability of the energy system. Hence, the design of future smart grids should be based on sound behavioural models, which should be integrated in ICT centred solutions that facilitate supply and demand matching. More specifically, we need to understand how truly complex information on energy production, use and storage can be effectively communicated among relevant actors, and what type of information and incentives should be provided to realise effective supply and demand matching. In addition, we need to understand how to take into account cultural specifics in the design of energy management systems, to enhance effective energy management across the world. The active and aware involvement of the end-user can only be achieved if a scalable and reliable ICT infrastructure is in place, which naturally interfaces the energy flows with the decision autonomy of the people. Hence, the design of effective energy management systems should be take into account key behavioural as well as ICT requirements in an integrated way. Our proposal is thus to investigate (1) ICT based solutions for monitoring and controlling the power grid focusing on the distribution networks (medium-low voltage) and buildings (offices and residential complexes) specific to the Qatar context; (2) behavioural models on effective and acceptable incentives to match energy supply and demand; (3) cultural specificities in the design of energy management systems by comparing results from research in Qatar to similar research in other parts of the world, in particular Europe. Together, these will provide important insights in how to promote efficient and stable renewable energy systems that meet Qatar's sustainability goals.

The online 3D bin packing problem(3D-BPP) is widely used in the logistics industry and is of great practical significance for promoting the intelligent transformation of the industry. The heuristic algorithm relies too much on manual experience to formulate more perfect packing rules. In recent years, many scholars solve 3D-BPP via deep reinforcement learning(DRL) algorithms. However, they ignore many skills used in manual packing, one of the most important skill is workers put the item aside if the item is packed improperly. Inspired by this skill, we propose a DRL algorithm with a buffer zone. Firstly, we define the wasted space and the buffer zone. And then, we integrate them into the DRL algorithm framework. Importantly, we compare the bin utilization with di erent thresholds of wasted space and di erent buffer zone sizes. Experimental results show that our algorithm outperforms existing heuristic algorithms and DRL algorithms.

Using Deep Reinforcement Learning (DRL) algorithm to deal with autonomous driving tasks usually have unsatisfied performance due to lack of robustness and means to escape local optimum. In this article, we designs a Survival-Oriented Reinforcement Learning (SORL) model that tackle these problems by setting survival rather than maximize total reward as first priority. In SORL model, we model autonomous driving task as Constrained Markov Decision Process (CMDP) and introduce Negative-Avoidance Function to learn from previous failure. The SORL model greatly speed up the training process and improve the robustness of normal Deep Reinforcement Learning algorithm.

Problem definition: The beer game is widely used in supply chain management classes to demonstrate the bullwhip effect and the importance of supply chain coordination. The game is a decentralized, multiagent, cooperative problem that can be modeled as a serial supply chain network in which agents choose order quantities while cooperatively attempting to minimize the network’s total cost, although each agent only observes local information. Academic/practical relevance: Under some conditions, a base-stock replenishment policy is optimal. However, in a decentralized supply chain in which some agents act irrationally, there is no known optimal policy for an agent wishing to act optimally. Methodology: We propose a deep reinforcement learning (RL) algorithm to play the beer game. Our algorithm makes no assumptions about costs or other settings. As with any deep RL algorithm, training is computationally intensive, but once trained, the algorithm executes in real time. We propose a transfer-learning approach so that training performed for one agent can be adapted quickly for other agents and settings. Results: When playing with teammates who follow a base-stock policy, our algorithm obtains near-optimal order quantities. More important, it performs significantly better than a base-stock policy when other agents use a more realistic model of human ordering behavior. We observe similar results using a real-world data set. Sensitivity analysis shows that a trained model is robust to changes in the cost coefficients. Finally, applying transfer learning reduces the training time by one order of magnitude. Managerial implications: This paper shows how artificial intelligence can be applied to inventory optimization. Our approach can be extended to other supply chain optimization problems, especially those in which supply chain partners act in irrational or unpredictable ways. Our RL agent has been integrated into a new online beer game, which has been played more than 17,000 times by more than 4,000 people.

Reinforcement Learning (RL) has become one of the most powerful methods in Artificial Intelligence (AI) which allows a system to learn as it is engaged in an interaction with its environment and to maximize its actions by performing a trial and error process. Although classical RL algorithms can give us a fundamental idea on RL, like the Q-Learning, its usefulness wanes in high-dimensional or continuous-state spaces. To overcome these shortcomings, Deep Reinforcement Learning (DRL) algorithms including Deep Q-Networks (DQN) build on neural networks to learn action-value functions, allowing horizontal scalability and more efficient learning. This paper reports on the experimental comparison of Q-Learning and DQN based on the environment CartPole-v1. Both of the algorithms were trained in the controlled environment and tested under various metrics that include cumulative rewards, success rate, convergence speed, safety (failure rate), and path efficiency. Findings indicate that Q-Learning moves towards moderate stability with slower convergence but DQN takes shorter learning time, is more reliable and performs better. Such results indicate the ground-breaking potential of the DRL in the progress of AI toward transcending classical RL restrictions. Future research topics encompass implementation of more challenging experimental settings, a test of other DRL algorithms (e.g., PPO, SAC), and generalization, scalability and safety in real-world environments.

<span>The utilization of games in training the reinforcement learning (RL) agent is to describe the complex and high-dimensional real-world data. By utilizing games, RL researchers will be able to evade high experimental costs in training an agent to do intelligence tasks. The objective of this research is to generate intelligent agent behaviors in multi-agent game artificial intelligence (AI) using deep reinforcement learning (DRL) algorithm. A basic RL algorithm called deep Q network is chosen to be implemented. The agent is trained by the environment's raw pixel images and the action list information. The experiments conducted by using this algorithm show the agent’s decision-making ability in choosing a favorable action. In the default setting for the algorithm, the training is set into 1 epoch and 0.0025 learning rate. The number of training iterations is set to one as the training function will be repeatedly called for every 4-timestep. However, the author also experimented with two different scenarios in training the agent and compared the results. The experimental findings demonstrate that our agents learn correctly and successfully while actively participating in the game in real time. Additionally, our agent can quickly adjust against a different enemy on a varied map because of the observed knowledge from prior training.</span>