Data Driven UAV Racing with Deep Reinforcemnet learning

The transition to sustainable and intelligent energy systems has intensified the development of smart microgrids, which offer decentralized, resilient, and efficient power solutions. This review critically examines the integration of Artificial Intelligence (AI) and Deep Reinforcement Learning (DRL) into smart microgrid platforms, focusing on their role in optimizing sustainable energy management. Traditional energy management systems often struggle to adapt to the dynamic nature of modern energy demands, renewable energy intermittency, and grid complexity. AI-driven solutions, particularly DRL, provide adaptive, autonomous, and data-driven mechanisms for real-time decision-making and predictive control within microgrids. DRL, by learning optimal policies through interaction with the environment, is capable of handling multi-objective problems, including demand-response optimization, energy storage control, load forecasting, and distributed generation scheduling. This paper synthesizes recent advancements and applications of DRL algorithms such as Deep Q-Networks (DQN), Deep Deterministic Policy Gradient (DDPG), and Proximal Policy Optimization (PPO) in smart microgrids. It also explores hybrid models that combine DRL with other AI techniques, such as fuzzy logic and neural networks, to improve performance under uncertainty and nonlinearity. Furthermore, the review evaluates benchmark testbeds, simulation tools, and real-time platforms used to implement and validate these intelligent systems. Challenges such as high computational costs, model generalizability, real-time implementation issues, and cybersecurity vulnerabilities are discussed. The study also highlights recent research trends emphasizing decentralized control, edge computing, and federated learning to enhance scalability and privacy in DRL-based microgrid applications. Future directions suggest the need for explainable AI (XAI), robust training environments, and standardization to facilitate the wider adoption of DRL in real-world microgrid systems. The integration of DRL into smart microgrids represents a transformative shift toward resilient, efficient, and sustainable energy ecosystems. This review offers a comprehensive understanding of how AI and DRL are revolutionizing energy management, providing a foundation for future research and practical deployment in smart energy infrastructures.

In recent years, deep reinforcement learning (DRL) has made impressive achievements in many fields. However, existing DRL algorithms usually require a large amount of exploration to obtain a good action policy. In addition, in many complex situations, the reward function cannot be well designed to meet task requirements. These two problems will make it difficult for DRL to learn a good action policy within a relatively short period. The use of expert data can provide effective guidance and avoid unnecessary exploration. This study proposes a deep imitation reinforcement learning (DIRL) algorithm that uses a certain amount of expert demonstration data to speed up the training of DRL. In the proposed method, the learning agent imitates the expert's action policy by learning from demonstration data. After imitation learning, DRL is used to optimise the action policy in a self‐learning way. By experimental comparison on a video game called the Mario racing game, it is shown that the proposed DIRL algorithm with expert demonstration data can obtain much better performance than previous DRL algorithms without expert guidance.

Reinforcement learning-based real-time optimization of friction stir welding parameters for copper–aluminium dissimilar interfaces

Introduction: Deep reinforcement learning (deep RL) integrates the principles of reinforcement learning with deep neural networks, enabling agents to excel in diverse tasks ranging from playing board games such as Go and Chess to controlling robotic systems and autonomous vehicles. By leveraging foundational concepts of value functions, policy optimization, and temporal difference methods, deep RL has rapidly evolved and found applications in areas such as gaming, robotics, finance, and healthcare. Objective: This paper seeks to provide a comprehensive yet accessible overview of the evolution of deep RL and its leading algorithms. It aims to serve both as an introduction for newcomers to the field and as a practical guide for those seeking to select the most appropriate methods for specific problem domains. Methods: We begin by outlining fundamental reinforcement learning principles, followed by an exploration of early tabular Q-learning methods. We then trace the historical development of deep RL, highlighting key milestones such as the advent of deep Q-networks (DQN). The survey extends to policy gradient methods, actor–critic architectures, and state-of-the-art algorithms such as proximal policy optimization, soft actor–critic, and emerging model-based approaches. Throughout, we discuss the current challenges facing deep RL, including issues of sample efficiency, interpretability, and safety, as well as open research questions involving large-scale training, hierarchical architectures, and multi-task learning. Results: Our analysis demonstrates how critical breakthroughs have driven deep RL into increasingly complex application domains. We highlight existing limitations and ongoing bottlenecks, such as high data requirements and the need for more transparent, ethically aligned systems. Finally, we survey potential future directions, highlighting the importance of reliability and ethical considerations for real-world deployments.

Improving the interpretability of deep reinforcement learning in urban drainage system operation

To address the problem of low training efficiency and slow convergence in existing end-to-end flight control strategies based on deep reinforcement learning (DRL) for unmanned aerial vehicle (UAV), this paper proposes a learning-based control framework that dynamically integrates imitation learning (IL) and DRL, denoted as LSTM-DIL-PPO. First, Long Short-Term Memory (LSTM) is introduced to capture temporal dependencies in UAV observation sequences, enabling the policy to make time-aware decisions. Second, a Dynamic Imitation Learning (DIL) mechanism is introduced to balance expert and self-learned strategies via an improved action selection process and a hybrid loss design. Finally, the policy is optimized using Proximal Policy Optimization (PPO). Semi-physical simulation experimental results demonstrate that the presented algorithm achieves better performance in terms of convergence speed, task success rate and trajectory quality. under complex racing scenarios

By addressing the drawbacks of static optimization techniques, this research seeks to improve the dynamic energy management of thermoelectric generators (TEGs). Finding the best deep reinforcement learning (DRL) algorithm to maximize energy distribution, prolong battery life, and boost system efficiency in the face of the variable conditions present in waste heat recovery systems is the goal. The TEG system was modeled using the Markov decision process and implemented in a computer-simulated environment. Three advanced DRL algorithms were used: soft actor-critic (SAC), proximal policy optimization (PPO), and deep Q-networks (DQN); which were trained to act as intelligent controllers. The performance of each algorithm was systematically evaluated and compared using key metrics, including average cumulative reward, battery health, system efficiency, and a novel metric termed the energy fulfillment rate, which measures the ability to meet demand while storing surplus energy. The comparative analysis revealed a critical trade-off between maximizing performance and ensuring hardware longevity. The SAC algorithm demonstrated the best overall performance, achieving the highest average reward (− 7.03) and energy fulfillment rate (22.84%). However, the A2C, DDPG, and PPO algorithms all achieved a perfect average battery health of 100.00%, highlighting their superior capability for preserving system longevity, albeit with slightly lower rewards. The DQN algorithm consistently showed the least effective performance across all metrics, particularly in maintaining battery health (60.73%). The SAC algorithm is the most suitable of the methods tested for dynamically managing TEG systems. Its underlying principle of maximization of entropy enables a better exploration of control strategies, leading to a better balance between immediate energy dispatch and long-term storage goals. The findings confirm the significant potential of DRL to create efficient and adaptive controllers for renewable energy applications, although further validation of physical hardware is required to confirm real-world viability.Supplementary InformationThe online version contains supplementary material available at 10.1038/s41598-025-27210-7.

In this paper, we apply the Proximal Policy Optimization (PPO) algorithm in intelligent traffic signal control at a single intersection with eight lanes and four signal phases. The optimization goal is to minimize the average waiting time of vehicles so as to improve the traffic efficiency of the intersection. Extensive experiments are conducted in Simulation of Urban MObility (SUMO) to evaluate the performance of the proposed algorithm, and compare it with other classic algorithms including Deep Q-network (DQN), Advantage Actor Critic (A2C) and Fixed Time. Simulation results show that the proposed PPO algorithm outperforms the others under various traffic scenarios to different extent. The performance gain is significant under unbalanced traffic where one direction is saturated while the other is not, and becomes marginal when all the directions are saturated or unsaturated. PPO also demonstrates good portability and robustness over time-varying traffic patterns, while implies it could be a preferable option for implementation in real world intelligent traffic signal control systems.

Water quality plays a vital role in aquaculture sustainability, particularly for sensitive freshwater species such as Salmo letnica, which require strict physicochemical conditions for survival. This study evaluates the suitability of household water sources from eight rural villages in North Macedonia for supporting Salmo letnica in home-based tank environments. Seven key water quality parameters were measured and compared against biological thresholds derived from ecological literature. A Deep Reinforcement Learning (DRL) agent, trained using the Proximal Policy Optimization (PPO) algorithm, was developed to classify the suitability of water samples based on these parameters. While the DRL model exhibited low precision and recall due to the limited sample size, it provided a framework for interpretability through reward dynamics and parameter correlations. Among the eight villages, Forino and Gradec were found to meet all critical biological thresholds, while Kamenjane and Vrapçishte were identified as marginally suitable. The remaining locations exhibited insufficient oxygen levels or excessive nutrient concentrations. These findings demonstrate the potential of AI-based classification models in supporting aquaculture planning and ecological risk assessment. Future work will focus on data expansion, reward function refinement, and field-level model deployment. Unlike traditional supervised classifiers, the DRL agent enables autonomous learning and decision-making without requiring large labeled datasets. This makes the approach suitable for real-time, remote water quality monitoring in resource-limited rural areas.

Learning ability is the basic characteristic of human intelligence. The July 1, 2005 issue of Science published a list of 125 important questions in science. Among them, the question 94 “What are the limits of learning by machines?”. The annotation “Computers can already beat the world’s best chess players, and they have a wealth of information on the Web to draw on. But abstract reasoning is still beyond any machine”. In recent artificial intelligence has made great progresses. In 1997, the rise of the man-machine war, IBM Supercomputer Deep Blue defeated the chess master Garry Kasparov. On February 14, 2011, IBM’s Watson supercomputer won a practice round against Jeopardy champions Ken Jennings and Brad Rutter. In March 2016, Google DeepMind’s AlphaGo sealed a 4-1 victory over a South Korean Go grandmaster Lee Se-dol. This paper focuses on the machine learning methods of AlphaGo, including reinforcement learning, deep learning, deep reinforcement learning, analysis of the existing problems and the latest research progress. Deep reinforcement learning is the combination of deep learning and reinforcement learning, which can realize the learning algorithm from the perception to action. Simply said, this is the same as human behavior, input sensing information such as vision, and then, direct output action through the deep neural network. Deep reinforcement learning has the potential to learn a variety of skills for the robot to achieve full autonomy. Even though reinforcement learning is practiced successfully, but feature states need to manually set, for complex scene is a difficult thing, especially easy to cause the dimension disaster, and expression is not good. In 2010, Sascha Lange and Martin Riedmiller proposed deep auto-encoder neural networks in reinforcement learning to extract feature, which is used to control the visual correlation. In 2013, DeepMind proposed deep Q-network (DQN) in NIPS 2013, using convolution neural network to extract features, and then applied in reinforcement learning. They continue to improve and published an improved version of DQN on Nature in 2015, which has aroused widespread concern. In order to break through the limits of learning by machines, cognitive machine learning is proposed, which is the combination of machine learning and brain cognition, so that the machine intelligence is constantly evolving, and gradually reaches the human level of artificial intelligence. A cognitive model entitled Consciousness And Memory (CAM) is proposed by author, which consists of memory, consciousness, high-level cognitive functions, perception and motor. High-level cognitive functions of the brain include learning, language, thinking, decision making, emotion, and so on. Learning is a course to accept the stimulus through the nervous system and obtain new behavior, habits and accumulation experience. According to the current research progress of brain science and cognitive science, cognitive machine learning may be interested in learning emergence, procedural memory knowledge learning, learning evolution and so on. For intelligence, so-called evolution is refers to the learning of learning and the structure also follows the change. It is important to record the learning result by structure changing and improve the learning method.

Deep reinforcement learning (DRL) has many achievements in playing games and robotic control. However, transfer learning, a normal way of training a deep neural network for computer vision tasks, cannot be directly applied to DRL. This is an issue in the field of DRL. Many methods are proposed to address this problem. In this paper, a new method based on the proximal policy optimization (PPO) algorithm for learning similar tasks is proposed. The method uses a function to transform the trajectory from the previous task to the current task, both of those are similar. As shown in the result, our method can bring a more stable performance to our model at the early stage of training with compare to the model trained by using PPO without our method.

Reinforcement learning is a mathematical framework for agents to interact intelligently with their environment. Unlike supervised learning, where a system learns with the help of labeled data, reinforcement learning agents learn how to act by trial and error only receiving a reward signal from their environments. A field where reinforcement learning has been prominently successful is robotics [3]. However, real-world control problems are also particularly challenging because of the noise and high- dimensionality of input data (e.g., visual input). In recent years, in the field of supervised learning, deep neural networks have been successfully used to extract meaning from this kind of data. Building on these advances, deep reinforcement learning was used to solve complex problems like Atari games and Go. Mnih et al. [1] built a system with fixed hyper parameters able to learn to play 49 different Atari games only from raw pixel inputs. However, in order to apply the same methods to real-world control problems, deep reinforcement learning has to be able to deal with continuous action spaces. Discretizing continuous action spaces would scale poorly, since the number of discrete actions grows exponentially with the dimensionality of the action. Furthermore, having a parametrized policy can be advantageous because it can generalize in the action space. Therefore with this thesis we study state-of-the-art deep reinforcement learning algorithm, Deep Deterministic Policy Gradients. We provide a theoretical comparison to other popular methods, an evaluation of its performance, identify its limitations and investigate future directions of research. The remainder of the thesis is organized as follows. We start by introducing the field of interest, machine learning, focusing our attention of deep learning and reinforcement learning. We continue by describing in details the two main algorithms, core of this study, namely Deep Q-Network (DQN) and Deep Deterministic Policy Gradients (DDPG). We then provide implementatory details of DDPG and our test environment, followed by a description of benchmark test cases. Finally, we discuss the results of our evaluation, identifying limitations of the current approach and proposing future avenues of research.

Optimization of the Operation and Maintenance of Renewable Energy Systems by Deep Reinforcement Learning

The usage of distributed Peer-to-Peer (P2P) networks has been growing steadily for a reasonable period. Various applications rely on the infrastructure of P2P networks, where nodes communicate to accomplish a task without the need for a central authority. One of the significant challenges in P2P networks is the ability of the network nodes to reach a consensus on a shared item; the challenge increases as time passes. Thus, this work proposes a new effective method for tweaking the Deep Reinforcement Learning (DRL) algorithm to train Deep Q Network (DQN) learning agents to reach a consensus among the P2P nodes. We propose various hierarchies of deep agents to address this crucial challenge in P2P networks. DRL is utilized to build and train agents; more precisely, DQN learning agents are constructed and trained. Two scenarios are proposed and evaluated. In the first scenario, one DQN agent is trained to find the consensus between the network nodes. In the second scenario, three hierarchies with different numbers of layers of agents are proposed and evaluated. In both scenarios, the P2P network used is a blockchain network. The best result was obtained using the third hierarchy of the second scenario; the overall accuracy of the model is 87.8%.

Sustainable agricultural intensification necessitates precise deficit irrigation strategies to address global water scarcity. However, optimizing intra-seasonal scheduling under stochastic climatic conditions remains a complex control problem. While Deep Reinforcement Learning (DRL) offers a promising approach to flexible decision-making, existing applications often exhibit instability due to high-dimensional state spaces and an inability to enforce physical constraints. This study advances state-of-the-art irrigation control by proposing and benchmarking a tailored DRL framework based on Proximal Policy Optimization (PPO), coupled with the AquaCrop-OSPy simulation model. Moving beyond standard implementations, the research introduces specific enhancements to the learning agent: a reduced observation space limited to five causal biophysical variables, action masking to ensure strict adherence to seasonal water quotas, and a dense reward function based on transpiration efficiency. To rigorously quantify the value of information, the proposed approach is benchmarked against both a standard DRL baseline and a global Evolutionary Algorithm configured with perfect foresight of future weather events. This "oracle" defines the theoretical upper bound of achievable crop water productivity. Experimental validation on maize cultivation under deterministic and stochastic scenarios (Tunis and Nebraska) demonstrates that the proposed agent effectively navigates the trade-off between conservation and yield. The enhanced agent captures approximately 93.5% of the theoretical yield potential defined by the oracle, indicating a minimal performance penalty for the lack of future weather knowledge. Conversely, the standard reference implementation failed to converge under tight resource constraints. Economically, the proposed strategy not only stabilizes yields during extreme drought years but also increases mean net profits by up to 66% compared to the baseline. These findings confirm that integrating domain knowledge through action masking and feature selection transforms DRL into a robust tool for near-optimal irrigation scheduling without requiring extensive weather forecasting.