Comparing the Performance of PPO and DQN Algorithms in Different Game Environments Using a Reinforcement Learning Approach

By addressing the drawbacks of static optimization techniques, this research seeks to improve the dynamic energy management of thermoelectric generators (TEGs). Finding the best deep reinforcement learning (DRL) algorithm to maximize energy distribution, prolong battery life, and boost system efficiency in the face of the variable conditions present in waste heat recovery systems is the goal. The TEG system was modeled using the Markov decision process and implemented in a computer-simulated environment. Three advanced DRL algorithms were used: soft actor-critic (SAC), proximal policy optimization (PPO), and deep Q-networks (DQN); which were trained to act as intelligent controllers. The performance of each algorithm was systematically evaluated and compared using key metrics, including average cumulative reward, battery health, system efficiency, and a novel metric termed the energy fulfillment rate, which measures the ability to meet demand while storing surplus energy. The comparative analysis revealed a critical trade-off between maximizing performance and ensuring hardware longevity. The SAC algorithm demonstrated the best overall performance, achieving the highest average reward (− 7.03) and energy fulfillment rate (22.84%). However, the A2C, DDPG, and PPO algorithms all achieved a perfect average battery health of 100.00%, highlighting their superior capability for preserving system longevity, albeit with slightly lower rewards. The DQN algorithm consistently showed the least effective performance across all metrics, particularly in maintaining battery health (60.73%). The SAC algorithm is the most suitable of the methods tested for dynamically managing TEG systems. Its underlying principle of maximization of entropy enables a better exploration of control strategies, leading to a better balance between immediate energy dispatch and long-term storage goals. The findings confirm the significant potential of DRL to create efficient and adaptive controllers for renewable energy applications, although further validation of physical hardware is required to confirm real-world viability.Supplementary InformationThe online version contains supplementary material available at 10.1038/s41598-025-27210-7.

Reinforcement Learning (RL) has become one of the most powerful methods in Artificial Intelligence (AI) which allows a system to learn as it is engaged in an interaction with its environment and to maximize its actions by performing a trial and error process. Although classical RL algorithms can give us a fundamental idea on RL, like the Q-Learning, its usefulness wanes in high-dimensional or continuous-state spaces. To overcome these shortcomings, Deep Reinforcement Learning (DRL) algorithms including Deep Q-Networks (DQN) build on neural networks to learn action-value functions, allowing horizontal scalability and more efficient learning. This paper reports on the experimental comparison of Q-Learning and DQN based on the environment CartPole-v1. Both of the algorithms were trained in the controlled environment and tested under various metrics that include cumulative rewards, success rate, convergence speed, safety (failure rate), and path efficiency. Findings indicate that Q-Learning moves towards moderate stability with slower convergence but DQN takes shorter learning time, is more reliable and performs better. Such results indicate the ground-breaking potential of the DRL in the progress of AI toward transcending classical RL restrictions. Future research topics encompass implementation of more challenging experimental settings, a test of other DRL algorithms (e.g., PPO, SAC), and generalization, scalability and safety in real-world environments.

Great attention has been attracted to the growing energy consumption of the subway system. It is significant to investigate the energy-efficient train control methodology which is regarded as an effective way to cut down the energy cost of the subway system. In this paper, a soft actor-critic-based method is proposed to determine the optimal driving strategy. Firstly, the inverse problem of the energy-efficient train control problem, i.e., minimizing the trip time with constant traction energy, is presented, and it is converted to a finite Markov decision process which can be solved with deep reinforcement learning (DRL) algorithms. Secondly, the soft actor-critic (SAC) algorithm is proposed to determine the optimal driving strategy. Finally, the effectiveness of the proposed method is verified via the case studies. It is also illustrated that the SAC-based method has a good performance in robustness and stability. Moreover, compared with other DRL algorithms such as the deep Q network (DQN), the SAC-based method saves about 66% training steps on average to achieve the optimum, which reduces the computational resources.

Blockchains have proven to provide a high level of performance in terms of security and reliability for various applications like cryptocurrencies and Internet-of-Things (IoT). Prism is a recent blockchain algorithm that achieves the physical limit on throughput and latency without compromising security. In recent days, reinforcement learning approaches are investigated in traditional blockchains, to improve performance. In this work, we apply Deep Reinforcement Learning (DRL) to one of the promising blockchain protocols, Prism, to optimize its performance. We propose a Deep Reinforcement Learning-based Prism Blockchain (DRLPB) scheme which dynamically optimizes the parameters of the Prism blockchain and helps in achieving a better performance. In DRLPB, we apply two widely used DRL algorithms, Dueling Deep Q Networks (DDQN) and Proximal Policy Optimization (PPO). This work presents a novel approach to applying DDQN and PPO to a blockchain protocol and comparing the performance. The DRLPB scheme adapts the Prism blockchain parameters to enhance the number of votes upto 84% more than Prism, while still preserving the security and latency performance guarantees of Prism.

Proximal policy optimization via enhanced exploration efficiency

A present challenge is the planning of a resilient power distribution system (PDS) as to energize critical loads after weather events. To that end, this paper sets forth: (i) A formulation for planning of resilient PDS that considers the formation of microgrids when isolating faulted lines. (ii) An approach to train a constrained-policy reinforcement learning (CPRL) planner that uses only two deep neural networks in lieu of one per constraint. These novelties are showcased using a 17-node and a 60-node PDS while contrasting against proximal policy optimization (PPO), deep Q-network (DQN), and greedy (GRD) algorithms. The CPRL planner achieved the lowest average unserved demand across stochastic scenarios and the lowest constraint violation when contrasted against PPO, DQN, and GRD methods.

Reinforcement learning-based real-time optimization of friction stir welding parameters for copper–aluminium dissimilar interfaces

This paper proposes a hybrid Unmanned Aerial Vehicle (UAV) path planning method that combines the Rapidly-exploring Random Tree (RRT) algorithm with Proximal Policy Optimization (PPO). The proposed method aims to enhance the efficiency and adaptability of UAV path planning in complex and dynamic environments. The RRT algorithm excels at quickly generating a feasible path from a start point to a goal point. However, the quality of its paths is often suboptimal, and it lacks adaptability in dynamic settings. In contrast, PPO, a deep reinforcement learning algorithm, optimizes paths through iterative policy updates, enabling the UAV to adapt to environmental changes. Our approach first employs RRT to generate an initial path, which is subsequently refined by PPO to improve smoothness and adaptability. Experimental results demonstrate that the RRT-PPO hybrid method performs favorably in terms of path length, computational time, and obstacle avoidance capability, effectively improving the task completion efficiency of UAVs in complex environments.

Advanced algorithms can promote the development of energy management strategies (EMSs) as a key technology in hybrid electric vehicles (HEVs). Reinforcement learning (RL) with distributed structure can significantly improve training efficiency in complex environments, and multi-threaded parallel computing provides a reliable algorithm basis for promoting adaptability. Dedicated to trying more efficient deep reinforcement learning (DRL) algorithms, this paper proposed a deep q-network (DQN)-based energy and emission management strategy (E&EMS) at first. Then, two distributed DRL algorithms, namely asynchronous advantage actor-critic (A3C) and distributed proximal policy optimization (DPPO), were adopted to propose EMSs, respectively. Finally, emission optimization was taken into account and then distributed DRL-based E&EMSs were proposed. Regarding dynamic programming (DP) as the optimal benchmark, simulation results show that three DRL-based control strategies can achieve near-optimal fuel economy and outstanding computational efficiency, and compared with DQN, two distributed DRL algorithms have improved the learning efficiency by four times.

Abstract: Online resource management in cloud computing must respond to rapidly changing demand and system conditions, often exceeding the adaptability of static or periodically recomputed optimisation plans. This study formulates cloud resource management as a Markov decision process (MDP) and investigates deep reinforcement learning (DRL) as an approximate dynamic programming (ADP) approach for deriving adaptive control policies. The system state encompasses workload, capacity utilisation, and performance indicators, while actions govern virtual machine placement, scaling, and server power-state transitions. Reward functions combine energy and operating costs with penalties associated with service-level agreement violations. DRL algorithms, including deep Q-networks, advantage actor–critic, and proximal policy optimization, are implemented in PyTorch and trained through interaction with a CloudSim-based simulation environment calibrated to realistic load and power profiles. The learned policies are evaluated against deterministic and scenario-based stochastic optimisation baselines using metrics such as average cost, energy consumption, SLA violations, and robustness to non-stationary demand. Results show that DRL policies can approach or surpass the performance of stochastic optimisation solutions in highly dynamic scenarios, while maintaining feasible control actions. The study demonstrates how DRL can be positioned within an OR/ADP framework and highlights conditions under which learning-based methods offer significant advantages for online cloud control. Keywords: deep reinforcement learning, approximate dynamic programming, cloud computing, Markov decision processes, online resource management, energy-aware control, PyTorch, simulation

Device‐to‐device (D2D) communication is a promising development in 5G networks, offering potential benefits such as increased data rates, reduced costs and latency, and improved energy efficiency (EE). This study analyzes the operation of millimeter‐wave (mmWave) in cellular networks. A client's device can establish a connection to either a base station or another client, facilitating D2D communication based on a distance threshold and accounting for interference. The research employs a deep reinforcement learning (DRL)–based resource allocation (RA) scheme for D2D‐enabled mmWave communications underlaying cellular networks. It evaluates the effectiveness of several metrics: coverage probability, area spectral efficiency, and network EE. Among networks limited by noise, the proposed strategy demonstrates the highest coverage probability performance. The paper also suggests an optimization approach based on the firefly algorithm for RA, taking into account the stochastic nature of wireless channels. An asynchronous advantage actor–critic (A3C) DRL algorithm is modeled for this purpose. The performance of the proposed scheme is compared with two existing algorithms: soft actor–critic and proximal policy optimization. Overall, the numerical results indicate that our proposed firefly algorithm–optimized A3C method outperforms the other analytical methods.

The transition to sustainable and intelligent energy systems has intensified the development of smart microgrids, which offer decentralized, resilient, and efficient power solutions. This review critically examines the integration of Artificial Intelligence (AI) and Deep Reinforcement Learning (DRL) into smart microgrid platforms, focusing on their role in optimizing sustainable energy management. Traditional energy management systems often struggle to adapt to the dynamic nature of modern energy demands, renewable energy intermittency, and grid complexity. AI-driven solutions, particularly DRL, provide adaptive, autonomous, and data-driven mechanisms for real-time decision-making and predictive control within microgrids. DRL, by learning optimal policies through interaction with the environment, is capable of handling multi-objective problems, including demand-response optimization, energy storage control, load forecasting, and distributed generation scheduling. This paper synthesizes recent advancements and applications of DRL algorithms such as Deep Q-Networks (DQN), Deep Deterministic Policy Gradient (DDPG), and Proximal Policy Optimization (PPO) in smart microgrids. It also explores hybrid models that combine DRL with other AI techniques, such as fuzzy logic and neural networks, to improve performance under uncertainty and nonlinearity. Furthermore, the review evaluates benchmark testbeds, simulation tools, and real-time platforms used to implement and validate these intelligent systems. Challenges such as high computational costs, model generalizability, real-time implementation issues, and cybersecurity vulnerabilities are discussed. The study also highlights recent research trends emphasizing decentralized control, edge computing, and federated learning to enhance scalability and privacy in DRL-based microgrid applications. Future directions suggest the need for explainable AI (XAI), robust training environments, and standardization to facilitate the wider adoption of DRL in real-world microgrid systems. The integration of DRL into smart microgrids represents a transformative shift toward resilient, efficient, and sustainable energy ecosystems. This review offers a comprehensive understanding of how AI and DRL are revolutionizing energy management, providing a foundation for future research and practical deployment in smart energy infrastructures.

Autonomous drone racing has emerged as a challenging benchmark for high-speed robotic perception, decision-making, and control, requiring systems to operate at the limits of agility and safety in complex, dynamic environments. This work presents an abstract overview of autonomous drone racing using deep reinforcement learning (DRL), where an agent learns end-to-end control policies that map raw sensory inputs directly to flight commands through trial-and-error interactions with the environment. Unlike traditional model-based or trajectory optimization approaches that rely on accurate system models and prior knowledge of the racing track, DRL enables drones to learn adaptive strategies that generalize across varying track layouts, gate configurations, and environmental conditions. By formulating drone racing as a sequential decision-making problem, the learning agent is trained to maximize a reward function that balances speed, stability, collision avoidance, and successful gate traversal. Advanced DRL algorithms such as Deep Q-Networks, Proximal Policy Optimization, and Soft Actor-Critic are commonly employed to handle continuous control, high-dimensional state spaces, and stochastic dynamics. Training is typically conducted in high-fidelity simulation environments that incorporate realistic aerodynamics, sensor noise, and delays, followed by sim-to-real transfer techniques to deploy learned policies on physical drones. The use of domain randomization, curriculum learning, and imitation learning from expert demonstrations further improves training efficiency and robustness. Experimental results reported in recent studies demonstrate that DRL-based autonomous drones can achieve competitive lap times, smooth trajectories, and resilient performance under disturbances, often rivaling or surpassing traditional planning-based methods. Overall, deep reinforcement learning provides a scalable and flexible framework for autonomous drone racing, advancing the development of intelligent, high-speed aerial robots and contributing valuable insights applicable to broader domains such as autonomous navigation, search and rescue, and agile robotic

Energy efficiency is a key to reduced carbon footprint, savings on energy bills, and sustainability for future generations. For instance, in hot climate countries such as Qatar, buildings are high energy consumers due to air conditioning that resulted from high temperatures and humidity. Optimizing the building energy management system will reduce unnecessary energy consumptions, improve indoor environmental conditions, maximize building occupant’s comfort, and limit building greenhouse gas emissions. However, lowering energy consumption cannot be done despite the occupants’ comfort. Solutions must take into account these tradeoffs. Conventional Building Energy Management methods suffer from a high dimensional and complex control environment. In recent years, the Deep Reinforcement Learning algorithm, applying neural networks for function approximation, shows promising results in handling such complex problems. In this work, a Deep Reinforcement Learning agent is proposed for controlling and optimizing a school building’s energy consumption. It is designed to search for optimal policies to minimize energy consumption, maintain thermal comfort, and reduce indoor contaminant levels in a challenging 21-zone environment. First, the agent is trained with the baseline in a supervised learning framework. After cloning the baseline strategy, the agent learns with proximal policy optimization in an actor-critic framework. The performance is evaluated on a school model simulated environment considering thermal comfort, CO2 levels, and energy consumption. The proposed methodology can achieve a 21% reduction in energy consumption, a 44% better thermal comfort, and healthier CO2 concentrations over a one-year simulation, with reduced training time thanks to the integration of the behavior cloning learning technique.

In this paper, we apply the Proximal Policy Optimization (PPO) algorithm in intelligent traffic signal control at a single intersection with eight lanes and four signal phases. The optimization goal is to minimize the average waiting time of vehicles so as to improve the traffic efficiency of the intersection. Extensive experiments are conducted in Simulation of Urban MObility (SUMO) to evaluate the performance of the proposed algorithm, and compare it with other classic algorithms including Deep Q-network (DQN), Advantage Actor Critic (A2C) and Fixed Time. Simulation results show that the proposed PPO algorithm outperforms the others under various traffic scenarios to different extent. The performance gain is significant under unbalanced traffic where one direction is saturated while the other is not, and becomes marginal when all the directions are saturated or unsaturated. PPO also demonstrates good portability and robustness over time-varying traffic patterns, while implies it could be a preferable option for implementation in real world intelligent traffic signal control systems.