A trustworthy reinforcement learning framework for autonomous control of a large-scale complex heating system: Simulation and field implementation

An adaptive PID controller for path following of autonomous underwater vehicle based on Soft Actor–Critic

With the continuous progress of UAV technology and the rapid development of mobile edge computing (MEC), the UAV-assisted MEC system has shown great application potential in special fields such as disaster rescue and emergency response. However, traditional deep reinforcement learning (DRL) decision-making methods suffer from limitations such as difficulty in balancing multiple objectives and training convergence when making mixed action space decisions for UAV path planning and task offloading. This article innovatively proposes a hybrid decision framework based on the improved Dynamic Adaptive Genetic Optimization Algorithm (DAGOA) and soft actor–critic with hierarchical action decomposition, an uncertainty-quantified critic ensemble, and adaptive entropy temperature, where DAGOA performs an effective search and optimization in discrete action space, while SAC can perform fine control and adjustment in continuous action space. By combining the above algorithms, the joint optimization of drone path planning and task offloading can be achieved, improving the overall performance of the system. The experimental results show that the framework offers significant advantages in improving system performance, reducing energy consumption, and enhancing task completion efficiency. When the system adopts a hybrid decision framework, the reward score increases by a maximum of 153.53% compared to pure deep reinforcement learning algorithms for decision-making. Moreover, it can achieve an average improvement of 61.09% on the basis of various reinforcement learning algorithms such as proposed SAC, proximal policy optimization (PPO), deep deterministic policy gradient (DDPG), and twin delayed deep deterministic policy gradient (TD3).

Deep Reinforcement Learning (DRL) has recently become a powerful tool for addressing complex optimization problems in mobile networks. Among DRL methods, Soft Actor-Critic (SAC), an off-policy algorithm, stands out due to its sample efficiency and ability to learn stable and resilient policies. This work explores the application of SAC to the resource allocation problem in Open Radio Access Networks (O-RAN) with stringent Quality of Service (QoS) requirements. The performance of SAC is benchmarked against Proximal Policy Optimization (PPO), a strong on-policy DRL algorithm, and a greedy baseline. Experimental results show that while both DRL methods outperform the greedy strategy, SAC consistently achieves faster convergence and better long-term stability compared to PPO, making it highly suitable for dynamic O-RAN environments.

Improving the interpretability of deep reinforcement learning in urban drainage system operation

Reinforcement learning and deep reinforcement learning, as a research hotspot in the field of machine learning, have been widely used in our daily life. In this field, game is playing an extremely important role in the developing of reinforcement learning algorithms. Based on the Tennis environment built by Unity ML Agents, this paper used three algorithms, Proximal Policy Optimization (PPO), Multi-Agent Deep Deterministic Policy Gradients (MADDPG) and Soft Actor-Critic (SAC), combined with PyTorch framework, solved the continuous control problem of this environment. Meanwhile, a group of optimal parameters are obtained through multiple trainings, so that Agents can achieve a perfect effect of solving the continuous control problem in Tennis environment. At the end, this paper compared and analyzed the difference among these three algorithms, summarized the application and properties of each algorithm. For different parameters of the algorithm, this paper also maked a comparison and explained the reasons for some special cases as well which can be used for the future work.

The control of quadrotors is inherently difficult due to their underactuated nature and highly coupled nonlinear dynamics. Although traditional control strategies often depend on model linearization, Deep Reinforcement Learning (DRL) has emerged as a robust, model-free approach suitable for handling complex aerodynamics and environmental uncertainties. Despite this potential, the identification of the most effective learning agent for specific flight regimes remains unresolved. This study conducts a comprehensive comparative assessment of three prominent algorithms: Proximal Policy Optimization (PPO), Soft Actor-Critic (SAC), and Deep Deterministic Policy Gradient (DDPG). Through rigorous simulation, we evaluate these algorithms based on convergence speed, sample efficiency, and steady-state tracking accuracy. Our findings reveal a distinct trade-off: while SAC exhibits faster convergence in early training phases, PPO ensures superior stability and reduced variance during aggressive maneuvering in the steady state. Additionally, the resilience of these policies to sensor noise and parametric uncertainty is investigated. By elucidating the performance disparities between on-policy and off-policy algorithms, this work establishes a practical framework for selecting DRL architectures for autonomous quadrotor deployment.

By addressing the drawbacks of static optimization techniques, this research seeks to improve the dynamic energy management of thermoelectric generators (TEGs). Finding the best deep reinforcement learning (DRL) algorithm to maximize energy distribution, prolong battery life, and boost system efficiency in the face of the variable conditions present in waste heat recovery systems is the goal. The TEG system was modeled using the Markov decision process and implemented in a computer-simulated environment. Three advanced DRL algorithms were used: soft actor-critic (SAC), proximal policy optimization (PPO), and deep Q-networks (DQN); which were trained to act as intelligent controllers. The performance of each algorithm was systematically evaluated and compared using key metrics, including average cumulative reward, battery health, system efficiency, and a novel metric termed the energy fulfillment rate, which measures the ability to meet demand while storing surplus energy. The comparative analysis revealed a critical trade-off between maximizing performance and ensuring hardware longevity. The SAC algorithm demonstrated the best overall performance, achieving the highest average reward (− 7.03) and energy fulfillment rate (22.84%). However, the A2C, DDPG, and PPO algorithms all achieved a perfect average battery health of 100.00%, highlighting their superior capability for preserving system longevity, albeit with slightly lower rewards. The DQN algorithm consistently showed the least effective performance across all metrics, particularly in maintaining battery health (60.73%). The SAC algorithm is the most suitable of the methods tested for dynamically managing TEG systems. Its underlying principle of maximization of entropy enables a better exploration of control strategies, leading to a better balance between immediate energy dispatch and long-term storage goals. The findings confirm the significant potential of DRL to create efficient and adaptive controllers for renewable energy applications, although further validation of physical hardware is required to confirm real-world viability.Supplementary InformationThe online version contains supplementary material available at 10.1038/s41598-025-27210-7.

In the domain of continuous control, deep reinforcement learning (DRL) demonstrates promising results. However, the dependence of DRL on deep neural networks (DNNs) results in the demand for extensive data and increased computational cost. To address this issue, a novel hybrid actor-critic reinforcement learning (RL) framework is introduced. The proposed framework integrates the broad learning system (BLS) with DNN, aiming to merge the strengths of both distinct architectural paradigms. Specifically, the critic network employs BLS for rapid value estimation via ridge regression, while the actor network retains the DNN structure to optimize policy gradients. This hybrid design is generalizable and can enhance existing actor-critic algorithms. To demonstrate its versatility, the proposed framework is integrated into three widely used actor-critic algorithms-deep deterministic policy gradient (DDPG), soft actor-critic (SAC), and twin delayed DDPG (TD3), resulting in BLS-augmented variants. The experimental results reveal that all BLS-enhanced versions surpass their original counterparts in terms of training efficiency and accuracy. These improvements highlight the suitability of the proposed framework for real-time control scenarios, where computational efficiency and rapid adaptation are critical.

The transition to sustainable and intelligent energy systems has intensified the development of smart microgrids, which offer decentralized, resilient, and efficient power solutions. This review critically examines the integration of Artificial Intelligence (AI) and Deep Reinforcement Learning (DRL) into smart microgrid platforms, focusing on their role in optimizing sustainable energy management. Traditional energy management systems often struggle to adapt to the dynamic nature of modern energy demands, renewable energy intermittency, and grid complexity. AI-driven solutions, particularly DRL, provide adaptive, autonomous, and data-driven mechanisms for real-time decision-making and predictive control within microgrids. DRL, by learning optimal policies through interaction with the environment, is capable of handling multi-objective problems, including demand-response optimization, energy storage control, load forecasting, and distributed generation scheduling. This paper synthesizes recent advancements and applications of DRL algorithms such as Deep Q-Networks (DQN), Deep Deterministic Policy Gradient (DDPG), and Proximal Policy Optimization (PPO) in smart microgrids. It also explores hybrid models that combine DRL with other AI techniques, such as fuzzy logic and neural networks, to improve performance under uncertainty and nonlinearity. Furthermore, the review evaluates benchmark testbeds, simulation tools, and real-time platforms used to implement and validate these intelligent systems. Challenges such as high computational costs, model generalizability, real-time implementation issues, and cybersecurity vulnerabilities are discussed. The study also highlights recent research trends emphasizing decentralized control, edge computing, and federated learning to enhance scalability and privacy in DRL-based microgrid applications. Future directions suggest the need for explainable AI (XAI), robust training environments, and standardization to facilitate the wider adoption of DRL in real-world microgrid systems. The integration of DRL into smart microgrids represents a transformative shift toward resilient, efficient, and sustainable energy ecosystems. This review offers a comprehensive understanding of how AI and DRL are revolutionizing energy management, providing a foundation for future research and practical deployment in smart energy infrastructures.

In the vehicle-to-everything scenario, the fuel cell bus can accurately obtain the surrounding traffic information, and quickly optimize the energy management problem while controlling its own safe and efficient driving. This paper proposes an energy management strategy (EMS) that considers speed control based on deep reinforcement learning (DRL) in complex traffic scenarios. Using SUMO simulation software (Version 1.15.0), a two-lane urban expressway is designed as a traffic scenario, and a hydrogen fuel cell bus speed control and energy management system is designed through the soft actor–critic (SAC) algorithm to effectively reduce the equivalent hydrogen consumption and fuel cell output power fluctuation while ensuring the safe, efficient and smooth driving of the vehicle. Compared with the SUMO–IDM car-following model, the average speed of vehicles is kept the same, and the average acceleration and acceleration change value decrease by 10.22% and 11.57% respectively. Compared with deep deterministic policy gradient (DDPG), the average speed is increased by 1.18%, and the average acceleration and acceleration change value are decreased by 4.82% and 5.31% respectively. In terms of energy management, the hydrogen consumption of SAC–OPT-based energy management strategy reaches 95.52% of that of the DP algorithm, and the fluctuation range is reduced by 32.65%. Compared with SAC strategy, the fluctuation amplitude is reduced by 15.29%, which effectively improves the durability of fuel cells.

Deep reinforcement learning (DRL) is one of the emerging areas of machine learning which focuses on maximized rewards. DRL is a type of machine learning that combines reinforcement learning and deep learning. It uses a series of algorithms to enable an agent to learn how to make decisions in a complex environment. DRL is a subset of artificial intelligence that focuses on making decisions based on the environment and the rewards associated with each action.The goal of DRL is to maximize the long-term reward of an agent. In order to do this, the agent must use a combination of deep learning, reinforcement learning and other AI techniques to learn which actions will lead to the highest reward. DRL is used to solve a variety of problems, from playing video games to controlling robots. It is also used in autonomous driving and robotics, as well as for financial trading. DRL is a powerful tool for solving complex problems and has been used in a variety of research projects. DRL has the potential to revolutionize the way we interact with machines and the environment.

In this paper, we propose a deep reinforcement learning (DRL) based predictive control scheme for reducing the energy consumption and energy cost of pumping systems in wastewater treatment plants (WWTP), in which the pumps are operated in a binary mode, using on/off signals. As global energy consumption increases, the efficient operation of energy-intensive facilities has also become important. A WWTP in Busan, Republic of Korea is used as the target of this study. This WWTP is a large energy-consuming facility, and the pumping station accounts for a significant portion of the energy consumption of the WWTP. The framework of the proposed scheme consists of a deep neural network (DNN) model for forecasting wastewater inflow and a DRL agent for controlling the on/off signals of the pumping system, where proximal policy optimization (PPO) and deep Q-neural network (DQN) are employed as the DRL agents. To implement smart control with DRL, a reward function is designed to consider the energy consumption amount and electricity price information. In particular, new features and penalty factors for pump switching, which are essential for preventing pump wear, are also considered. The performance of our designed DRL agents is compared with those of WWTP experts and conventional approaches such as scheduling method and model predictive control (MPC), in which integer linear programming (ILP) optimization is employed. Results show that the designed agents outperform the other approaches in terms of compliance with operating rules and reducing energy costs.

Magnetic levitation (Maglev) technology is widely adopted for transportation, photolithography, clean room, and material-handling applications, owing to its vibration- and friction-free, untethered motion. The magnetic movers are actuated using the stator coil array, which functions as mobile maglev robots (MLRs) with precision position and orientation motion control. Therefore, this article studies the MLRs for intelligent manufacturing, such as material handling and collaboration with manufacturing components, by integrating them with deep reinforcement learning (DRL) policy controllers. Linear DRL policies are developed and compared between Soft Actor-Critic (SAC) and Proximal Policy Optimization (PPO) for the horizontal motion control, which takes 455 μs to compute the actuating forces for real-time operations. During policy training, random magnitude horizontal step responses are ticked every 500 ms to explore and exploit the dynamic capacity of the MLRs. The PPO policies are finalized based on the discovered deterministic nature of the maglev environment. Furthermore, the symmetry of the MLR configuration is utilized for policy training, with a threshold boundary included to enable MLRs to translate smoothly across the horizontal operating range. The trained policies exhibit aggressive oscillations and persistent offsets relative to references, respectively. To alleviate the fitting problem, distance-based soft state machines are introduced to integrate the DRL policy with the basic controller. The state machine parameters are empirically tuned using various location step responses. This feature enables the dynamic responses of the DRL policy to be combined with the precision responses at references. The soft state machines are also trained as auxiliary policy actions and evaluated using full operating-range trajectory tracking and long-stroke step responses alongside the policy and the basic controller. The DRL policy controllers outperform the basic controllers, with improved rise and settling times in step responses, while showing enhanced performance in tracking trajectories. Finally, the policy robustness is empirically evaluated.

In recent years, deep reinforcement learning (DRL) has made impressive achievements in many fields. However, existing DRL algorithms usually require a large amount of exploration to obtain a good action policy. In addition, in many complex situations, the reward function cannot be well designed to meet task requirements. These two problems will make it difficult for DRL to learn a good action policy within a relatively short period. The use of expert data can provide effective guidance and avoid unnecessary exploration. This study proposes a deep imitation reinforcement learning (DIRL) algorithm that uses a certain amount of expert demonstration data to speed up the training of DRL. In the proposed method, the learning agent imitates the expert's action policy by learning from demonstration data. After imitation learning, DRL is used to optimise the action policy in a self‐learning way. By experimental comparison on a video game called the Mario racing game, it is shown that the proposed DIRL algorithm with expert demonstration data can obtain much better performance than previous DRL algorithms without expert guidance.

The rapid advancement of artificial intelligence (AI) has accelerated the semiconductor industry, driving chip performance requirements to unprecedented levels. However, conventional system-on-chip (SoC) designs are increasingly constrained by the physical and economic limits of monolithic scaling, where continued transistor miniaturization yields diminishing returns and rising fabrication costs. To address these challenges, the industry has shifted toward chiplet-based heterogeneous integration and advanced packaging technologies. By interconnecting multiple specialized chiplets through high-density, high-bandwidth interfaces, this modular approach improves scalability, reusability, and cost efficiency while enabling faster time-to-market. However, chiplet-based systems introduce significant design complexity, particularly in signal integrity (SI) and power integrity (PI), where highly constrained interconnects and power delivery networks must meet stringent electrical requirements. Traditional design methodologies rely heavily on expert-driven manual tuning and extensive full-wave electromagnetic simulations using electronic design automation (EDA) tools, resulting in long design cycles that can span weeks or months. To overcome these limitations, this thesis investigates the application of AI, particularly deep reinforcement learning (RL), to automate and optimize chiplet-based system design with a focus on SI and PI challenges in advanced packaging technologies. The research initially explores a supervised learning approach using convolutional neural networks (CNNs) to classify signal impairments from eye diagrams. While effective, this approach reveals a critical dependence on large labeled datasets, which are scarce in SI/PI domains, motivating a shift toward RL for autonomous design space exploration without labeled data. The first major contribution proposes an RL framework for automated power distribution network (PDN) design in a four-chiplet system. Two RL algorithms, Dueling Double Deep Q-Network (DDDQN) and Proximal Policy Optimization (PPO), are evaluated under different reward strategies. Results show that dense analytical rewards based on power domain squareness and bump proximity enable the RL agent to achieve lower self-impedance than human-designed PDNs, ensuring power integrity across 28 power domains. The complete PDN layout, which typically requires three months of manual effort, is generated in under one hour using a single GPU. The second contribution focuses on signal integrity optimization for chiplet interconnects in CoWoS redistribution layers (RDLs). An RL agent is trained using a reward function that balances eye width, voltage transfer function (VTF) crosstalk, and interconnect density. The agent explores design variations across data rates from 4 to 32 GT/s using a simulation environment integrated with commercial EDA tools. Experimental results demonstrate up to 24.67% improvement in eye width and over 22 dB reduction in VTF crosstalk while satisfying UCIe electrical specifications. The training process also generates labeled datasets for future supervised learning. The third contribution introduces an RL-based routing framework for UCIe bridge interconnects in advanced System-in-Package (SiP). Using Proximal Policy Optimization, the framework learns to generate routing layouts that comply with constraints such as ground-signal-ground structures, layer assignment, and congestion avoidance. It supports all UCIe module configurations and produces design rule check-compliant solutions within 100 training episodes, reducing design time from four weeks to less than one hour while improving signal distribution and reducing via usage. In summary, this thesis presents a unified AI-driven methodology for addressing SI and PI challenges in chiplet-based systems. The proposed RL frameworks significantly outperform traditional design approaches in both efficiency and quality, demonstrating the potential of deep RL to enable scalable and intelligent design automation for advanced packaging and chiplet-based architectures.