Deep reinforcement learning control for non-stationary building energy management

In this paper, we present a novel guidance scheme based on model-based deep reinforcement learning (RL) technique. With model-based deep RL method, a deep neural network is trained as a predictive model of guidance dynamics which is incorporated into a model predictive path integral (MPPI) control framework. However the traditional MPPI framework assumes the actual environment similar to the training dataset for the deep neural network which is impractical in practice with different maneuvering of target, other perturbations and actuator failures. To address this problem, our method utilize meta-learning technique to make the deep neural dynamics model adapt to such changes online. With this approach we can alleviate the performance deterioration of standard MPPI control caused by the difference between actual environment and training data. Then, a novel guidance law for a varying velocity interceptor intercepting maneuvering target with desired terminal impact angle under actuator failure is constructed based on aforementioned techniques. Simulation and experiment results under different cases show the effectiveness and robustness of the proposed guidance law in achieving successful interceptions of maneuvering target.

The heating, ventilation, and air conditioning (HVAC) system is a major energy consumer in office buildings, and its operation is critical for indoor thermal comfort. While previous studies have indicated that reinforcement learning control can improve HVAC energy efficiency, they did not provide enough information about end-to-end control (i.e., from raw observations to ready-to-implement control signals) for centralized HVAC systems in multizone buildings due to the limitations of reinforcement learning methods or the test buildings being single zones with independent HVAC systems. This study developed a model-free end-to-end dynamic HVAC control method based on a recently proposed deep reinforcement learning framework to control the centralized HVAC system of a multizone office building. By using the deep neural network, the proposed control method could directly take measurable parameters, including weather and indoor environment conditions, as inputs and control indoor temperature setpoints at a supervisory level. In some test cases, the proposed control method could successfully learn a dynamic control policy to reduce HVAC energy consumption by 12.8% compared with the baseline case using conventional control methods, without compromising thermal comfort. However, an over-fitting problem was noted, indicating that future work should first focus on the generalization of deep reinforcement learning.

This paper considers path following problem of an under-actuated unmanned surface vehicle (USV) being lack of model information. A fully data-driven path following algorithm is presented for the USV based on model-based reinforcement learning (RL) and model predictive path integral (MPPI) control. Specifically, a fully-connected neural network is trained to approximate the state transition model of the USV based on model-based RL. Next, based on the learned model, a sampling-based MPPI control is used to obtain the optimal actions. Simulation results demonstrate the performance of the proposed data-driven path following algorithm for the USV without using any model information.

The rapid advancement of artificial intelligence (AI) has accelerated the semiconductor industry, driving chip performance requirements to unprecedented levels. However, conventional system-on-chip (SoC) designs are increasingly constrained by the physical and economic limits of monolithic scaling, where continued transistor miniaturization yields diminishing returns and rising fabrication costs. To address these challenges, the industry has shifted toward chiplet-based heterogeneous integration and advanced packaging technologies. By interconnecting multiple specialized chiplets through high-density, high-bandwidth interfaces, this modular approach improves scalability, reusability, and cost efficiency while enabling faster time-to-market. However, chiplet-based systems introduce significant design complexity, particularly in signal integrity (SI) and power integrity (PI), where highly constrained interconnects and power delivery networks must meet stringent electrical requirements. Traditional design methodologies rely heavily on expert-driven manual tuning and extensive full-wave electromagnetic simulations using electronic design automation (EDA) tools, resulting in long design cycles that can span weeks or months. To overcome these limitations, this thesis investigates the application of AI, particularly deep reinforcement learning (RL), to automate and optimize chiplet-based system design with a focus on SI and PI challenges in advanced packaging technologies. The research initially explores a supervised learning approach using convolutional neural networks (CNNs) to classify signal impairments from eye diagrams. While effective, this approach reveals a critical dependence on large labeled datasets, which are scarce in SI/PI domains, motivating a shift toward RL for autonomous design space exploration without labeled data. The first major contribution proposes an RL framework for automated power distribution network (PDN) design in a four-chiplet system. Two RL algorithms, Dueling Double Deep Q-Network (DDDQN) and Proximal Policy Optimization (PPO), are evaluated under different reward strategies. Results show that dense analytical rewards based on power domain squareness and bump proximity enable the RL agent to achieve lower self-impedance than human-designed PDNs, ensuring power integrity across 28 power domains. The complete PDN layout, which typically requires three months of manual effort, is generated in under one hour using a single GPU. The second contribution focuses on signal integrity optimization for chiplet interconnects in CoWoS redistribution layers (RDLs). An RL agent is trained using a reward function that balances eye width, voltage transfer function (VTF) crosstalk, and interconnect density. The agent explores design variations across data rates from 4 to 32 GT/s using a simulation environment integrated with commercial EDA tools. Experimental results demonstrate up to 24.67% improvement in eye width and over 22 dB reduction in VTF crosstalk while satisfying UCIe electrical specifications. The training process also generates labeled datasets for future supervised learning. The third contribution introduces an RL-based routing framework for UCIe bridge interconnects in advanced System-in-Package (SiP). Using Proximal Policy Optimization, the framework learns to generate routing layouts that comply with constraints such as ground-signal-ground structures, layer assignment, and congestion avoidance. It supports all UCIe module configurations and produces design rule check-compliant solutions within 100 training episodes, reducing design time from four weeks to less than one hour while improving signal distribution and reducing via usage. In summary, this thesis presents a unified AI-driven methodology for addressing SI and PI challenges in chiplet-based systems. The proposed RL frameworks significantly outperform traditional design approaches in both efficiency and quality, demonstrating the potential of deep RL to enable scalable and intelligent design automation for advanced packaging and chiplet-based architectures.

Given that buildings consume approximately 33% of global energy, and HVAC systems contribute nearly half of a building’s total energy demand, optimizing their efficiency is imperative for sustainable energy use. Many existing buildings operate HVAC systems inefficiently, displaying non-stationary behavior. Current reinforcement learning (RL) training methods rely on historical data, which is often obtained through costly modeling or trial-and-error methods in real buildings. This paper introduces a novel reinforcement learning construction framework designed to improve the robustness and learning speed of RL control while reducing learning costs. The framework is specifically tailored for existing office buildings. Applying this framework to control HVAC systems in real office buildings in Beijing, engineering practice results demonstrate: during the data collection phase, energy efficiency surpasses traditional rule-based control methods from the previous year, achieving significantly improved energy performance (a 17.27% reduction) with minimal comfort sacrifices. The system achieves acceptable robustness, learning speed, and control stability. Reduced ongoing manual supervision leads to savings in optimization labor. Systematic exploration of actions required for RL training lays the foundation for RL algorithm development. Furthermore, by leveraging collected data, a reinforcement learning control algorithm is established, validating the reliability of this approach. This construction framework reduces the prerequisites for historical data and models, providing an acceptable alternative for systems with insufficient data or equipment conditions.

The transition to sustainable and intelligent energy systems has intensified the development of smart microgrids, which offer decentralized, resilient, and efficient power solutions. This review critically examines the integration of Artificial Intelligence (AI) and Deep Reinforcement Learning (DRL) into smart microgrid platforms, focusing on their role in optimizing sustainable energy management. Traditional energy management systems often struggle to adapt to the dynamic nature of modern energy demands, renewable energy intermittency, and grid complexity. AI-driven solutions, particularly DRL, provide adaptive, autonomous, and data-driven mechanisms for real-time decision-making and predictive control within microgrids. DRL, by learning optimal policies through interaction with the environment, is capable of handling multi-objective problems, including demand-response optimization, energy storage control, load forecasting, and distributed generation scheduling. This paper synthesizes recent advancements and applications of DRL algorithms such as Deep Q-Networks (DQN), Deep Deterministic Policy Gradient (DDPG), and Proximal Policy Optimization (PPO) in smart microgrids. It also explores hybrid models that combine DRL with other AI techniques, such as fuzzy logic and neural networks, to improve performance under uncertainty and nonlinearity. Furthermore, the review evaluates benchmark testbeds, simulation tools, and real-time platforms used to implement and validate these intelligent systems. Challenges such as high computational costs, model generalizability, real-time implementation issues, and cybersecurity vulnerabilities are discussed. The study also highlights recent research trends emphasizing decentralized control, edge computing, and federated learning to enhance scalability and privacy in DRL-based microgrid applications. Future directions suggest the need for explainable AI (XAI), robust training environments, and standardization to facilitate the wider adoption of DRL in real-world microgrid systems. The integration of DRL into smart microgrids represents a transformative shift toward resilient, efficient, and sustainable energy ecosystems. This review offers a comprehensive understanding of how AI and DRL are revolutionizing energy management, providing a foundation for future research and practical deployment in smart energy infrastructures.

Despite the fact that deep reinforcement learning (RL) has surpassed human-level performances in various tasks, it still has several fundamental challenges. First, most RL methods require intensive data from the exploration of the environment to achieve satisfactory performance. Second, the use of neural networks in RL renders it hard to interpret the internals of the system in a way that humans can understand. To address these two challenges, we propose a framework that enables an RL agent to reason over its exploration process and distill high-level knowledge for effectively guiding its future explorations. Specifically, we propose a novel RL algorithm that learns high-level knowledge in the form of a finite reward automaton by using the L* learning algorithm. We prove that in episodic RL, a finite reward automaton can express any non-Markovian bounded reward functions with finitely many reward values and approximate any non-Markovian bounded reward function (with infinitely many reward values) with arbitrary precision. We also provide a lower bound for the episode length such that the proposed RL approach almost surely converges to an optimal policy in the limit. We test this approach on two RL environments with non-Markovian reward functions, choosing a variety of tasks with increasing complexity for each environment. We compare our algorithm with the state-of-the-art RL algorithms for non-Markovian reward functions, such as Joint Inference of Reward machines and Policies for RL (JIRP), Learning Reward Machine (LRM), and Proximal Policy Optimization (PPO2). Our results show that our algorithm converges to an optimal policy faster than other baseline methods.

Improving the interpretability of deep reinforcement learning in urban drainage system operation

Traditional control approaches heavily rely on hard-coded expert knowledge, complicating the development of optimal control solutions as system complexity increases. Deep Reinforcement Learning (DRL) offers a self-learning control solution, proving advantageous in scenarios where crafting expert-based solutions becomes intricate. This study investigates the potential of DRL for supervisory control in a unique and complex heating system within a large-scale university building. The DRL framework aims to minimize energy costs while ensuring occupant comfort. However, the trial-and-error learning approach of DRL raises concerns about the trustworthiness of executed actions, hindering practical implementation. To address this, the study incorporates action masking, enabling the integration of hard constraints into DRL to enhance user trust. Maskable Proximal Policy Optimization (MPPO) is evaluated alongside standard Proximal Policy Optimization (PPO) and Soft Actor–Critic (SAC). Simulation results reveal that MPPO achieves comparable energy savings (8% relative to the baseline control) with fewer comfort violations than other methods. Therefore, it is selected among the candidate algorithms and experimentally implemented in the university building over one week. Experimental findings demonstrate that MPPO reduces energy costs while maintaining occupant comfort, resulting in a 36% saving compared to a historical day with similar weather conditions. These results underscore the proactive decision-making capability of DRL, establishing its viability for autonomous control in complex energy systems.

Epidemics of infectious diseases are an important threat to public health and global economies. Yet, the development of prevention strategies remains a challenging process. For this reason, we investigate a deep reinforcement learning approach to automatically learn prevention strategies in an epidemiological model, in the context of pandemic influenza. To this end, we construct a new epidemiological meta-population model, with 379 patches, that balances between model complexity and computational efficiency such that the use of reinforcement learning techniques becomes attainable. First, we set up a ground truth such that we can evaluate the performance of the ""Proximal Policy Optimization"" algorithm to learn in a single district of this epidemiological model. Next, we consider a larger scale problem, by conducting an experiment where we aim to learn a joint policy to control the districts in a community of 11 tightly coupled districts, for which no ground truth can be established. This experiment shows that deep reinforcement learning can be used to learn mitigation policies in complex epidemiological models with a large state space. Moreover, through this experiment, we demonstrate that there can be an advantage to consider collaboration between districts when designing prevention strategies.

Aiming at the energy-saving control problem of Heating, Ventilation and Air Conditioning (HVAC) system, a Deep Reinforcement Learning (DRL) control method based on Deep Deterministic Policy Gradient (DDPG) is proposed. The optimization capability of DDPG is improved by collecting sensor data inside the HVAC system and using Autoencoder (AE) for feature extraction. DDPG improves the system Coefficient of Performance (system COP) by controlling the mass flow of the chilled water pump of the HVAC system. This method is simulated, trained and verified on the refined HVAC system model built in this paper. The experimental results show that the method proposed can effectively improve the system COP and achieve the goal of saving energy in the HVAC system.

Study on deep reinforcement learning techniques for building energy consumption forecasting

Introduction: Deep reinforcement learning (deep RL) integrates the principles of reinforcement learning with deep neural networks, enabling agents to excel in diverse tasks ranging from playing board games such as Go and Chess to controlling robotic systems and autonomous vehicles. By leveraging foundational concepts of value functions, policy optimization, and temporal difference methods, deep RL has rapidly evolved and found applications in areas such as gaming, robotics, finance, and healthcare. Objective: This paper seeks to provide a comprehensive yet accessible overview of the evolution of deep RL and its leading algorithms. It aims to serve both as an introduction for newcomers to the field and as a practical guide for those seeking to select the most appropriate methods for specific problem domains. Methods: We begin by outlining fundamental reinforcement learning principles, followed by an exploration of early tabular Q-learning methods. We then trace the historical development of deep RL, highlighting key milestones such as the advent of deep Q-networks (DQN). The survey extends to policy gradient methods, actor–critic architectures, and state-of-the-art algorithms such as proximal policy optimization, soft actor–critic, and emerging model-based approaches. Throughout, we discuss the current challenges facing deep RL, including issues of sample efficiency, interpretability, and safety, as well as open research questions involving large-scale training, hierarchical architectures, and multi-task learning. Results: Our analysis demonstrates how critical breakthroughs have driven deep RL into increasingly complex application domains. We highlight existing limitations and ongoing bottlenecks, such as high data requirements and the need for more transparent, ethically aligned systems. Finally, we survey potential future directions, highlighting the importance of reliability and ethical considerations for real-world deployments.

Since the sample data after one exploration process can only be used to update network parameters once in on-policy deep reinforcement learning (DRL), a high sample efficiency is necessary to accelerate the training process of on-policy DRL. In the proposed method, a submartingale criterion is proposed on the basis of the equivalence relationship between the optimal policy and martingale, and then an advanced value iteration (AVI) method is proposed to conduct value iteration with a high accuracy. Based on this foundation, an anti-martingale (AM) reinforcement learning framework is established to efficiently select the sample data that is conducive to policy optimization. In succession, an AM proximal policy optimization (AMPPO) method, which combines the AM framework with proximal policy optimization (PPO), is proposed to reasonably accelerate the updating process of state value that satisfies the submartingale criterion. Experimental results on the Mujoco platform show that AMPPO can achieve better performance than several state-of-the-art comparative DRL methods.

Optimizing the fuel cycle cost through the optimization of nuclear reactor core loading patterns (LPs) involves multiple objectives and constraints, leading to a vast number of candidate solutions that cannot be explicitly solved. To advance the state of the art in core reload patterns, we have developed methods based on deep Reinforcement Learning (RL) for both single- and multi-objective optimization. Our previous research laid the groundwork for these approaches and demonstrated their ability to discover high-quality patterns within a reasonable time frame. On the other hand, Stochastic Optimization (SO) approaches are commonly used in the literature, but there is no rigorous explanation that shows which approach is better in which scenario. In this paper, we demonstrate the advantage of our RL-based approach, specifically using Proximal Policy Optimization (PPO) against the most commonly used SO-based methods: Genetic Algorithm, Parallel Simulated Annealing with mixing of states, and Tabu Search, as well as an ensemble-based method, i.e. the Prioritized replay Evolutionary and Swarm Algorithm. We found that the LP scenarios derived in this paper are amenable to a global search to identify promising research directions rapidly but then need to transition into a local search to exploit these directions efficiently and prevent getting stuck in local optima. PPO adapts its search capability via a policy with learnable weights, allowing it to function as both a global search method and a local search method. Subsequently, we compared all algorithms against PPO in long runs, which exacerbated the differences seen in the shorter cases. Overall, the work demonstrates the statistical superiority of PPO compared to the other considered algorithms.