Reinforcement learning optimal control method for multi chiller HVAC system in an existing office building

A combined system of chilled ceiling, displacement ventilation and desiccant dehumidification

Data analysis and stochastic modeling of lighting energy use in large office buildings in China

Deep reinforcement learning control for non-stationary building energy management

Symbiosis of the double skin façade with the HVAC system

Model Based Indirect Submetering Method of HVAC End-use in Existing Office Building

The heating, ventilation, and air conditioning (HVAC) system is a major energy consumer in office buildings, and its operation is critical for indoor thermal comfort. While previous studies have indicated that reinforcement learning control can improve HVAC energy efficiency, they did not provide enough information about end-to-end control (i.e., from raw observations to ready-to-implement control signals) for centralized HVAC systems in multizone buildings due to the limitations of reinforcement learning methods or the test buildings being single zones with independent HVAC systems. This study developed a model-free end-to-end dynamic HVAC control method based on a recently proposed deep reinforcement learning framework to control the centralized HVAC system of a multizone office building. By using the deep neural network, the proposed control method could directly take measurable parameters, including weather and indoor environment conditions, as inputs and control indoor temperature setpoints at a supervisory level. In some test cases, the proposed control method could successfully learn a dynamic control policy to reduce HVAC energy consumption by 12.8% compared with the baseline case using conventional control methods, without compromising thermal comfort. However, an over-fitting problem was noted, indicating that future work should first focus on the generalization of deep reinforcement learning.

Model predictive control of a building renewable energy system based on a long short-term hybrid model

Reinforcement learning (RL) for controlling building heating, ventilation, and air conditioning (HVAC) systems has shown promise in delivering energy-efficient, energy-flexible and resilient buildings. However, training RL agents in real-time in buildings is generally considered time and cost prohibitive. RL agents can be trained offline in simulated environments, reducing the time to performance once online, but complex models for simulation can be difficult to generate. What level of complexity is required in an offline training model to produce acceptable online results is unknown. This work seeks to determine how simplified models can be used to train RL agents for building HVAC control. The work uses the EnergyPlus simulation environment paired with HVAC systems modeled in Modelica to act as the real-world analogue environment. Simplified reduced-order models of the building were developed using the lumped capacitance method. The results show that an RL agent trained using Deep Q Learning on a simplified building model can be transferable to a more complex building simulation environment with comparable performance. In virtually all cases, the RL agents were able to deliver improved performance over the baseline proportional controller. Online learning gave mixed results in terms of enhancing the policy transfer process. The implementation in this paper was more suitable for improving the performance of poor policy transfer rather than fine-tuning already well-performing policies.

Reinforcement learning (RL) was first demonstrated to be a feasible approach to controlling heating, ventilation, and air conditioning (HVAC) systems more than a decade ago. However, there has been limited progress towards a practical and scalable RL solution for HVAC control. While one can train an RL agent in simulation, it is not cost-effective to create a model for each thermal zone or building. Likewise, existing RL agents generally take a long time to learn and are opaque to expert interrogation, making them unattractive for real-world deployment. To tackle these challenges, we propose Gnu-RL: a novel approach that enables practical deployment of RL for HVAC control and requires no prior information other than historical data from existing HVAC controllers. To achieve this, Gnu-RL adopts a recently-developed Differentiable Model Predictive Control (MPC) policy, which encodes domain knowledge on planning and system dynamics, making it both sample-efficient and interpretable. Prior to any interaction with the environment, a Gnu-RL agent is pre-trained on historical data using imitation learning, which enables it to match the behavior of the existing controller. Once it is put in charge of controlling the environment, the agent continues to improve its policy end-to-end, using a policy gradient algorithm. We evaluate Gnu-RL on both an EnergyPlus model and a real-world testbed. In both experiments, our agents were directly deployed in the environment after offline pre-training on expert demonstration. In the simulation experiment, our approach saved 6.6% energy compared to the best published RL result for the same environment, while maintaining a higher level of occupant comfort. Next, Gnu-RL was deployed to control the HVAC of a real-world conference room for a three-week period. Our results show that Gnu-RL saved 16.7% of cooling demand compared to the existing controller and tracked temperature set-point better.

Heating, ventilation and air conditioning (HVAC) chilled water systems offer significant potential for energy saving and reinforcement learning (RL) methods have been extensively studied and validated for optimizing HVAC energy consumption. However, RL's low sample efficiency and reliance on randomized exploration limit its practical application. To enhance the robustness and stability of RL‐based energy optimization methods, an RL optimization approach based on a mechanism‐data hybrid‐driven model is proposed, derived from the MBPO (Model‐based policy optimization) scheme. Firstly, a novel end‐to‐end HVAC chilled water system model is developed to serve as the foundation for the hybrid model design. Second, a hybrid‐driven RL environment model framework is introduced, combining a mechanistic model with a probabilistic neural network. The mechanistic component provides generalization capabilities, while the data‐driven component offers adaptability. Third, improvements to MBPO are proposed, including double policy optimization and adaptive branch rollout, further to enhance dynamic environmental adaptability and model utilization efficiency. Finally, comparative and ablation experiments conducted using both simulation environments and measured data demonstrate that the proposed method achieves higher learning efficiency and improved robustness.

Automation-assisted fault detection of an air-handling unit; implementing the method in a real building

Data-driven predictive control for smart HVAC system in IoT-integrated buildings with time-series forecasting and reinforcement learning

MF^2: Model-free reinforcement learning for modeling-free building HVAC control with data-driven environment construction in a residential building

Towards optimal control of air handling units using deep reinforcement learning and recurrent neural network

Buildings are immensely energy-demanding and this fact is enhanced by the expectation of even more increment of energy consumption in the future. In order to mitigate this problem, a low-cost, flexible and high-quality Decision-Making Mechanism for supporting the tasks of a Smart Thermostat is proposed. Energy efficiency and thermal comfort are the two primary quantities regarding control performance of a building's HVAC system. Apart from demonstrating a conflicting relationship, they depend not only on the building's dynamics, but also on the surrounding climate and weather, thus rendering the problem of finding a long-term control scheme hard, and of stochastic nature. The introduced mechanism is inspired by Reinforcement Learning techniques and aims at satisfying both occupants' thermal comfort and limiting energy consumption. In contrast to to existing methods, this approach focuses on a plug&play solution, that does not require detailed building models and is applicable to a wide variety of buildings as it learns the dynamics using gathered information from the environment. The proposed control mechanisms were evaluated via a well-known building simulation framework and implemented on ARM-based, low-cost embedded devices.