Gnu-RL: A Practical and Scalable Reinforcement Learning Solution for Building HVAC Control Using a Differentiable MPC Policy

Reinforcement learning (RL) was first demonstrated to be a feasible approach to controlling heating, ventilation, and air conditioning (HVAC) systems more than a decade ago. However, there has been limited progress towards a practical and scalable RL solution for HVAC control. While one can train an RL agent in simulation, it is not cost-effective to create a model for each thermal zone or building. Likewise, existing RL agents generally take a long time to learn and are opaque to expert interrogation, making them unattractive for real-world deployment. To tackle these challenges, we propose Gnu-RL: a novel approach that enables practical deployment of RL for HVAC control and requires no prior information other than historical data from existing HVAC controllers. To achieve this, Gnu-RL adopts a recently-developed Differentiable Model Predictive Control (MPC) policy, which encodes domain knowledge on planning and system dynamics, making it both sample-efficient and interpretable. Prior to any interaction with the environment, a Gnu-RL agent is pre-trained on historical data using imitation learning, which enables it to match the behavior of the existing controller. Once it is put in charge of controlling the environment, the agent continues to improve its policy end-to-end, using a policy gradient algorithm. We evaluate Gnu-RL on both an EnergyPlus model and a real-world testbed. In both experiments, our agents were directly deployed in the environment after offline pre-training on expert demonstration. In the simulation experiment, our approach saved 6.6% energy compared to the best published RL result for the same environment, while maintaining a higher level of occupant comfort. Next, Gnu-RL was deployed to control the HVAC of a real-world conference room for a three-week period. Our results show that Gnu-RL saved 16.7% of cooling demand compared to the existing controller and tracked temperature set-point better.

MF^2: Model-free reinforcement learning for modeling-free building HVAC control with data-driven environment construction in a residential building

Application of optimization techniques on lumped HVAC models for energy conservation

Purpose Load shedding and grid instability continue to pose a significant challenge for commercial buildings in West Africa. Various artificial intelligence–Heating, Ventilation and Air Conditioning (AI–HVAC) system algorithms show promise in improving energy efficiency across different models. However, its performance under load-shedding conditions in shopping malls remains understudied. This study assesses the resilience of AI-controlled systems under load-shedding conditions in shopping malls in Ghana. Design/methodology/approach A mixed quasi-experimental design was employed, integrating sensor data with simulation models and conducting interviews with HVAC professionals in Ghana’s shopping malls. AI-based control strategies, such as Model Predictive Control (MPC), Data-enabled Predictive Control (DeePC) and Reinforcement Learning (RL), were tested across the malls to assess their response times, thermal comfort and energy consumption. Findings The study found that RL achieved the lowest response latency, the highest energy consumption, the highest thermal comfort and the highest uptime (95.4%) compared with MPC and DeePC. Furthermore, DeePC established moderate performance across the indicators, while MPC reported the lowest values. Qualitative results indicate limited awareness of AI-based systems but a strong willingness to adopt them, given their expected benefits. Practical implications The RL–HVAC systems offer a viable solution for enhancing energy resilience during load shedding in shopping malls. Although beneficial, this requires shopping management readiness and professional training. Originality/value This critical performance of RL, along with other AI-based algorithms, in AI-controlled HVAC systems under unstable power in urban areas represents a new direction compared with MPC and DeePC functionalities in energy simulations in cold-climate regions.

Optimization of thermal processes in a variable air volume HVAC system

Heating, ventilation, and air conditioning (HVAC) systems in buildings are an emerging application area for model predictive control (MPC) due to the significant cost benefits that can be achieved via load shifting in modern electricity markets. In this paper, we discuss some of the opportunities and challenges associated with applying MPC to commercial HVAC systems. After defining the control problem, a decomposition of the centralized MPC is presented and demonstrated for an example system. Recent work at the Stanford University campus is also highlighted to show these ideas in practice, and an outlook for the field is given.

The heating, ventilation, and air conditioning (HVAC) system is a major energy consumer in office buildings, and its operation is critical for indoor thermal comfort. While previous studies have indicated that reinforcement learning control can improve HVAC energy efficiency, they did not provide enough information about end-to-end control (i.e., from raw observations to ready-to-implement control signals) for centralized HVAC systems in multizone buildings due to the limitations of reinforcement learning methods or the test buildings being single zones with independent HVAC systems. This study developed a model-free end-to-end dynamic HVAC control method based on a recently proposed deep reinforcement learning framework to control the centralized HVAC system of a multizone office building. By using the deep neural network, the proposed control method could directly take measurable parameters, including weather and indoor environment conditions, as inputs and control indoor temperature setpoints at a supervisory level. In some test cases, the proposed control method could successfully learn a dynamic control policy to reduce HVAC energy consumption by 12.8% compared with the baseline case using conventional control methods, without compromising thermal comfort. However, an over-fitting problem was noted, indicating that future work should first focus on the generalization of deep reinforcement learning.

This paper introduces an improved real-time Model Predictive Climate Control (MPCC) technique to reduce power consumption in Heating, Ventilation and Air Conditioning (HVAC) system of the Battery Electric Vehicles (BEVs). In the presented control technique, the fan, heating and cooling power usage is optimized considering the effect of the ambient temperature. Moreover, the need of a physical temperature measurement sensor is eliminated using an ambient temperature observer. In a typical BEV, the HVAC load is significant and it influences the overall vehicle performance and driving range. Therefore, a real-time control system capable of maintaining desired cabin temperature while achieving maximum HVAC efficiency is highly desirable. In the proposed MPCC, the optimum split of the battery power between the heating\cooling and the fan power is identified as a function of ambient temperature through an offline optimization process and used in the form of a lookup-table for real-time implementation. It is demonstrated that the proposed MPCC improves energy consumption efficiency of the entire HVAC system up to 5% by dynamically adapting to the variations in the ambient temperature. Furthermore, owing to the online temperature observation process, the proposed HVAC control system allows elimination of ambient temperature sensor and corresponding maintenance efforts. In this paper, the performance of the proposed MPCC is evaluated over a single-zone HVAC model of a BEV in MATLAB\Simulink <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">®</sup> environment.

As major modules of Heating, Ventilation and Air Conditioning (HVAC) systems, Air Handling Units (AHUs) and Variable Air Volume (VAV) boxes are used to condition and circulate air to rooms. To save energy and satisfy human comfort and ventilation requirements, accurately and timely diagnosing their faults is critical. The problem, however, is challenging since (1) Hidden Markov Models (HMMs) can capture coupling among components, but many states are required to represent failure modes and fault severities; and (2) diagnosing faults of many VAV boxes leads to high computational requirements. In this paper, dynamic HMMs are used to identify state transitions of components since (1) component processes have long time constants; and (2) dynamic HMMs have various state transition matrices under different conditions, thus can capture fault propagations. Dynamic HMMs are used to identify failure modes but not fault severities to reduce computational effort. To diagnose faults of VAV boxes with low computational effort, faults are localized in all VAV boxes. Unscented particle filter and statistical process control are used to further localize faults down to individual VAV level and to assess fault severities. Experimental results demonstrate that the method can accurately diagnose dependent faults.

The proactive control of building’s heating, ventilation, and air conditioning (HVAC) system can reduce energy consumption and provides cost saving. An efficient design of a controller for a complex system such as a building, which is also prone to uncertainties and disturbances, calls for advanced control methods such as model predictive control (MPC). This article presents a tube-based MPC to minimize the net cost of energy usage by the building’s HVAC system while satisfying the comfort level of the building’s occupants. Contrary to the conventional MPC, this controller is robust against model uncertainty and exogenous disturbances affecting the building thermal load model. Compared to other robust approaches such as the min–max controller, the proposed controller is of lower computational complexity and can be used with the environment having complex models, e.g., with nonlinear dynamics. Moreover, the proposed controller shows a satisfactory performance when optimizing an economic objective, such as cost minimization. The merits of using the proposed MPC controller are demonstrated using several simulation cases.

Reinforcement learning (RL) for controlling building heating, ventilation, and air conditioning (HVAC) systems has shown promise in delivering energy-efficient, energy-flexible and resilient buildings. However, training RL agents in real-time in buildings is generally considered time and cost prohibitive. RL agents can be trained offline in simulated environments, reducing the time to performance once online, but complex models for simulation can be difficult to generate. What level of complexity is required in an offline training model to produce acceptable online results is unknown. This work seeks to determine how simplified models can be used to train RL agents for building HVAC control. The work uses the EnergyPlus simulation environment paired with HVAC systems modeled in Modelica to act as the real-world analogue environment. Simplified reduced-order models of the building were developed using the lumped capacitance method. The results show that an RL agent trained using Deep Q Learning on a simplified building model can be transferable to a more complex building simulation environment with comparable performance. In virtually all cases, the RL agents were able to deliver improved performance over the baseline proportional controller. Online learning gave mixed results in terms of enhancing the policy transfer process. The implementation in this paper was more suitable for improving the performance of poor policy transfer rather than fine-tuning already well-performing policies.

Battery electric buses (BEB) will increasingly replace buses with internal combustion engines in the fleets of transport companies. However, range prevents the application of BEB on all bus routes. Auxiliary consumers highly affect the range and the heating, ventilation and air conditioning (HVAC) system plays a major role within all. The high energy consumption of the HVAC system can possibly be reduced with intelligent control methods since their conventional counterparts guarantee compliance with specifications but do not consider energy consumptions. Thus, an energy-saving control is desired, which considers the minimization of energy consumption, but simultaneously complies with given specifications. To meet these requirements, following controllers were implemented: (1) model predictive control (MPC) and (2) reinforcement learning (RL) based control. This paper describes the implementation and application of both controllers on a Simulink model of a modern heat pump HVAC system and compares the results with PID control.

Deep reinforcement learning control for non-stationary building energy management

Heating, ventilation and air conditioning (HVAC) chilled water systems offer significant potential for energy saving and reinforcement learning (RL) methods have been extensively studied and validated for optimizing HVAC energy consumption. However, RL's low sample efficiency and reliance on randomized exploration limit its practical application. To enhance the robustness and stability of RL‐based energy optimization methods, an RL optimization approach based on a mechanism‐data hybrid‐driven model is proposed, derived from the MBPO (Model‐based policy optimization) scheme. Firstly, a novel end‐to‐end HVAC chilled water system model is developed to serve as the foundation for the hybrid model design. Second, a hybrid‐driven RL environment model framework is introduced, combining a mechanistic model with a probabilistic neural network. The mechanistic component provides generalization capabilities, while the data‐driven component offers adaptability. Third, improvements to MBPO are proposed, including double policy optimization and adaptive branch rollout, further to enhance dynamic environmental adaptability and model utilization efficiency. Finally, comparative and ablation experiments conducted using both simulation environments and measured data demonstrate that the proposed method achieves higher learning efficiency and improved robustness.

This paper proposes a distributionally robust optimization approach (DROA) based on a disjoint layered ambiguity set for scheduling the energy consumption of the heating, ventilation and air conditioning (HVAC) system. The uncertainties of the predicted outdoor temperature error and indoor temperature variations that are caused by human activities are taken into account in the energy consumption scheduling of the HVAC based on historical data. The maximum uncertainty set of outdoor temperature is divided into disjoint subintervals and the probabilistic information of these subintervals is obtained to construct a disjoint layered ambiguity set. A nonlinear HVAC's energy consumption problem is formulated by using the DROA method to deal with these two uncertainties based on a disjoint layered ambiguity set with distributionally robust chance constraints (DRCCs). In order to solve this nonlinear problem, these DRCCs are converted to a linear programming problem and solved by using linear programming. Simulation results illustrate the effectiveness of the proposed method and comparing them with the DROA based on a nest layered ambiguity set and the traditional robust approach (ROA), the proposed DROA based on a disjoint layered ambiguity set reduces the electricity cost and guarantees the thermal comfort level of users.