Increasing large-scale data center capacity by statistical power control

Large energy consumption in data centers has become a challenging problem with the emergence of cloud computing and large scale data centers. In this paper, we present an architectural framework for thermal-aware resource management while considering energy efficiency. The framework consists of a layered architecture and integrates a set of easy-to-use client tools and a thermal-aware task management middleware to schedule tasks based on thermal conditions within a cluster and among different data centers. As part of this paper we focus on the development of a thermal-aware task scheduling component for a single data center. This component is fundamental to our future activities, while considering to balance the temperature distribution in a single data center, thus implicitly minimizing energy cost in data centers.

Cooling power constitutes a large portion of the total electrical power consumption in data centers. Approximately 25%∼40% of the electricity used within a production data center is consumed by the cooling system. Improving the cooling energy efficiency has attracted a great deal of research attention. Many strategies have been proposed for cutting the data center energy costs. One of the effective strategies for increasing the cooling efficiency is using dynamic thermal management. Another effective strategy is placing cooling devices (heat exchangers) closer to the source of heat. This is the basic design principle of many hybrid cooling systems and liquid cooling systems for data centers. Dynamic thermal management of data centers is a huge challenge, due to the fact that data centers are operated under complex dynamic conditions, even during normal operating conditions. In addition, hybrid cooling systems for data centers introduce additional localized cooling devices, such as in row cooling units and overhead coolers, which significantly increase the complexity of dynamic thermal management. Therefore, it is of paramount importance to characterize the dynamic responses of data centers under variations from different cooling units, such as cooling air flow rate variations. In this study, a detailed computational analysis of an in row cooler based hybrid cooled data center is conducted using a commercially available computational fluid dynamics (CFD) code. A representative CFD model for a raised floor data center with cold aisle-hot aisle arrangement fashion is developed. The hybrid cooling system is designed using perimeter CRAH units and localized in row cooling units. The CRAH unit supplies centralized cooling air to the under floor plenum, and the cooling air enters the cold aisle through perforated tiles. The in row cooling unit is located on the raised floor between the server racks. It supplies the cooling air directly to the cold aisle, and intakes hot air from the back of the racks (hot aisle). Therefore, two different cooling air sources are supplied to the cold aisle, but the ways they are delivered to the cold aisle are different. Several modeling cases are designed to study the transient effects of variations in the flow rates of the two cooling air sources. The server power and the cooling air flow variation combination scenarios are also modeled and studied. The detailed impacts of each modeling case on the rack inlet air temperature and cold aisle air flow distribution are studied. The results presented in this work provide an understanding of the effects of air flow variations on the thermal performance of data centers. The results and corresponding analysis is used for improving the running efficiency of this type of raised floor hybrid data centers using CRAH and IRC units.

This paper talks about systems and protocols related to the data center along with problems occurring with running data centers and solutions addressing those errors. To begin with, basic concepts of the data center is introduced by presenting the design and evaluation of a representative system-Portland-which is a specifically designed protocol for data centers with efficient routing, loop-avoided forwarding, and effective detection and recovery about faults. After introducing some basic ideas and topology about the data center, problems regarding data center are discussed; specifically, the Google and Facebook Data Center is analyzed. Google Data Center's four-clusterpost network could no longer satisfy the demand for the bandwidth, a necessary solution to scale out, and a cost-effective means to update. On the other hand, the operational data of production network infrastructure at Facebook is presented to show the roles that the reliability plays in large scale data centers' network. Lastly, after exquisitely describing the nature, pros and cons, and the characteristics of Data Centers, an application developed upon the data centers is interpreted: CrystalNet, a network emulator, is discussed, which can generate a virtue network functioning like a real network, allowing network software operators to test or validate their ideas and software tools on the virtue network before implementing them to the real network.

As cloud computing is expected to expand rapidly in the coming years, the large-scale computing and data centers are becoming more and more widespread in the world. Energy consumption of these distributed systems has become a urgent problem and received much attention. Application Scheduling can alleviate this problem by reducing the number of running nodes and effectively maximizing total system efficiency. This paper focuses on scheduling applications in large-scale data centers using genetic algorithm. Specifically, we present the design and implementation of the cost function, the modification of the genetic operators and the choice of the data transition weight. The algorithm is studied via simulation and implementation in a large-scale data center. Test results and performance discussion justify the feasibility of the scheduling algorithm. From the results, we know that the proposed application scheduling method can be useful in practice, which can reduce the running nodes and minimize the cost of data transferred among the nodes efficiently.

Data center (DC) electricity use is increasing at an annual rate of over 20% and presents a concern for the Information Technology (IT) industry, governments, and the society. A large fraction of the energy use is consumed by the compressor cooling to maintain the recommended operating conditions for IT equipment. The most common way to improve the DC efficiency is achieved by optimally provisioning the cooling power to match the global heat dissipation in the DC. However, at a more granular level, the large range of heat densities of today's IT equipment makes the task of provisioning cooling power optimized to the level of individual computer room air conditioning (CRAC) units much more challenging. Distributed sensing within a DC enables the development of new strategies to improve energy efficiency, such as hot spot elimination through targeted cooling, matching power consumption at rack level with workload schedule, and minimizing power losses. The scope of Measurement and Management Technologies (MMT) is to develop a software tool and the underlying sensing technology to provide critical decision support and control for DC and telecommunication facilities (TF) operations. A key aspect of MMT technology is integration of modeling tools to understand how changes in one operational parameter affect the overall DC response. It is demonstrated that reduced ordered models for DC can generate, in less than 2 seconds computational time, a three dimensional thermal model in a 50 kft{sup 2} DC. This rapid modeling enables real time visualization of the DC conditions and enables 'what if' scenarios simulations to characterize response to 'disturbances'. One such example is thermal zone modeling that matches the cooling power to the heat generated at a local level by identifying DC zones cooled by a specific CRAC. Turning off a CRAC unit can be simulated to understand how the other CRAC utilization changes and how server temperature responds. Several new sensing technologies were added to the existing MMT platform: (1) air contamination (corrosion) sensors, (2) power monitoring, and (3) a wireless environmental sensing network. All three technologies are built on cost effective sensing solutions that increase the density of sensing points and enable high resolution mapping of DCs. The wireless sensing solution enables Air Conditioning Unit (ACU) control while the corrosion sensor enables air side economization and can quantify the risk of IT equipment failure due to air contamination. Validation data for six test sites demonstrate that leveraging MMT energy efficiency solutions combined with industry best practices results in an average of 20% reduction in cooling energy, without major infrastructure upgrades. As an illustration of the unique MMT capabilities, a data center infrastructure efficiency (DCIE) of 87% (industry best operation) was achieved. The technology is commercialized through IBM System and Technology Lab Services that offers MMT as a solution to improve DC energy efficiency. Estimation indicates that deploying MMT in existing DCs can results in an 8 billion kWh savings and projection indicates that constant adoption of MMT can results in obtainable savings of 44 billion kWh in 2035. Negotiations are under way with business partners to commercialize/license the ACU control technology and the new sensor solutions (corrosion and power sensing) to enable third party vendors and developers to leverage the energy efficiency solutions.

Cooling is a major component in the enormous energy consumption in data centers. Accurate evaluation of cooling inside a data center forms the backbone of all the attempts for improving cooling efficiency. Models based on computational fluid dynamics (CFD) are typically used for accurate evaluation, but have a drawback of high computation time. This paper presents a novel thermal predictor to evaluate data center cooling in seconds. The key idea is to extract information from a single instance of CFD simulation using metrics called as influence indices to build the fast thermal predictor. Then, this predictor can evaluate the cooling for altered operation of data center with comparable accuracy in seconds without the need for repetitive CFD simulations. This paper demonstrates the accuracy of the thermal predictor by comparing with CFD simulations for a sample, but realistic data center. The fast thermal predictor then successfully passed more challenging tests in a real production data center and proved its practical utility. The results of the thermal predictor compared with measurements carried out in the production data center are also presented. This fast thermal predictor is an important milestone in the development of a method for model-based real time control of data center cooling.

With the advent of cloud computing and online services, large enterprises rely heavily on their data centers to serve end users. Among different server components, hard disk drives are known to contribute significantly to server failures. Disk failures as well as their impact on the performance of storage systems and operating costs are becoming an increasingly important concern for data center designers and operators. However, there is very little understanding on the characteristics of disk failures in data centers. Effective disk failure management and data recovery also requires a deep understanding of the nature of disk failures. In this paper, we present a systematic approach to provide a holistic and insightful view of disk failures. We study a large-scale storage system from a production data center. We categorize disk failures based on their distinctive manifestations and properties. Then we characterize the degradation of disk errors to failures by deriving the degradation signatures for each failure category. The influence of disk health attributes on failure degradation is also quantified. We discuss leveraging the derived degradation signatures to forecast disk failures even in their early stages. To the best of our knowledge, this is the first work that shows how to discover the categories of disk failures and characterize their degradation processes on a production data center.

Data centers are growing in size and complexity driven by trends such as cloud computing and on-line services. Such large data centers pose several challenges for system management. Key among them is anomaly detection which is required to monitor and analyze metrics across several thousands servers and across multiple layers of abstractions to detect anomalous system behavior. In practice, multiple anomaly detection tools are used to continuously raise alarms across multiple metrics and servers. These alarms include both true positives and false alarms. Administrators and management tools act on these alarms for diagnosis and deeper root cause analysis and take appropriate management actions to mitigate the anomalous behaviors. Given the scale and scope of the system, the administrators and management tools are overwhelmed with the large number of alarms at any given instant, many of which are false alarms. It is therefore necessary to prioritize and rank these alarms, so as to take timely actions that maintain the service level agreements for the data center. Existing techniques for such ranking are ad-hoc and not scalable. We propose ranking windows of monitored metrics based on their probability of occurrence. We explain how these probabilities can be computed based either on the false positive rates for which the accompanying anomaly detectors were designed, or, when available, on the probability models underlying the false positive rates. In the simplest case, the ranking procedure reduces to computing the Z-score of the observed measurements and computing a statistic from a window of Z-scores to use as a basis for ranking. The proposed techniques are reliable, lightweight and easy to deploy in the modern data center. We have validated these techniques on synthetic data containing injected anomalies and on data acquired from production data centers.

The widespread demand for hosting application services in the cloud has been fueled by the deployment of cloud data centers (CDCs) on a global scale. Furthermore, modern apps' resource needs have sharply increased, especially in industries that use a lot of data. As a result, more cloud servers have been made available, resulting in higher energy usage and, ecological problems. Large-scale data centers have been developed as a result of the rapidly increasing demand for cloud services, allowing application service providers to rent data center space for application deployment by user-required quality of service (QoS). These data centers use a lot of electricity, which raises running expenses and produces more carbon dioxide (CO2) emissions. Modern cloud computing environments must also provide QoS for their users, necessitating a trade-off between power performance, energy consumption, and service-level agreement (SLA) compliance. We present an intelligent resource management policy using enforcement learning for CDCs. The objective is to continuously consolidate and dynamically allocate virtual machines (VMs). Utilizing live migration and disabling inactive nodes to reduce power consumption in this cloud environment while maintaining service quality. To enable dynamic resource management, a better power-performance tradeoff, and significantly lower energy consumption, we integrate several artificial intelligence concepts. Based on the result the proposed approach is more efficient as compared with other techniques.

Independent optimization for workload and power management, and active cooling control have been studied extensively to improve data center energy efficiency. Recently, proposals have started to advocate unified workload, power, and cooling management for further energy savings. In this paper, we study this problem with the objectives of both saving energy and capping power. We present the detailed models derived in our previous work from experiments on an blade enclosure system that can be representative of a data center, discuss the optimization opportunities for coordinated power and cooling management, and the challenges for controller design. We then propose a few design principles and examples for unified workload management, power minimization, and power capping. Our simulation-based evaluation shows that the controllers can cap the total power consumption while maintaining the thermal conditions and improve the overall energy efficiency. We argue that the same opportunities, challenges, and designs are also generally applicable to data center level management.

Data center power is a scarce resource that often goes underutilized due to conservative planning. This is because the penalty for overloading the data center power delivery hierarchy and tripping a circuit breaker is very high, potentially causing long service outages. Recently, dynamic server power capping, which limits the amount of power consumed by a server, has been proposed and studied as a way to reduce this penalty, enabling more aggressive utilization of provisioned data center power. However, no real at-scale solution for data center-wide power monitoring and control has been presented in the literature. In this paper, we describe Dynamo -- a data center-wide power management system that monitors the entire power hierarchy and makes coordinated control decisions to safely and efficiently use provisioned data center power. Dynamo has been developed and deployed across all of Facebook's data centers for the past three years. Our key insight is that in real-world data centers, different power and performance constraints at different levels in the power hierarchy necessitate coordinated data center-wide power management. We make three main contributions. First, to understand the design space of Dynamo, we provide a characterization of power variation in data centers running a diverse set of modern workloads. This characterization uses fine-grained power samples from tens of thousands of servers and spanning a period of over six months. Second, we present the detailed design of Dynamo. Our design addresses several key issues not addressed by previous simulation-based studies. Third, the proposed techniques and design have been deployed and evaluated in large scale data centers serving billions of users. We present production results showing that Dynamo has prevented 18 potential power outages in the past 6 months due to unexpected power surges, that Dynamo enables optimizations leading to a 13% performance boost for a production Hadoop cluster and a nearly 40% performance increase for a search cluster, and that Dynamo has already enabled an 8% increase in the power capacity utilization of one of our data centers with more aggressive power subscription measures underway.

Data Center reliability and the soaring cost of infrastructure are of major concern to businesses and Telcos. Traditionally data centers have been UPS-powered however on reliability grounds UPS are considered the "poor cousin" to their DC counterpart. This paper presents an analysis of UPS systems to show that a reliability matching DC systems is readily achievable. The paper will also discuss design options that are not required to achieve "DC-grade" reliability and which add substantially to the cost of a data center. The analysis was conducted to determine a set of configurations for use in a national standard by a leading Australian Telco. Configurations graded according to reliability enable a value comparison to be made between different UPS systems. A detailed reliability assessment was made of each configuration using Monte Carlo power system reliability modeling software. Traditionally DC tele power systems have been the cornerstone of reliable telecommunications power. As telecommunications advances rapidly towards an IP-centric network, the role and importance of Data Centers in telecommunications has surged. Data Centers however are traditionally AC powered and accordingly the importance of UPS systems to Telcos has increased dramatically. Emerging telecommunications platforms do not always have a DC powering option therefore highly-dependable UPS systems are needed. The problem is that UPS systems typically provide an order of magnitude lower reliability. High-end DC systems can readily attain an MTBF of 200+ years while UPS systems typically achieve as little as -10-30 years MTBF. With the cost of modern data centers soaring, the paper will show how such costs can be contained while designing UPS systems with "DC-grade" reliability.

Data centers today consume tremendous amount of energy in terms of power distribution and cooling. Dynamic capacity provisioning is a promising approach for reducing energy consumption by dynamically adjusting the number of active machines to match resource demands. However, despite extensive studies of the problem, existing solutions for dynamic capacity provisioning have not fully considered the heterogeneity of both workload and machine hardware found in production environments. In particular, production data centers often comprise several generations of machines with different capacities, capabilities and energy consumption characteristics. Meanwhile, the workloads running in these data centers typically consist of a wide variety of applications with different priorities, performance objectives and resource requirements. Failure to consider heterogenous characteristics will lead to both sub-optimal energy-savings and long scheduling delays, due to incompatibility between workload requirements and the resources offered by the provisioned machines. To address this limitation, in this paper we present HARMONY, a Heterogeneity-Aware Resource Management System for dynamic capacity provisioning in cloud computing environments. Specifically, we first use the K-means clustering algorithm to divide the workload into distinct task classes with similar characteristics in terms of resource and performance requirements. Then we present a novel technique for dynamically adjusting the number of machines of each type to minimize total energy consumption and performance penalty in terms of scheduling delay. Through simulations using real traces from Google's compute clusters, we found that our approach can improve data center energy efficiency by up to 28% compared to heterogeneity-oblivious solutions.

Modern data centers must provide performance assurance for complex system software such as multi-tier web applications. In addition, the power consumption of data centers needs to be minimized to reduce operating costs and avoid system overheating. Various power-efficient performance management strategies have been proposed based on dynamic voltage and frequency scaling (DVFS). Virtualization technologies have also made it possible to consolidate multiple virtual machines (VMs) onto a smaller number of active physical servers for even greater power savings, but at the cost of a higher overhead. This paper proposes a performance-controlled power optimization solution for virtualized data centers with multi-tier applications. While existing work relies on either DVFS or server consolidation in a separate manner, our solution utilizes both strategies for maximized power savings by integrating feedback control with optimization strategies. At the application level, a multi-input-multi-output controller is designed to achieve the desired performance for applications spanning multiple VMs, on a short time scale, by reallocating the CPU resources and DVFS. At the data center level, a power optimizer is proposed to incrementally consolidate VMs onto the most power-efficient servers on a longer time scale. Empirical results on a hardware testbed demonstrate that our solution can effectively achieve performance-assured power savings. Extensive simulation results, based on a trace file of 5,415 real servers, demonstrate the efficacy of our solution in large-scale data centers.

As the scale and density of data centers continue to grow, cost-effective data center management (DCM) is becoming a significant challenge for enterprises hosting large-scale online and cloud services. Machines need to be monitored, and the scale of operations mandates an automated management with high reliability and real-time performance. The limitations of today’s typical DCM network are many-fold. Primarily, it is a fixed wired network, and hence scaling it for a large number of servers increases its cost. In addition, with server densities increasing over recent years, this network also has to be cabled correctly and the management of this network parallels the complexity of managing a data network, since it needs to be networked with multiple switches and routers. In this article, we propose a wireless sensor network as a cost-effective networking solution for DCM while satisfying the reliability and latency performance requirements of DCM. We have developed CapNet, a real-time wireless sensor network for power capping, a time-critical DCM function for power management in a cluster of servers. CapNet employs an efficient event-driven protocol that triggers data collection only on the detection of a potential power capping event. We deploy and evaluate CapNet in a data center. Using server power traces, our experimental results on a cluster of 480 servers inside the data center show that CapNet can meet the real-time requirements of power capping. CapNet demonstrates the feasibility and efficacy of wireless sensor networks for time-critical DCM applications.