Characterizing Disk Failures with Quantified Disk Degradation Signatures: An Early Experience

Data centers are growing in size and complexity driven by trends such as cloud computing and on-line services. Such large data centers pose several challenges for system management. Key among them is anomaly detection which is required to monitor and analyze metrics across several thousands servers and across multiple layers of abstractions to detect anomalous system behavior. In practice, multiple anomaly detection tools are used to continuously raise alarms across multiple metrics and servers. These alarms include both true positives and false alarms. Administrators and management tools act on these alarms for diagnosis and deeper root cause analysis and take appropriate management actions to mitigate the anomalous behaviors. Given the scale and scope of the system, the administrators and management tools are overwhelmed with the large number of alarms at any given instant, many of which are false alarms. It is therefore necessary to prioritize and rank these alarms, so as to take timely actions that maintain the service level agreements for the data center. Existing techniques for such ranking are ad-hoc and not scalable. We propose ranking windows of monitored metrics based on their probability of occurrence. We explain how these probabilities can be computed based either on the false positive rates for which the accompanying anomaly detectors were designed, or, when available, on the probability models underlying the false positive rates. In the simplest case, the ranking procedure reduces to computing the Z-score of the observed measurements and computing a statistic from a window of Z-scores to use as a basis for ranking. The proposed techniques are reliable, lightweight and easy to deploy in the modern data center. We have validated these techniques on synthetic data containing injected anomalies and on data acquired from production data centers.

Redundant array of independent disks (RAID) has been widely used to address the reliability and performance issues of storage systems. As the scale of modern storage systems continues growing, disk failure becomes the norm. With the ever-increasing disk capacity, RAID recovery based on disk rebuild becomes more and more costly, which causes significant performance degradation and even unavailability of storage systems. Declustered data layout enables parallel RAID reconstruction by shuffling data and parity blocks among all drives (including spares) in a RAID group. However, the reliability and performance of declustered RAID in real-world storage environments have not been thoroughly studied. With the popularity of ZFS file system and software RAID used in production data centers, in this paper, we extensively evaluate declustered RAID with regard to the RAID recovery time and I/O performance on a high-performance storage platform at Los Alamos National Laboratory. Our empirical study reveals the advantages and disadvantages of declustered RAID technology. We qualitatively characterize the recovery performance of declustered RAID and compare with that of ZFS RAIDZ under various I/O workloads and access patterns. The experimental results show that the speedup of declustered RAID over traditional RAID is sub-linear to the parallelism of recovery I/O. Furthermore, we formally model and analyze the reliability of declustered RAID in terms of the mean-time-to-data-loss (MTTDL) and discover that the improved recovery performance leads to higher storage reliability compared with the traditional RAID.

Cooling power constitutes a large portion of the total electrical power consumption in data centers. Approximately 25%∼40% of the electricity used within a production data center is consumed by the cooling system. Improving the cooling energy efficiency has attracted a great deal of research attention. Many strategies have been proposed for cutting the data center energy costs. One of the effective strategies for increasing the cooling efficiency is using dynamic thermal management. Another effective strategy is placing cooling devices (heat exchangers) closer to the source of heat. This is the basic design principle of many hybrid cooling systems and liquid cooling systems for data centers. Dynamic thermal management of data centers is a huge challenge, due to the fact that data centers are operated under complex dynamic conditions, even during normal operating conditions. In addition, hybrid cooling systems for data centers introduce additional localized cooling devices, such as in row cooling units and overhead coolers, which significantly increase the complexity of dynamic thermal management. Therefore, it is of paramount importance to characterize the dynamic responses of data centers under variations from different cooling units, such as cooling air flow rate variations. In this study, a detailed computational analysis of an in row cooler based hybrid cooled data center is conducted using a commercially available computational fluid dynamics (CFD) code. A representative CFD model for a raised floor data center with cold aisle-hot aisle arrangement fashion is developed. The hybrid cooling system is designed using perimeter CRAH units and localized in row cooling units. The CRAH unit supplies centralized cooling air to the under floor plenum, and the cooling air enters the cold aisle through perforated tiles. The in row cooling unit is located on the raised floor between the server racks. It supplies the cooling air directly to the cold aisle, and intakes hot air from the back of the racks (hot aisle). Therefore, two different cooling air sources are supplied to the cold aisle, but the ways they are delivered to the cold aisle are different. Several modeling cases are designed to study the transient effects of variations in the flow rates of the two cooling air sources. The server power and the cooling air flow variation combination scenarios are also modeled and studied. The detailed impacts of each modeling case on the rack inlet air temperature and cold aisle air flow distribution are studied. The results presented in this work provide an understanding of the effects of air flow variations on the thermal performance of data centers. The results and corresponding analysis is used for improving the running efficiency of this type of raised floor hybrid data centers using CRAH and IRC units.

Cooling is a major component in the enormous energy consumption in data centers. Accurate evaluation of cooling inside a data center forms the backbone of all the attempts for improving cooling efficiency. Models based on computational fluid dynamics (CFD) are typically used for accurate evaluation, but have a drawback of high computation time. This paper presents a novel thermal predictor to evaluate data center cooling in seconds. The key idea is to extract information from a single instance of CFD simulation using metrics called as influence indices to build the fast thermal predictor. Then, this predictor can evaluate the cooling for altered operation of data center with comparable accuracy in seconds without the need for repetitive CFD simulations. This paper demonstrates the accuracy of the thermal predictor by comparing with CFD simulations for a sample, but realistic data center. The fast thermal predictor then successfully passed more challenging tests in a real production data center and proved its practical utility. The results of the thermal predictor compared with measurements carried out in the production data center are also presented. This fast thermal predictor is an important milestone in the development of a method for model-based real time control of data center cooling.

Reliability, availability, and life-cycle cost (LCC) analysis of combined cooling, heating and power (CCHP) integration to data centers considering electricity and cooling supplies

Hard disk drive failure prediction using SMART attribute

The application of Artificial Intelligence (AI) and Machine Learning (ML) models is increasingly leveraged to automate and optimize Data Centre (DC) operations. However, the interpretability and transparency of these complex models pose critical challenges. Hence, this paper explores the Shapley Additive exPlanations (SHAP) values model explainability method for addressing and enhancing the critical interpretability and transparency challenges of predictive maintenance models. This method computes and assigns Shapley values for each feature, then quantifies and assesses their impact on the model’s output. By quantifying the contribution of each feature, SHAP values can assist DC operators in understanding the underlying reasoning behind the model’s output in order to make proactive decisions. As DC operations are dynamically changing, we additionally investigate how SHAP can capture the temporal behaviors of feature importance in the dynamic DC environment over time. We validate our approach with selected predictive models using an actual dataset from a High-Performance Computing (HPC) DC sourced from the Enea CRESCO6 cluster in Italy. The experimental analyses are formalized using summary, waterfall, force, and dependency explanations. We delve into temporal feature importance analysis to capture the features’ impact on model output over time. The results demonstrate that model explainability can improve model transparency and facilitate collaboration between DC operators and AI systems, which can enhance the operational efficiency and reliability of DCs by providing a quantitative assessment of each feature’s impact on the model’s output.

Modern-day data center administrators are finding it increasingly difficult to lower the costs incurred in mechanical cooling of their IT equipment. This is especially true for high-performance computing facilities like Artificial Intelligence, Bitcoin Mining, and Deep Learning, etc. Airside Economization or free air cooling has been out there as a technology for a long time now to reduce the mechanical cooling costs. In free air cooling, under favorable ambient conditions of temperature and humidity, outside air can be used for cooling the IT equipment. In doing so, the IT equipment is exposed to sub-micron particulate/gaseous contaminants that might enter the data center facility with the cooling airflow. The present investigation uses a computational approach to model the airflow paths of particulate contaminants entering inside the IT equipment using a commercially available CFD code. A Discrete Phase Particle modeling approach is chosen to calculate trajectories of the dispersed contaminants. Standard RANS approach is used to model the airflow in the airflow and the particles are superimposed on the flow field by the CFD solver using Lagrangian particle tracking. The server geometry was modeled in 2-D with a combination of rectangular and cylindrical obstructions. This was done to comprehend the effect of change in the obstruction type and aspect ratio on particle distribution. Identifying such discrete areas of contaminant proliferation based on concentration fields due to changing geometries will help with the mitigation of particulate contamination related failures in data centers.

In order to reduce power and energy costs, giant cloud providers now mix online and batch jobs on the same cluster. Although the co-allocation of such jobs improves machine utilization, it challenges the data center scheduler and workload assignment in terms of quality of service, fault tolerance, and failure recovery, especially for latency critical online services. In this paper, we explore various characteristics of co-allocated online services and batch jobs from a production cluster containing 1.3k servers in Alibaba Cloud. From the trace data, we find the following: 1) For batch jobs with multiple tasks and instances, 50.8% failed tasks wait and halted after a very long time interval when their first and the only one instance fails. This wastes much time and resources as the remaining instances are running for an impossible successful termination. 2) For online services jobs, they are clustered in 25 categories according to their requested CPU, memory, and disk resources. Such clustering can help the co-allocation of online services jobs with batch jobs. 3) Servers are clustered into seven groups by CPU utilization, memory utilization, and their correlations. Machines with a strong correlation between CPU and memory utilization provides an opportunity for job co-allocation and resource utilization estimation. 4) The MTBF (mean time between failures) of instances are in the interval [400, 800] seconds while the average completion time of the 99th percentile is 1003 seconds. We also compare the cumulative distribution functions of jobs and servers and explain the differences and opportunities for workload assignment between them. Our findings and insights presented in this paper can help the community and data center operators better understand the workload characteristics, improve resource utilization, and failure recovery design.

Data centers today consume tremendous amount of energy in terms of power distribution and cooling. Dynamic capacity provisioning is a promising approach for reducing energy consumption by dynamically adjusting the number of active machines to match resource demands. However, despite extensive studies of the problem, existing solutions for dynamic capacity provisioning have not fully considered the heterogeneity of both workload and machine hardware found in production environments. In particular, production data centers often comprise several generations of machines with different capacities, capabilities and energy consumption characteristics. Meanwhile, the workloads running in these data centers typically consist of a wide variety of applications with different priorities, performance objectives and resource requirements. Failure to consider heterogenous characteristics will lead to both sub-optimal energy-savings and long scheduling delays, due to incompatibility between workload requirements and the resources offered by the provisioned machines. To address this limitation, in this paper we present HARMONY, a Heterogeneity-Aware Resource Management System for dynamic capacity provisioning in cloud computing environments. Specifically, we first use the K-means clustering algorithm to divide the workload into distinct task classes with similar characteristics in terms of resource and performance requirements. Then we present a novel technique for dynamically adjusting the number of machines of each type to minimize total energy consumption and performance penalty in terms of scheduling delay. Through simulations using real traces from Google's compute clusters, we found that our approach can improve data center energy efficiency by up to 28% compared to heterogeneity-oblivious solutions.

Given the high cost of large-scale data centers, an important design goal is to fully utilize available power resources to maximize the computing capacity. In this paper we present Ampere, a novel power management system for data centers to increase the computing capacity by over-provisioning the number of servers. Instead of doing power capping that degrades the performance of running jobs, we use a statistical control approach to implement dynamic power management by indirectly affecting the workload scheduling, which can enormously reduce the risk of power violations. Instead of being a part of the already over-complicated scheduler, Ampere only interacts with the scheduler with two basic APIs. Instead of power control on the rack level, we impose power constraint on the row level, which leads to more room for over provisioning. We have implemented and deployed Ampere in our production data center. Controlled experiments on 400+ servers show that by adding 17% servers, we can increase the throughput of the data center by 15%, leading to significant cost savings while bringing no disturbances to the job performance.

Network performance monitoring and troubleshooting is a critical but challenging task in data center management. Although many solutions have been proposed in the past decades, it is still difficult to deploy them in the real world because of the expensive cost. In this work, we propose LMon, a network measurement and fault localization system for facilitating long-term maintenance of datacenter networks. LMon employs a lightweight network monitoring tool based on packet pair technique, which supports bi-directional monitoring only on one end host without replacing any device. Moreover, the least measurement deployment policy introduces very small overhead to the network. LMon has been deployed to our production data center on a scale of 50 racks and 500 servers for months. The running experience confirms the efficiency of LMon and points out the future direction of improvement.

Energy management and reduction is one of the major challenges in the Cloud industry's data centres. The Uptime Institute's 2012 report, the industry's leading authority, highlighted the crisis in data centre energy management. 30% of data centres will run out of power, space and cooling in 2012. On average there's a 90% energy overhead in data centre operations, yet 55% of servers use less than 10% of their capacity. The 500,000 data centres worldwide have a server electricity consumption of 400 Billion kWh/year (approx $40 Billion ) which is growing at 12% CAGR, yet only 11% of the centres monitor their consumption. The global CO2 emissions of these centres, is equivalent to the airline industry, 2% of global emissions. It is expected that by 2020 20% of Europe's electrical energy will be data centre related. Data centre cooling requirements also have an enormous hydro footprint. A 1 MW data centre in a U.S-type climate has an annual hydro footprint of 99 million litres/annum Yet studies have shown that operational energy savings of 40% and substantial reductions in physical space, cooling requirements and water consumption can be achieved, where comprehensive and effective energy monitoring and management systems are employed. These can deliver a ROI within months, and further savings which extend over the centre's lifetime. The technologies which have the potential to deliver these benefits, DCIM (Data Centre Infrastructure Management) systems have only emerged in the last 3 to 4 years and are still in their infancy. Not surprisingly, a new global industry providing innovative, green, sustainable data centre technologies will be worth €45 Billion by 2016. The main barriers to effective monitoring/management is instrumentation and training. Vast volumes of data are gathered and must be competently analysed. Most monitoring systems, involve some form of physical metering and cabling and downtime, a major challenge when 10's thousands of servers are involved. Furthermore, all approaches physical and hybrid approaches limit the level of monitoring visibility to the server, individual processes cannot be monitored. A spin-out company from University College Dublin Ireland, Stratergia, has developed a unique data centre energy management system PAPILLON. It is totally software-based, measuring every server's energy consumption in real-time. It is intuitive to use, operates on any platform and can be installed in hours without any downtime or retrofitting. It uses a client-server type architecture. Software agents on servers communicate periodically, to a master server which has power models for each server type. Using the power models, the master computes in real-time and saves in a data base, all relevant energy information of the centre. In operation, Papillon identifies and quantifies energy saving actions that can be implemented, and coupled with electricity network data, more sustainable energy sources can be introduced into the power input of the centre. Papillon has been trialled at Swedish Compare Testlab and is one of the core tools being installed at state-of-the-art data centres at Kajaani, Finland and Cenit, Spain for a new international EU-funded Masters programme in green, sustainable data centre management.

This study examines data center characteristics, loads, control systems, and technologies to identify demand response (DR) and automated DR (Open Auto-DR) opportunities and challenges. The study was performed in collaboration with technology experts, industrial partners, and data center facility managers and existing research on commercial and industrial DR was collected and analyzed. The results suggest that data centers, with significant and rapidly growing energy use, have significant DR potential. Because data centers are highly automated, they are excellent candidates for Open Auto-DR. 'Non-mission-critical' data centers are the most likely candidates for early adoption of DR. Data center site infrastructure DR strategies have been well studied for other commercial buildings; however, DR strategies for information technology (IT) infrastructure have not been studied extensively. The largest opportunity for DR or load reduction in data centers is in the use of virtualization to reduce IT equipment energy use, which correspondingly reduces facility cooling loads. DR strategies could also be deployed for data center lighting, and heating, ventilation, and air conditioning. Additional studies and demonstrations are needed to quantify benefits to data centers of participating in DR and to address concerns about DR's possible impact on data center performance or quality of service and equipment life span.

Data centres are large energy consumers, which have become more energy efficient in recent years due to increased awareness of opportunities, increasing energy costs and corporate social responsibility pressures. A lifecycle assessment approach identifies two other significant areas of impact: the embodied impact of IT equipment and mechanical and electrical plant, and the electricity source used in operational and embodied processes. Many data centre operators publicise their energy efficiency, some also report on their carbon footprint. However, there is a need for simple tools in order to help operators better understand and quantify the embodied impact and inform green procurement. Focussing purely on energy efficiency may cause a burden shift, e.g. by replacing equipment with more efficient equipment but increasing the embodied impact. The total environmental impact could remain the same/increase, but with the perception of a ‘greener’ data centre. Although currently there is limited data available for data centre life cycle assessments and the process is resource intensive, research has identified which factors significantly impact a facility’s environmental impact. This knowledge should be used in the design process and throughout the data centre lifecycle to minimise data centre environmental impact. Practical application: As the data centre industry continues to grow and its sustainability receives closer scrutiny, it is important to increase awareness of where its highest environment impacts lie and analyse the factors which influence this. This can help inform policy and decision-making to support the design and operation of data centres which are truly more sustainable. It is important that this research is not just theoretical but can translate into practical actions which can be implemented in a cost-effective manner for the benefit of everyone. The current focus in industry has been energy consumption and energy efficiency; many practices which were considered innovative a few years ago have now become standard best practice. It is time to examine which other areas should be prioritised for improvement.