Investigative Report on Electrical Commissioning in HPC Data Centers

Servers in a data center are underutilized due to over-provisioning, which contributes heavily toward the high-power consumption of the data centers. Recent research in optimizing the energy consumption of High Performance Computing (HPC) data centers mostly focuses on consolidation of Virtual Machines (VMs) and using dynamic voltage and frequency scaling (DVFS). These approaches are inherently hardware-based, are frequently unique to individual systems, and often use simulation due to lack of access to HPC data centers. Other approaches require profiling information on the jobs in the HPC system to be available before run-time. In this paper, we propose a reinforcement learning based approach, which jointly optimizes profit and energy in the allocation of jobs to available resources, without the need for such prior information. The approach is implemented in a software scheduler used to allocate real applications from the Princeton Application Repository for Shared-Memory Computers (PARSEC) benchmark suite to a number of hardware nodes realized with Odroid-XU3 boards. Experiments show that the proposed approach increases the profit earned by 40% while simultaneously reducing energy consumption by 20% when compared to a heuristic-based approach. We also present a network-aware server consolidation algorithm called Bandwidth-Constrained Consolidation (BCC), for HPC data centers which can address the under-utilization problem of the servers. Our experiments show that the BCC consolidation technique can reduce the power consumption of a data center by up-to 37%.

While High Performance Computing (HPC) is critical for scientific breakthroughs and AI development, its growing energy use (often dependent on fossil fuels) requires a detailed environmental impact analysis. This study addresses the ever-increasing ecological challenges posed by high-performance computing (HPC) data centers by developing a model to quantify and predict their energy consumption and carbon emissions. Our model estimates the annual energy consumption of HPC data centers globally using data on the size and capacity of small and massive data centers. It distinguishes between theoretical maximum and actual energy consumption based on average utilization. The results show an upward trend in carbon emissions associated with HPC, which are projected to reach 5.81 x 1017 grams of carbon dioxide by 2030. As a result, we propose a series of recommendations to reduce the environmental impact of HPC at both the technical and policy levels in favor of promoting a healthy and sustainable environment.

With the advent of myriad different technologies in the modern computing era, the future of high performance computing (HPC) data centers comprising heterogeneous compute servers and networks involves a significantly greater degree of cyber-threats than we are observing currently. As a result, conducting cybersecurity operations at the HPC data centers encompasses a highly cognitive workload environment, where information is rapidly generated, is exceedingly dynamic and is vastly unpredictable. An important challenge faced by cybersecurity specialists is managing the extreme levels of stress and utilizing emerging technologies for monitoring, assessing and defending multiple cyber-attacks in real-time. In this paper, we present an augmented reality based framework for storing, analyzing, visualizing and augmenting data at the HPC centers. This framework is proposed with the goal of improving the data presentation, forecasting, analysis and decision making, thus reducing the time to insight and learning the HPC security operations to combat increasing cyber physical threats.

Efficiently locating information in large-scale distributed systems is a challenging problem to which Peer-to-Peer (P2P) Distributed Hash Tables (DHTs) can provide a highly scalable and cost-effective solution. However, there is very little experience on using DHTs in performance sensitive environments such as High Performance Computing (HPC) datacenters, and there is no published experimental comparison among low-latency DHTs. To fill this gap, we conducted an in-depth performance comparison of three proposed low-latency single-hop DHTs namely 1h-Calot, D1HT, and OneHop. Specifically, we compared experimentally the lookup latency and CPU use of D1HT with those of 1h-Calot by running each of them concurrently with the normal workload production for a subset of 1,800 nodes of a heavy-loaded HPC datacenter. In addition, we carried out an analytical performance comparison among the three single-hop DHTs for system sizes of up to 10 million nodes. The results showed that D1HT consistently had the smallest overhead and in most cases it required one order of magnitude less bandwidth than 1h-Calot and OneHop. Overall, the combination of our experimental and analytical results suggests that D1HT can provide a very effective solution for a broad range of environments, from large-scale HPC datacenters to widely deployed Internet P2P applications such as BitTorrent with up to one million peers. This ability to support such a wide range of environments may allow D1HT to be used as an inexpensive and scalable commodity software substrate for large-scale distributed applications.

The 2007 US EPA report to Congress (US EPA, 2007) on the state of energy consumption in data centers brought to light the true energy inefficiencies built into today’s data centers. Marquez et al. (2008) conducted an initial analysis on the productivity of a Pacific Northwest National Lab computer using The Green Grid’s Data Center Energy Productivity metric (The Green Grid, 2008). Their study highlights how the Top500 ranking of computers disguises the serious energy inefficiency of today’s High Performance Computing data centers. In the rapidly expanding Cloud Computing space, the race will be won by the providers that deliver the lowest cost of computing — such cost is heavily influenced by the operational costs incurred by data centers. As a means to address the urgent need to lower the cost of computing, solution providers have been intensely focusing on real-time monitoring, visualization, and control/management of data centers. The monitoring aspect involves the widespread use of networks of sensors that are used to monitor key data center environmental variables such as temperature, relative humidity, air flow rate, pressure, and energy consumption. Such data is then used to visualize and analyze data center problem areas (e.g., hotspots), which is then followed by control/management actions designed to alleviate such problem areas. The authors have been researching the operational benefits of a network of sensors tied in to a software package that uses the data to visualize, analyze, and control/manage the data center cooling system and IT Equipment for maximum operational efficiency. The research is being conducted in a corporate production data center that is networked in to the authors’ company’s global network of data centers. Results will be presented that highlight the operational benefits that are realizable through real-time monitoring and visualization.

Nowadays, moderating energy consumption and building eco-friendly computing infrastructure is a major goal in large data centers. Moreover, data center energy usage has risen dramatically over the past decade and will continue to grow in-step with the High Performance Computing (HPC) intensive workloads which are at the heart of our modern life. The recent advances in the technology has driven the data center into a new phase of expansion featuring solutions with higher density. To this end, much has been done to increase server efficiency and IT space utilization. In this chapter, we will provide a state-of-the-art overview as regards energy-efficiency in High Performance Computing (HPC) facilities while describing the open challenges the research community has to face in the coming years to enable the building and usage of an Exascale platform by 2020.

High performance datacenters that are being built and operated to ensure optimized compute density for high performance computing (HPC) workloads are constrained by the requirement to provide adequate cooling for the servers. Traditional methods of cooling dense high power servers using air cooling imposes a large cooling and power burden on datacenters. Airflow optimization of the datacenter is a constraint subject to a high energy penalty when dense power hungry racks each capable of consuming 30 to 40 kW are populated in a dense datacenter environment. The work documented using a simulation model (TileFlow) in this paper demonstrates the challenges associated with a standard air cooled approach in a HPC datacenter. Alternate cooling approaches to traditional air cooling are simulated as a comparison to traditional air cooling. These include models using a heat exchanger assisted rack cooling solution with conventional chilled water and, a direct to node cooling model simulated for the racks. These three distinct data center models are simulated at varying workloads and the resulting data is presented for typical and maximal inlet temperatures to the racks. For each cooling solution an estimate of the energy spend for the servers is determined based on the estimated PUEs of the cooling solutions chosen.

Silicon photonics has emerged as a promising solution to address the interconnect bottleneck in high performance computing (HPC) systems and data centers. Silicon photonics provides unprecedented I/O bandwidth, enabling ultrahigh aggregated bandwidth (~ 10 Tbps), high bandwidth density (~ Tbps/mm), and high energy efficiency (~ pJ/bit). In addition, silicon photonics also enable optical switching with large radix and short switching time. This talk will provide an overview of the state of the art of silicon photonic switches, with emphasis on new micro-electromechanical-system (MEMS)-actuated switching mechanism. Large scale switches with 64x64 and 128x128 port counts have been demonstrated, as well as wavelength-selective switches with 8x8 ports and 8 wavelengths. Future scaling to even larger port count will be discussed.

Data rates are continuing to increase for box-to-box, rack-to-rack, board-to-board, and chip-to-chip interconnects for terabit switches and routers, multiprocessor computers and high-end servers. The increase in individual line rates and bandwidth drives the need to replace copper interconnects with optical interconnects. Fiber optics are advantageous for these applications because they allow for longer link lengths, increased bandwidth, smaller cables and connectors, less susceptibility to electromagnetic interference, and potentially lower power dissipation, depending on the data rate and link length. Towards this direction, this thesis aims to design and demonstrate low-cost, low-latency, high throughput, rack-to-rack optical interconnect architectures for exascale (i.e., performing 10^18 floating point operations per second) high-performance computing (HPC) systems and data centers. In particular, a novel, cost-effective, optical interconnect architecture for ultrafast optical switching, based on semiconductor optical amplifiers (SOAs), is studied. The proposed design of a fast N×N all-optical, wavelength-space crossbar switch for optical interconnects uses a minimum number of ON/OFF gates. This thesis compares and proves the superiority of the proposed architecture with respect to its originally-proposed counterpart, both theoretically and experimentally. Additionally, in order to increase the capacity and to minimize the impact of transmission effects (especially self-gain modulation (SGM), cross-gain modulation (XGM), self-phase modulation (SPM), cross-phase modulation (XPM), and polarization dependent gain (PDG)), we investigate the performance of conventional binary intensity modulation (IM), in conjunction with direct detection, as well as of advanced, more resilient, spectrally-efficient modulation formats (e.g., Differential Phase Shift Keying (DPSK), Orthogonal Frequency Division Multiplexing (OFDM), Polarization Division Multiplexed Quadrature Phase Shift Keying (PDM-QPSK), Single (SP)- and PDM- 16-ary Quadrature Amplitude Modulation (16QAM) in conjunction with coherent detection). Finally, as a seperate research activity, we study the performance of point-to-point links based on vertical-cavity surface-emitting lasers (VCSELs) and single- or multi- mode fibers, in conjuction with IM/DD, four-level Pulse Amplitude Modulation (4-PAM), and OFDM, to enable state-of-the-art, high-capacity, low-cost optical interconnects.

As the science and engineering demand for high performance computing (HPC) grows beyond leading edge research institutions and communities to encompass routine activities of many disciplines, computing center infrastructures expand in their size, density and power demands. Long-practiced HPC center enhancements are increasingly demanding cooling methods that extend well beyond the capabilities of implementations that have become the staple of facility designs, even beyond those of the now-dominant architecture - commodity-processor-based, air-cooled rack clusters. As compute capacity aggressively increases in HPC centers, the power and cooling requirements, and thus cost of operation of the facilities, continues to rise correspondingly. This growth increasingly taxes the abilities of the hosting organizations to accommodate these demands. To address the challenge of meeting such pervasive demands, this paper examines energy efficiency in existing data centers from a two-pronged approach: employing direct water cooling and optimizing the facility infrastructure with as little capital investment to the building as possible.

Finding the best energy-performance tradeoffs for High Performance Computing (HPC) applications is a major challenge for many modern supercomputing centers. With the increased focus on data center energy efficiency and the emergence of possible data center power constraints, making the right decision at a given time is becoming more important. A real-world situation like can a given 1000 compute node application be executed at a maximum of 2.7 GHz CPU frequency without going over the energy provider defined power band, or the available monthly energy limit? is just one example of the types of decisions HPC data centers will face. The previously developed Adaptive Energy and Power Consumption Prediction (AEPCP) model answers this question for the case of a fixed CPU frequency. This paper will extend the AEPCP process to enable the development of analytical models for estimating application execution time, power, and energy consumptions as functions of the number of compute nodes and maximum operating CPU frequency. Based on these analytical models, an adaptive model (Lightweight Adaptive Consumption Prediction (LACP)) is presented that implements the extended prediction process. This information allows for improved estimation of potential energy-performance costs and tradeoffs of applications and thus identifies the optimal resource configuration for specific data center boundary conditions.

Modern High Performance Computing (HPC) applications, such as Earth science simulations, produce large amounts of data due to the surging of computing power, while big data applications have become more compute-intensive due to increasingly sophisticated analysis algorithms. The needs of both HPC and big data technologies for advanced HPC and big data applications create a demand for integrated system support. In this study, we introduce Scientific Data Processing (SciDP) to support both HPC and big data applications via integrated scientific data processing. SciDP can directly process scientific data stored on a Parallel File System (PFS), which is typically deployed in an HPC environment, in a big data programming environment running atop Hadoop Distributed File System (HDFS). SciDP seamlessly integrates PFS, HDFS, and the widely-used R data analysis system to support highly efficient processing of scientific data. It utilizes the merits of both PFS and HDFS for fast data transfer, overlaps computing with data accessing, and integrates R into the data transfer process. Experimental results show that SciDP accelerates analysis and visualization of a production NASA Center for Climate Simulation (NCCS) climate and weather application by 6x to 8x when compared to existing solutions.

Relevance. Data centers are a specialized facility that is an interconnected system of IT infrastructure, engineering infra-structure, equipment (server and network) and parts of which are located in a building or room connected to external networks, both engineering and telecommunications. Due to the significant heat generation of these systems, there is a need to increase the efficiency of cooling systems through efficient heat recovery.The purpose. Calculation of the energy balance of a typical data center designed for operation in central Russia.Methods. The developed energy model of a data center is a mathematical description of all energy processes in a building, including heat transfer processes, the effects of ventilation and air conditioning, the operation of lighting, electrical appliances and other equipment, and the operation of engineering systems.Results. According to the simulation results, the specific energy consumption was 1,353 kWh/m2, the specific annual cost of energy resources was 8,572.90 rubles/m2. The following key results were obtained: energy consumed by data center modules – 38,399.53 MWh/year; energy consumed by IT equipment – 31,685.20 MWh/year; PUE (it represents the ratio of the total energy consumed by the data center to the energy directly used by the IT equipment) – 1.21.Conclusion. Data on annual energy consumption, the cost of consumed energy resources and the amount of CO2 released into the atmosphere during the production of energy resources were obtained. A monthly energy consumption chart has been developed by category (heating, cooling, fans, heat recovery, pumps, interior lighting, outlet equipment, servers). In the course of further research, it is planned to consider priority schemes for the utilization of thermal energy of the data center.

Data center (DC) and high performance computing (HPC) applications have traditionally used a combination of copper, multimode fiber and single-mode fiber interconnects with relative percentages that depend on factors such as the line rate, reach and connectivity costs. The balance between these transmission media has increasingly shifted towards optical fiber due to the reach constraints of copper at data rates of 10 Gb/s and higher. The percentage of single-mode fiber deployed in the DC has also grown slightly since 2014, coinciding with the emergence of mega DCs with extended distance needs beyond 100 m. This trend will likely continue in the next few years as DCs expand their capacity from 100G to 400G, increase the physical size of their facilities and begin to utilize silicon-photonics transceiver technology. However there is a still a need for the low-cost and high-density connectivity, and this is sustaining the deployment of multimode fiber for links ≤ 100 m. In this paper, we discuss options for single-mode and multimode fibers in DCs and HPCs and introduce a reduced diameter multimode fiber concept which provides intra-and inter-rack connectivity as well as compatibility with silicon-photonic transceivers operating at 1310 nm. We also discuss the trade-offs between single-mode fiber attributes such as bend-insensitivity, attenuation and mode field diameter and their roles in capacity and connectivity in data centers.

The application of Artificial Intelligence (AI) and Machine Learning (ML) models is increasingly leveraged to automate and optimize Data Centre (DC) operations. However, the interpretability and transparency of these complex models pose critical challenges. Hence, this paper explores the Shapley Additive exPlanations (SHAP) values model explainability method for addressing and enhancing the critical interpretability and transparency challenges of predictive maintenance models. This method computes and assigns Shapley values for each feature, then quantifies and assesses their impact on the model’s output. By quantifying the contribution of each feature, SHAP values can assist DC operators in understanding the underlying reasoning behind the model’s output in order to make proactive decisions. As DC operations are dynamically changing, we additionally investigate how SHAP can capture the temporal behaviors of feature importance in the dynamic DC environment over time. We validate our approach with selected predictive models using an actual dataset from a High-Performance Computing (HPC) DC sourced from the Enea CRESCO6 cluster in Italy. The experimental analyses are formalized using summary, waterfall, force, and dependency explanations. We delve into temporal feature importance analysis to capture the features’ impact on model output over time. The results demonstrate that model explainability can improve model transparency and facilitate collaboration between DC operators and AI systems, which can enhance the operational efficiency and reliability of DCs by providing a quantitative assessment of each feature’s impact on the model’s output.