Mitigating the Environmental Impact of High-Performance Computing Data Centers

Servers in a data center are underutilized due to over-provisioning, which contributes heavily toward the high-power consumption of the data centers. Recent research in optimizing the energy consumption of High Performance Computing (HPC) data centers mostly focuses on consolidation of Virtual Machines (VMs) and using dynamic voltage and frequency scaling (DVFS). These approaches are inherently hardware-based, are frequently unique to individual systems, and often use simulation due to lack of access to HPC data centers. Other approaches require profiling information on the jobs in the HPC system to be available before run-time. In this paper, we propose a reinforcement learning based approach, which jointly optimizes profit and energy in the allocation of jobs to available resources, without the need for such prior information. The approach is implemented in a software scheduler used to allocate real applications from the Princeton Application Repository for Shared-Memory Computers (PARSEC) benchmark suite to a number of hardware nodes realized with Odroid-XU3 boards. Experiments show that the proposed approach increases the profit earned by 40% while simultaneously reducing energy consumption by 20% when compared to a heuristic-based approach. We also present a network-aware server consolidation algorithm called Bandwidth-Constrained Consolidation (BCC), for HPC data centers which can address the under-utilization problem of the servers. Our experiments show that the BCC consolidation technique can reduce the power consumption of a data center by up-to 37%.

The Energy Efficient High Performance Computing Working Group (EE HPC WG) has assembled a small diverse team to write a short investigative report on electrical commissioning. The purpose of the investigative report is to evaluate the need for electrical commissioning guidelines specific to High Performance Computing (HPC) data centers given their unique IT equipment load densities and power profiles. It is the consensus of the team that special electrical commissioning guidelines are needed and the EE HPC WG will author the initial guidelines. The scope of the guidelines will include the static and dynamic electrical aspects of commissioning practices that are specific to high performance computing and more importantly, cover the transient aspects of electrical commissioning. The fluctuating nature of many compute nodes can dramatically influence generation, transmission, and distribution of electrical power. HPC data center lessons learned and best practices will be examined and used to enhance the electrical commissioning guidelines. The primary audience for the guidelines is facility engineers and operators of HPC data centers. The guidelines will also be applicable to others that support HPC data centers, ranging from utilities and their electrical grid infrastructure to IT equipment manufacturers whose machines are being commissioned at the end of the process.

With the advent of myriad different technologies in the modern computing era, the future of high performance computing (HPC) data centers comprising heterogeneous compute servers and networks involves a significantly greater degree of cyber-threats than we are observing currently. As a result, conducting cybersecurity operations at the HPC data centers encompasses a highly cognitive workload environment, where information is rapidly generated, is exceedingly dynamic and is vastly unpredictable. An important challenge faced by cybersecurity specialists is managing the extreme levels of stress and utilizing emerging technologies for monitoring, assessing and defending multiple cyber-attacks in real-time. In this paper, we present an augmented reality based framework for storing, analyzing, visualizing and augmenting data at the HPC centers. This framework is proposed with the goal of improving the data presentation, forecasting, analysis and decision making, thus reducing the time to insight and learning the HPC security operations to combat increasing cyber physical threats.

The 2007 US EPA report to Congress (US EPA, 2007) on the state of energy consumption in data centers brought to light the true energy inefficiencies built into today’s data centers. Marquez et al. (2008) conducted an initial analysis on the productivity of a Pacific Northwest National Lab computer using The Green Grid’s Data Center Energy Productivity metric (The Green Grid, 2008). Their study highlights how the Top500 ranking of computers disguises the serious energy inefficiency of today’s High Performance Computing data centers. In the rapidly expanding Cloud Computing space, the race will be won by the providers that deliver the lowest cost of computing — such cost is heavily influenced by the operational costs incurred by data centers. As a means to address the urgent need to lower the cost of computing, solution providers have been intensely focusing on real-time monitoring, visualization, and control/management of data centers. The monitoring aspect involves the widespread use of networks of sensors that are used to monitor key data center environmental variables such as temperature, relative humidity, air flow rate, pressure, and energy consumption. Such data is then used to visualize and analyze data center problem areas (e.g., hotspots), which is then followed by control/management actions designed to alleviate such problem areas. The authors have been researching the operational benefits of a network of sensors tied in to a software package that uses the data to visualize, analyze, and control/manage the data center cooling system and IT Equipment for maximum operational efficiency. The research is being conducted in a corporate production data center that is networked in to the authors’ company’s global network of data centers. Results will be presented that highlight the operational benefits that are realizable through real-time monitoring and visualization.

Nowadays, moderating energy consumption and building eco-friendly computing infrastructure is a major goal in large data centers. Moreover, data center energy usage has risen dramatically over the past decade and will continue to grow in-step with the High Performance Computing (HPC) intensive workloads which are at the heart of our modern life. The recent advances in the technology has driven the data center into a new phase of expansion featuring solutions with higher density. To this end, much has been done to increase server efficiency and IT space utilization. In this chapter, we will provide a state-of-the-art overview as regards energy-efficiency in High Performance Computing (HPC) facilities while describing the open challenges the research community has to face in the coming years to enable the building and usage of an Exascale platform by 2020.

Environment-conscious scheduling of HPC applications on distributed Cloud-oriented data centers

Efficiently locating information in large-scale distributed systems is a challenging problem to which Peer-to-Peer (P2P) Distributed Hash Tables (DHTs) can provide a highly scalable and cost-effective solution. However, there is very little experience on using DHTs in performance sensitive environments such as High Performance Computing (HPC) datacenters, and there is no published experimental comparison among low-latency DHTs. To fill this gap, we conducted an in-depth performance comparison of three proposed low-latency single-hop DHTs namely 1h-Calot, D1HT, and OneHop. Specifically, we compared experimentally the lookup latency and CPU use of D1HT with those of 1h-Calot by running each of them concurrently with the normal workload production for a subset of 1,800 nodes of a heavy-loaded HPC datacenter. In addition, we carried out an analytical performance comparison among the three single-hop DHTs for system sizes of up to 10 million nodes. The results showed that D1HT consistently had the smallest overhead and in most cases it required one order of magnitude less bandwidth than 1h-Calot and OneHop. Overall, the combination of our experimental and analytical results suggests that D1HT can provide a very effective solution for a broad range of environments, from large-scale HPC datacenters to widely deployed Internet P2P applications such as BitTorrent with up to one million peers. This ability to support such a wide range of environments may allow D1HT to be used as an inexpensive and scalable commodity software substrate for large-scale distributed applications.

Finding the best energy-performance tradeoffs for High Performance Computing (HPC) applications is a major challenge for many modern supercomputing centers. With the increased focus on data center energy efficiency and the emergence of possible data center power constraints, making the right decision at a given time is becoming more important. A real-world situation like can a given 1000 compute node application be executed at a maximum of 2.7 GHz CPU frequency without going over the energy provider defined power band, or the available monthly energy limit? is just one example of the types of decisions HPC data centers will face. The previously developed Adaptive Energy and Power Consumption Prediction (AEPCP) model answers this question for the case of a fixed CPU frequency. This paper will extend the AEPCP process to enable the development of analytical models for estimating application execution time, power, and energy consumptions as functions of the number of compute nodes and maximum operating CPU frequency. Based on these analytical models, an adaptive model (Lightweight Adaptive Consumption Prediction (LACP)) is presented that implements the extended prediction process. This information allows for improved estimation of potential energy-performance costs and tradeoffs of applications and thus identifies the optimal resource configuration for specific data center boundary conditions.

In the presence of latency sensitive geo-distributed applications, users require fast service for their queries. Cloud computing provides physical servers from its data center in order to process user requests. The cloud data center consumes a huge amount of energy due to lack of management of the data center servers as the container-based service consolidation is a nontrivial task. Since the containers require less resource footprint, consolidating it in servers might make resource availability sparse. In order to reduce the energy consumption of the cloud data center, we have proposed a green container-based consolidation of the services so that the maximum number of servers can be put into idle mode without affecting the application quality of experience. The service consolidation problem has been formulated as an optimization problem considering minimization of total energy consumption of the data center as the objective, and an algorithm named Energy Aware Service consolidation using baYesian optimization (EASY) has been proposed to solve the optimization. We have evaluated the EASY algorithm in simulation using python. The experimental results have shown that EASY improves the total energy consumption of the data centers. This improvement comes at the cost of a small increase of service response time as there exists a trade-off between energy consumption and service response time.

Silicon photonics has emerged as a promising solution to address the interconnect bottleneck in high performance computing (HPC) systems and data centers. Silicon photonics provides unprecedented I/O bandwidth, enabling ultrahigh aggregated bandwidth (~ 10 Tbps), high bandwidth density (~ Tbps/mm), and high energy efficiency (~ pJ/bit). In addition, silicon photonics also enable optical switching with large radix and short switching time. This talk will provide an overview of the state of the art of silicon photonic switches, with emphasis on new micro-electromechanical-system (MEMS)-actuated switching mechanism. Large scale switches with 64x64 and 128x128 port counts have been demonstrated, as well as wavelength-selective switches with 8x8 ports and 8 wavelengths. Future scaling to even larger port count will be discussed.

High energy consumption in data center has become a key problem, and the Carbon (CO $$_2$$ ) emissions from data center have serious impacts on environment. To improve traditional performance-energy model and minimize energy consumption in cloud data center, an energy-efficient and QoS-aware (EEQoS) model is developed for virtual resources consolidation. The EEQoS model is designed to investigate the trade-off between energy and QoS. The user’s satisfaction degree and Euclidean distance of resources are also respectively applied to the QoS model and power model. In addition, this EEQoS model is combined with the particle swarm optimization algorithm by setting the power consumption per QoS value as the objective function to consolidate virtual resources in data center. Compared with traditional power model, the proposed model reduces response time by 27.2%, cost by 31.4%, SLA violations by 40.5% and improves throughput by 13.5% on average while increasing less than 3.8% energy consumption during the whole test period.

High performance datacenters that are being built and operated to ensure optimized compute density for high performance computing (HPC) workloads are constrained by the requirement to provide adequate cooling for the servers. Traditional methods of cooling dense high power servers using air cooling imposes a large cooling and power burden on datacenters. Airflow optimization of the datacenter is a constraint subject to a high energy penalty when dense power hungry racks each capable of consuming 30 to 40 kW are populated in a dense datacenter environment. The work documented using a simulation model (TileFlow) in this paper demonstrates the challenges associated with a standard air cooled approach in a HPC datacenter. Alternate cooling approaches to traditional air cooling are simulated as a comparison to traditional air cooling. These include models using a heat exchanger assisted rack cooling solution with conventional chilled water and, a direct to node cooling model simulated for the racks. These three distinct data center models are simulated at varying workloads and the resulting data is presented for typical and maximal inlet temperatures to the racks. For each cooling solution an estimate of the energy spend for the servers is determined based on the estimated PUEs of the cooling solutions chosen.

Cloud computing is the combination of grid computing, distributed computing and utility computing. Cloud computing provides various types of services (servers and storage facility) in the on demand basis. The main goal of the cloud computing is to preserve and organize the very huge data center or data forms. The data forms are composed of thousands of servers that absorb the giant (ample) amount of electricity in the word of energy. The decreasing the energy consumption in datacenter is the major challenge in cloud computing now a day. This research article is going to address the problem of high energy consumption at datacenter. Concentrate on virtual machine scheduling in cloud datacenter with Dynamic Voltage Frequency Scaling (DVFS) approach. We have combined shortest job first and Round Robin algorithms with Vibrant Quantum. This combination of algorithm is considered as shortest round vibrant queue (SRVQ) algorithm. SRVQ reduces the waiting time of the scheduling process and minimize the starvation. The DVFS and SRVQ worked together and produced fruitful results in the final experiments. This work reduced the server’s energy consumption in the cloud data center. In the final results, our proposed framework exhibits 45% of energy efficiency compare to other previously proposed algorithms. 33% of QoS performance were enhanced by our framework.

Data rates are continuing to increase for box-to-box, rack-to-rack, board-to-board, and chip-to-chip interconnects for terabit switches and routers, multiprocessor computers and high-end servers. The increase in individual line rates and bandwidth drives the need to replace copper interconnects with optical interconnects. Fiber optics are advantageous for these applications because they allow for longer link lengths, increased bandwidth, smaller cables and connectors, less susceptibility to electromagnetic interference, and potentially lower power dissipation, depending on the data rate and link length. Towards this direction, this thesis aims to design and demonstrate low-cost, low-latency, high throughput, rack-to-rack optical interconnect architectures for exascale (i.e., performing 10^18 floating point operations per second) high-performance computing (HPC) systems and data centers. In particular, a novel, cost-effective, optical interconnect architecture for ultrafast optical switching, based on semiconductor optical amplifiers (SOAs), is studied. The proposed design of a fast N×N all-optical, wavelength-space crossbar switch for optical interconnects uses a minimum number of ON/OFF gates. This thesis compares and proves the superiority of the proposed architecture with respect to its originally-proposed counterpart, both theoretically and experimentally. Additionally, in order to increase the capacity and to minimize the impact of transmission effects (especially self-gain modulation (SGM), cross-gain modulation (XGM), self-phase modulation (SPM), cross-phase modulation (XPM), and polarization dependent gain (PDG)), we investigate the performance of conventional binary intensity modulation (IM), in conjunction with direct detection, as well as of advanced, more resilient, spectrally-efficient modulation formats (e.g., Differential Phase Shift Keying (DPSK), Orthogonal Frequency Division Multiplexing (OFDM), Polarization Division Multiplexed Quadrature Phase Shift Keying (PDM-QPSK), Single (SP)- and PDM- 16-ary Quadrature Amplitude Modulation (16QAM) in conjunction with coherent detection). Finally, as a seperate research activity, we study the performance of point-to-point links based on vertical-cavity surface-emitting lasers (VCSELs) and single- or multi- mode fibers, in conjuction with IM/DD, four-level Pulse Amplitude Modulation (4-PAM), and OFDM, to enable state-of-the-art, high-capacity, low-cost optical interconnects.

The growth of energy consumption in data centers is becoming one of the significant problems all over the world. The power usage of IT devices and air-conditioning (AC) units accounts for up to three-quarters of total energy consumption in typical data centers. This study discusses the effectiveness of using shutter-controlled air flow management to control cold air flow into the servers. It can impact the power consumption of both, the servers and the AC units. A raised floor data center has two types of aisles - cold and hot. In most cases, excessive volumes of cold air are pumped into the servers using cold aisles. We are proposing the utilization of a shutter to control the air flow and reduce wastage while preventing damage due to increased CPU temperature. The shutter is intended to control the air flow and maintain the temperature appropriately according to the workload of the servers. We evaluated the proposed system for controlling the flow of air into the servers. The results show that cold air volumes can be reduced by up to 92% while maintaining appropriate control of CPU temperature.