Combining cooling technology and facility design to improve HPC data center energy efficiency

With the advent of myriad different technologies in the modern computing era, the future of high performance computing (HPC) data centers comprising heterogeneous compute servers and networks involves a significantly greater degree of cyber-threats than we are observing currently. As a result, conducting cybersecurity operations at the HPC data centers encompasses a highly cognitive workload environment, where information is rapidly generated, is exceedingly dynamic and is vastly unpredictable. An important challenge faced by cybersecurity specialists is managing the extreme levels of stress and utilizing emerging technologies for monitoring, assessing and defending multiple cyber-attacks in real-time. In this paper, we present an augmented reality based framework for storing, analyzing, visualizing and augmenting data at the HPC centers. This framework is proposed with the goal of improving the data presentation, forecasting, analysis and decision making, thus reducing the time to insight and learning the HPC security operations to combat increasing cyber physical threats.

Traditional high performance computing (HPC) centers that operate a single large supercomputer cluster have not required sophisticated mechanisms to manage and enforce network policies. Recently, HPC centers have expanded to support a wide range of computational infrastructure, such as OpenStack-based private clouds and Ceph object stores, each with its own unique characteristics and network security requirements. Network security policies are becoming more complex and harder to manage. To address the challenge, this paper explores ways to define and manage the new network policies required by emerging HPC systems. As the first step, we identify the new types of policies that are required and the technical capabilities needed to support them. We present example policies and discuss ways to implement those policies using emerging programmable networks and intent-based networks. We describe our initial work toward automatically converting human readable network policies into network configurations and programmable network controllers that implement those policies using business rule management systems.

Monitoring and assessing the energy efficiency of supercomputers and data centers is crucial in order to limit and reduce their energy consumption. Applications from the domain of High Performance Computing (HPC), such as MPI applications, account for a significant fraction of the overall energy consumed by HPC centers. Simulation is a popular approach for studying the behavior of these applications in a variety of scenarios, and it is therefore advantageous to be able to study their energy consumption in a cost-efficient, controllable, and also reproducible simulation environment. Alas, simulators supporting HPC applications commonly lack the capability of predicting the energy consumption, particularly when target platforms consist of multi-core nodes. In this work, we aim to accurately predict the energy consumption of MPI applications via simulation. Firstly, we introduce the models required for meaningful simulations: The computation model, the communication model, and the energy model of the target platform. Secondly, we demonstrate that by carefully calibrating these models on a single node, the predicted energy consumption of HPC applications at a larger scale is very close (within a few percents) to real experiments. We further show how to integrate such models into the SimGrid simulation toolkit. In order to obtain good execution time predictions on multi-core architectures, we also establish that it is vital to correctly account for memory effects in simulation. The proposed simulator is validated through an extensive set of experiments with wellknown HPC benchmarks. Lastly, we show the simulator can be used to study applications at scale, which allows researchers to save both time and resources compared to real experiments.

The Energy Efficient High Performance Computing Working Group (EE HPC WG) has assembled a small diverse team to write a short investigative report on electrical commissioning. The purpose of the investigative report is to evaluate the need for electrical commissioning guidelines specific to High Performance Computing (HPC) data centers given their unique IT equipment load densities and power profiles. It is the consensus of the team that special electrical commissioning guidelines are needed and the EE HPC WG will author the initial guidelines. The scope of the guidelines will include the static and dynamic electrical aspects of commissioning practices that are specific to high performance computing and more importantly, cover the transient aspects of electrical commissioning. The fluctuating nature of many compute nodes can dramatically influence generation, transmission, and distribution of electrical power. HPC data center lessons learned and best practices will be examined and used to enhance the electrical commissioning guidelines. The primary audience for the guidelines is facility engineers and operators of HPC data centers. The guidelines will also be applicable to others that support HPC data centers, ranging from utilities and their electrical grid infrastructure to IT equipment manufacturers whose machines are being commissioned at the end of the process.

Continuous integration (CI) has become a ubiquitous practice in modern software development, with major code hosting services offering free automation on popular platforms. CI offers major benefits, as it enables detecting bugs in code prior to committing changes. While high-performance computing (HPC) research relies heavily on software, HPC machines are not considered “common” platforms. This presents several challenges that hinder the adoption of CI in HPC environments, making it difficult to maintain bug-free HPC projects, and resulting in adverse effects on the research community. In this article, we explore the challenges that impede HPC CI, such as hardware diversity, security, isolation, administrative policies, and nonstandard authentication, environments, and job submission mechanisms. We propose several solutions that could enhance the quality of HPC software and the experience of developers. Implementing these solutions would require significant changes at HPC centers, but if these changes are made, it would ultimately enable faster and better science.

Servers in a data center are underutilized due to over-provisioning, which contributes heavily toward the high-power consumption of the data centers. Recent research in optimizing the energy consumption of High Performance Computing (HPC) data centers mostly focuses on consolidation of Virtual Machines (VMs) and using dynamic voltage and frequency scaling (DVFS). These approaches are inherently hardware-based, are frequently unique to individual systems, and often use simulation due to lack of access to HPC data centers. Other approaches require profiling information on the jobs in the HPC system to be available before run-time. In this paper, we propose a reinforcement learning based approach, which jointly optimizes profit and energy in the allocation of jobs to available resources, without the need for such prior information. The approach is implemented in a software scheduler used to allocate real applications from the Princeton Application Repository for Shared-Memory Computers (PARSEC) benchmark suite to a number of hardware nodes realized with Odroid-XU3 boards. Experiments show that the proposed approach increases the profit earned by 40% while simultaneously reducing energy consumption by 20% when compared to a heuristic-based approach. We also present a network-aware server consolidation algorithm called Bandwidth-Constrained Consolidation (BCC), for HPC data centers which can address the under-utilization problem of the servers. Our experiments show that the BCC consolidation technique can reduce the power consumption of a data center by up-to 37%.

Nowadays, moderating energy consumption and building eco-friendly computing infrastructure is a major goal in large data centers. Moreover, data center energy usage has risen dramatically over the past decade and will continue to grow in-step with the High Performance Computing (HPC) intensive workloads which are at the heart of our modern life. The recent advances in the technology has driven the data center into a new phase of expansion featuring solutions with higher density. To this end, much has been done to increase server efficiency and IT space utilization. In this chapter, we will provide a state-of-the-art overview as regards energy-efficiency in High Performance Computing (HPC) facilities while describing the open challenges the research community has to face in the coming years to enable the building and usage of an Exascale platform by 2020.

Modern High Performance Computing (HPC) applications process very large amounts of data. A critical research challenge lies in transporting input data to the HPC center from a number of distributed sources, e.g., scientific experiments and web repositories, etc., and offloading the result data to geographically distributed, intermittently available end-users, often over under-provisioned connections. Such end-user data services are typically performed using point-to-point transfers that are designed for well-endowed sites and are unable to reconcile the center's resource usage and users' delivery deadlines, unable to adapt to changing dynamics in the end-to-end data path and are not fault-tolerant. To overcome these inefficiencies, decentralized HPC data services are emerging as viable alternatives. In this paper, we develop and enhance such distributed data services by designing CATCH, a Cloud-based Adaptive data Transfer service for HPC. CATCH leverages a bevy of cloud storage resources to orchestrate a decentralized data transport with fail-over capabilities. Our results demonstrate that CATCH is a feasible approach, and can help improve the data transfer times at the HPC center by as much as 81.1\% for typical HPC workloads.

The EUDAT project is a pan-European data initiative that started in October 2011. The project brings together a unique consortium of 25 partners - including research communities, national data and high performance computing (HPC) centres, technology providers, and funding agencies - from 13 countries. EUDAT aims to build a sustainable cross-disciplinary and cross-national data infrastructure that provides a set of shared services for accessing and preserving research data. The design and deployment of these services is being coordinated by multi-disciplinary task forces comprising representatives from research communities and data centres. This short paper presents the achievements of the project during its first year and describes the services that have been chosen to meet the requirements of the initial research communities involved in the project.

Performance models of high performance computing (HPC) applications are important for several reasons. First, they provide insight to designers of HPC systems on the role of subsystems such as the processor or the network in determining application performance. Second, they allow HPC centers more accurately to target procurements to resource requirements. Third, they can be used to identify application performance bottlenecks and to provide insights about scalability issues. The suitability of a performance model, however, for a particular performance investigation is a function of both the accuracy and the cost of the model. A semi-empirical model previously published by the authors for an astrophysics application was shown to be inaccurate when predicting communication cost for large numbers of processors. It is hypothesized that this deficiency is due to the inability of the model adequately to capture communication contention (threshold effects) as well as other unmodeled components such as noise and I/O contention. In this paper we present a new approach to capture these unknown features to improve the predictive capabilities of the model. This approach uses a systematic model error-correction procedure that uses evolutionary algorithms to find an error correction term to augment the eXisting model. Four variations of this procedure were investigated and all were shown to produce better results than the original model. Successful cross-platform application of this approach showed that it adequately captures machine dependent characteristics. This approach was then successfully demonstrated for a second application, further showing its versatility.

While High Performance Computing (HPC) is critical for scientific breakthroughs and AI development, its growing energy use (often dependent on fossil fuels) requires a detailed environmental impact analysis. This study addresses the ever-increasing ecological challenges posed by high-performance computing (HPC) data centers by developing a model to quantify and predict their energy consumption and carbon emissions. Our model estimates the annual energy consumption of HPC data centers globally using data on the size and capacity of small and massive data centers. It distinguishes between theoretical maximum and actual energy consumption based on average utilization. The results show an upward trend in carbon emissions associated with HPC, which are projected to reach 5.81 x 1017 grams of carbon dioxide by 2030. As a result, we propose a series of recommendations to reduce the environmental impact of HPC at both the technical and policy levels in favor of promoting a healthy and sustainable environment.

Quality of service is important feature for a High Performance Computing Center (HPC) center like Partnership for an Advanced Computing Environment (PACE) center in Georgia Institute of Technology (Georgia Tech). The user’s job fails running on a HPC center may due to a spectral of reasons, one of major contributor is the hardware and network failure. Reducing the hardware failure rate can significantly increase a data center’s quality of service as well as reducing the cost of human intervention. This is critical during PACE’s transition to a fee-based service model in which uptime correlates directly with revenue. PACE has around 9 millions jobs each year with 12% of job failure rate. In order to extend service life of hardware and reduce the potential failure and data center’s cost, we present a machine learning method to understand the center’s host usage pattern. By clustering the hosts based on multiple features, we reshuffle the host list to avoid the hosts being overused over time. We build a test framework which runs the complex combination of experiments, and presents the ad hoc comparisons. We intend to make the machine learning method in a rack aware fashion, and show the meaningful result with rack information included.

1 - Data center architectures

Cyber Equations, Inc. is reporting on the results of its DOE SBIR Phase I project for the development of a Scalable HPC Insider Threat Monitoring System. This final report to the Department of Energy for Task DE-FOA-0001618 covers the period from 02/21/2017 to 02/20/2018 and summarizes the larger research accomplishments, which can be divided in theoretical and laboratory experiments. One of the many challenges for the U.S. Department of Energy (DOE) is the support of the high-performance computing (HPC) centers while providing quality information services, strategic vision, leadership, technical expertise, and capabilities necessary to enable the DOE to deliver effective, efficient, economical, and secure HPC services. In order to realize a secure DOE laboratory enterprise, DOE facilities operations must counter a wide range of cyber insider threats, from unsophisticated adversaries up to advanced, nation state sponsored actors. The DOE laboratories also presents an enormous insider attack surface of an ever-growing number of computers, networks, and devices all running a myriad of software. In addition, the DOE laboratories are connected to a variety of external organizations which multiplies the possibility of insider threat. During this SBIR project Cyber Equations developed a strategic combination of supervised and unsupervised machine learning techniques to control scale and classify whether entities, such as hosts, actors, or documents, are showing suspicious insider behavior. We conducted experiments using real cyber data to show that changes in observable behavior (e.g., changes in outbound traffic patterns) are indicative of attacks executing (e.g., insiders exfiltrating data, bots calling home, or an attack spreading). The features and methods we use, have been used in the anomaly detection research and development community for a significant amount of time and robust implementations are openly available. K-Means clustering, Decision Trees, and Support Vector Machine based classifiers are readily available through the R language and open source H20 machine learning framework. The Cyber Equations approach is novel in its use of these classifiers combined with Hadoop based parallel processing technology. The five stages of processing used by CyberInsider improve scalability and reduce the high number of false positives normally associated with network anomaly detection. In summary, the Cyber Equations has successfully accomplished all of its Phase I tasks and demonstrated the feasibility of the Insider Threat Monitoring System.

High-Performance Computing (HPC) has become a powerful tool to solve highly complex projects that could not otherwise be addressed. However, developing parallel programs to take advantage of its benefits, making a proper and effective use of the available computing resources, can be a difficult task for beginners or scientists of specific branches of science which have very little to do with computer programming. For this purpose, a transcompiler for automatic parallelization of sequential codes was developed to train beginners in parallel programming and achieve better performances in HPC centers. Current efforts are focused on expanding the transcompiler with an additional module for optimizing sequential and parallel code fragments with the objective of speeding up their execution times. To accomplish this, several techniques for writing efficient code have been evaluated with a view to integrate them into the transcompiler in the future. This manuscript discusses the importance of these techniques to decrease running times of HPC programs. In addition, a set of tests have been developed for analysing and measuring the improvements achieved when applying each of these techniques, all of them with a significant impact on the execution times.