The Experience in Designing and Building the High Performance Cluster Netuno

For the High Performance Linpack (HPL) benchmark at the coming Exascale and beyond, silent errors like bit flipping in memory are expected to become inevitable. However, since bit flipping errors are difficult to be detected and located, their impact to the numerical correctness of HPL has not been evaluated thoroughly and quantitatively, while the impact at Exascale is especially susceptible. In this paper, an initial quantitative analysis of the impact of bit flipping errors to the numerical correctness of HPL has been investigated. To validate the numerical correctness of computed solution using HPL, there is a residual check after the approximate solution obtained. This paper has shown that in the case of only one bit flipping to any element in the original data matrix, if the flipped position is not the leading position of exponent, the residual check in HPL will almost surely pass at the scale of Exa flops and beyond. Experiments on modified HPL in single precision at small scales have verified the theoretical results in double precision at Exascale. The results obtained in this paper can provide a better understanding to the impact of bit flipping errors to numerical correctness of scientific computing applications.

Typically, a High Performance Computing (HPC) cluster loosely couples multiple Symmetric Multi-Processor (SMP) platforms into a single processing complex. Each SMP uses shared memory for its processors to communicate, whereas communication across SMPs goes through the intra-cluster interconnect. By analyzing the communication pattern of processes, it is possible to arrive at a mapping of processes to processors to ensure optimal communication paths for critical traffic. This critical traffic refers to the communication pattern of the program, which can be characterized by either frequency or size (or both) of the messages. To find an ideal mapping, it is imperative to understand the communication characteristics of the SMP memory system, intra-cluster interconnection, and the Message Passing Interface (MPI) program running on a cluster.Our approach is to study the ideal mapping for two classes of interconnects: 1) standard, high-volume Ethernet interconnects, and 2) proprietary, low-latency high-bandwidth interconnects. In the first installment of our work presented in this paper, we have focused on the ideal mapping for the first class.We configured a 16-node dual-processor cluster, interconnected with Fast Ethernet, Gigabit Ethernet, Giganet, and Myrinet. We used High Performance Linpack (HPL) benchmark to demonstrate that re-mapping of processes to processors (or changing the order of processors used) can affect the overall performance. The mappings are based on the HPL program analysis obtained from running a MPI profiling tool. Our results suggest that the performance of HPL using Fast Ethernet as the interconnect can be improved from 10% to 50% depending on the process mapping and the problem size. Conversely, an ad hoc mapping can adversely affect the cluster performance.

After 15 years of exponential improvement in microprocessor clock rates, the physical principles allowing for Dennard scaling, which enabled performance improvements without a commensurate increase in power consumption, have all but ended. Until now, most HPC systems have not focused on power efficiency. However, as the cost of power reaches parity with capital costs, it is increasingly important to compare systems with metrics based on the sustained performance per watt. Therefore we need to establish practical methods to measure power consumption of such systems in- situ in order to support such metrics. Our study provides power measurements for various computational loads on the largest scale HPC systems ever involved in such an assessment. This study demonstrates clearly that, contrary to conventional wisdom, the power consumed while running the high performance Linpack (HPL) benchmark is very close to the power consumed by any subset of a typical compute-intensive scientific workload. Therefore, HPL, which in most cases cannot serve as a suitable workload for performance measurements, can be used for the purposes of power measurement. Furthermore, we show through measurements on a large scale system that the power consumed by smaller subsets of the system can be projected straightforwardly and accurately to estimate the power consumption of the full system. This allows a less invasive approach for determining the power consumption of large-scale systems.

High-performance Linpack (HPL) benchmark is used to evaluate the performance of super computers. It implements blocked, right-looking Gaussian elimination with row partial pivoting. Block cyclic distribution is used in the parallel variant.In this paper, we implement the HPL in X10. X10 is a high level high-performance programming language based on the sequential semantics of Java augmented with new concurrency constructs, namely, places, asyncs, finish, atomic and clock. X10 also provides a rich array language which includes region, distributions and distributed arrays. We use HPL as a case study to evaluate the global view paradigm of X10. A global view program uses globally distributed arrays and global array indices without any process-specific code. We compare the elegance of global view programs over the local view version. We also show that the performance of global view program is no worse than the local view version.

Summary form only given. The effect of Intel/spl reg/ hyper-threading (HT) technology on a system's performance varies according to the characteristics of the application running on the system and the configuration of the system. High performance computing (HPC) clusters introduce additional variables, such as the math libraries used in solving linear algebra equations that can affect the performance of scientific and engineering applications in particular. We study the effect of HT on MPI-based applications by varying the math library. We configure an Intel-based HPC cluster and used the High Performance Linpack (HPL) benchmark to study the performance characteristics without changing hardware or communication parameters, but by linking the application with different math libraries. What we have found is that, even though the application or hardware parameters remain the same, HT may help or hinder the overall performance of the cluster depending on the computational efficiency of the mathematical functions.

Aiming at realizing a high performance computing (HPC) cloud with vector supercomputers, this paper presents the world’s first prototype of wide-area vector meta computing infrastructure named a vector computing cloud by virtualizing remote vector computing resources as an HPC service over the Internet. The prototype system consists of two remote NEC SX-9 nodes connected through a long fat-pipe network (LFN), and each node is located at Tohoku University and Osaka University with a distance of 800 km. The vector computing cloud also provides a single sign on environment and jobs are automatically assigned to appropriate sites. Wide-area co-operation of distributed vector supercomputers is realized by adopting the NAREGI Grid Middleware, and a virtual machine for NEC SX vector supercomputer series, job scheduling algorithms, and an MPI operating environment are newly developed to enhance the job and resource management capabilities of the NAREGI Grid Middleware. In addition, to achieve fairness and efficient job scheduling on the vector computing cloud, this paper presents a history-based job scheduling mechanism for a queue system. Based on the estimation, the job scheduling mechanism automatically allocates the job to an appropriate site, which can execute the job earlier. The operation tests and experiment results indicate that the prototype system realizes single sign on for multiple vector resources and has enough potential for transparently operating jobs between the two SX-9 systems. This paper also evaluates and discusses the performance of the proposed job scheduling mechanism and MPI operation between both SX-9 systems using the High Performance Linpack (HPL) benchmark.

Dense LU factorization has a high ratio of computation to communication and, as evidenced by the High Performance Linpack (HPL) benchmark, this property makes it scale well on most parallel machines. Nevertheless, the standard algorithm for this problem has non-trivial dependence patterns which limit parallelism, and local computations require large matrices in order to achieve good single processor performance. We present an alternative programming model for this type of problem, which combines UPC's global address space with lightweight multithreading. We introduce the concept of memory-constrained lookahead where the amount of concurrency managed by each processor is controlled by the amount of memory available. We implement novel techniques for steering the computation to optimize for high performance and demonstrate the scalability and portability of UPC with Teraflop level performance on some machines, comparing favourably to other state-of-the-art MPI codes.

The probability that a failure will occur before the end of the computation increases as the number of processors used in a high performance computing application increases. For long running applications using a large number of processors, it is essential that fault tolerance be used to prevent a total loss of all finished computations after a failure. While checkpointing has been very useful to tolerate failures for a long time, it often introduces a considerable overhead especially when applications modify a large amount of memory between checkpoints and the number of processors is large. In this paper, we propose an algorithm-based recovery scheme for the High Performance Linpack benchmark (which modifies a large amount of memory in each iteration) to tolerate fail-stop failures without checkpointing. It was proved by Huang and Abraham that a checksum added to a matrix will be maintained after the matrix is factored. We demonstrate that, for the right-looking LU factorization algorithm, the checksum is maintained at each step of the computation. Based on this checksum relationship maintained at each step in the middle of the computation, we demonstrate that fail-stop process failures in High Performance Linpack can be tolerated without checkpointing. Because no periodical checkpoint is necessary during computation and no roll-back is necessary during recovery, the proposed recovery scheme is highly scalable and has a good potential to scale to extreme scale computing and beyond. Experimental results on the supercomputer Jaguar demonstrate that the fault tolerance overhead introduced by the proposed recovery scheme is negligible.

Widely used benchmarks, such as High Performance Linpack (HPL), do not always provide direct insights are notoriously poor indicators of into the actual application performance of systems. When real applications are used, and there have been are criticisms indicating that the performance of simplified benchmarks such as HPL no longer strongly correlate to real application performance. In contrast, performance evaluations based on real or mini applications may give a direct estimation into application performance. The Sustained System Performance (SSP) metric, which is used to evaluate systems based on the performance at scale of various applications, has been successfully adopted to procure systems at the National Energy Research Scientific Computing Center (NERSC), the National Center for Supercomputing Applications (NCSA) and other facilities. However, significant effort is required to tune and optimize several mini applications for each of systems. In this paper, we propose a new performance metric — the Simplified Sustained System Performance (SSSP) metric — based on a suite of simple benchmarks, which enables performance projection that correlates with full applications, but use onto a suite of mini applications. While the SSP metric is calculated over a set of applications, the SSSP metric applies its methodology to a set of benchmarks. Preliminary weighting factors for benchmarks are introduced to approximate the original SSP metric more accurately by the SSSP metric. To define the weighting factors, we perform a simple learning algorithm. Our preliminary experiments show that even though our metric is still easy to measure because it is based on a combination of simple benchmarks, it can provide projections of the performance of applications.

The High Performance Linpack (HPL), or Top 500, benchmark [1] is the most widely recognized and discussed metric for ranking high performance computing systems. However, HPL is increasingly unreliable as a true measure of system performance for a growing collection of important science and engineering applications. In this paper we describe a new high performance conjugate gradient (HPCG) benchmark. HPCG is composed of computations and data access patterns more commonly found in applications. Using HPCG we strive for a better correlation to real scientific application performance and expect to drive computer system design and implementation in directions that will better impact performance improvement.

INTRODUCTION With the breakdown of Dennard’s scaling rule, chip power for a given area size is not constant as transistors shrink from generation to generation [1]. In particular, the power wall problem worsens with the continual improvement in performance of high-performance computers. The power challenge has been identified as the biggest research challenge [2]. Practical exascale systems of the future should meet a power constraint of 20 MW [2]. The Tianhe-2 system, the fastest supercomputer on the TOP500 list of June 2015, consumes 17.8 MW in running the high-performance Linpack (HPL) benchmark, achieving a performance of 33.86 Pflops. Compared with the Tianhe-2, an exascale system must obtain 30 times the performance improvement with 1.12 times the increase in power consumption. Clearly, in creating such exascale systems, satisfying such power requirements using evolutionary approaches is near impossible. Realizing this challenging power target depends on improvements in many technologies, especially, the viable and revolutionary technologies including new devices, innovative architectures and systems, and energy-aware intelligent technologies; see Figure 1.

Comparable to time performance models, it is now possible to estimate performance based upon energy consumption for HPC systems. The predictive ability of the analytical modeling is an interesting feature that motivates us to approach this methodology for the case of energy consumption. In this paper, we present an analytical model for predicting the energy consumption for the High Performance Linpack (HPL). The derived model can be used to know in advance the energy consumed by the HPL over a target architecture, and can be integrated into the schedulers of operating systems or queue managers. We established an experimental setup using a standard metered PDU that allowed us to measure the energy consumption for the HPL benchmark on our cluster. With the monitoring system in place, we can obtain the architectural and algorithmic parameters associated for both performance and energy analytical models. Also this has made possible watts and gflops-per-watt prediction when we execute Linpack executions with concrete algorithm parameters in our cluster.

This paper focuses on theoretical analysis, computational test and optimization of High Performance Computing Cluster (HPCC). Also known as Data Analytics Supercomputer (DAS), HPCC is built on Intel Many Integrated Core (MIC) for High Performance Linpack (HPL) test. Initially, a platform is configured by 5 nodes with an Intel® Xeon Phi <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">™</sup> Coprocessor 31S1P per node to analyze power consumption as well as parallel level of Intel MIC accelerator. Compiling with Message Passing Interface (MPI) library and Math Kernel Library (MKL), the "Make" file is modified and debugged by adjusting parameters in hpccinf.txt. According to multiple Infiniband nodes evaluation, the libhpl library of Intel is employed and value of NB is set to be 960 for a single node with one MIC while debugging. Moreover, this paper optimized the algorithm on Double Precision General Matrix Multiplication (DGEMM) test and PTRANS to acquire efficient, precise and truncated working time duration. From HPL and Gridding Program experimental results, it's clear that the theoretical analysis and experiments were performed successfully.

In recent years, many commercial Massively Parallel Processor (MPP) systems have been available available to the computing community. These systems provide very high processing power (up to hundreds of GFLOPs), and can scale efficiently with the number of processors. However, many scientific and commercial applications that run on these multiprocessors may not experience significant benefit in terms of speedup and are bottlenecked by their I/O requirements. Although these multiprocessors may be configured with sufficient I/O hardware, the file system software often fails to provide the available I/O bandwidth to the application, and causes severe performance performance degradation for I/O intensive applications. A highly efficient parallel file system has been implemented on Intel's Teraflops (TFLOPS) machine and provides a sustained I/O bandwidth of 1 GB/sec. This file system provides almost 95% of the available raw hardware I/O bandwidth and the I/O bandwidth scales proportional to the available I/O nodes. Intel's TFLOPS machine is the first Accelerated Strategic Computing Initiative (ASCI) machine that DOE has acquired. This computer is 10 times more powerful than the fastest machine today, and will be used primarily to simulate nuclear testing and to ensure the safety and effectiveness of the nation's nuclear weapons stockpile. This machine contains over 9000 Intel's Pentium Pro processors, and will provide a peak CPU performance of 1.8 teraflops. This papers presents the I/O design and architecture of Intel's TFLOPS supercomputer, describes the Cougar OS I/O and its interface with the Intel's Parallel File System.

Current processor trend is to couple a commodity processor with a GPU, a co-processor, or an accelerator. To unleash the full computational power of such heterogeneous systems is a daunting task: programmers often resort to heterogeneous scheduling runtime frameworks that use device specific library routines. However, highly-tuned libraries do not compose very well across heterogeneous architectures. That is, important performance-oriented optimizations such as data locality and reuse “across” library calls is not fully exploited. In this paper, we present a framework, called Mozart, to extend existing library frameworks to efficiently compose a sequence of library calls for heterogeneous execution. Mozart consists of two components: library description (LD) and library composition runtime. We advocate library writers to wrap existing libraries using LD in order to provide their performance parameters on heterogeneous cores, no programmer intervention is necessary. Our runtime performs composition of libraries via task-fission, load balances among heterogeneous cores using information from LD, and automatically adapts to runtime behavior of an application. We evaluate Mozart on a Xeon + 2 Xeon Phi system using the High Performance Linpack benchmark which is the most popular benchmark to rank supercomputers in TOP500 and show GFLOPS improvement of 31.7 % over MKL with Automatic Offload and 6.7 % over hand-optimized ninja code.