Compiler and Runtime Approaches to Enable Large-Scale Irregular Programs. Final report, July 2013 - July 2019

An architecture of high performance computer,called Hyper Parallel Processing(HPP),is proposed to satisfy the requirements of both High Performance Computing,and Utility Computing which will be the application model of data centers.HPP combines the benefits of the scalability of MPP,the communication efficiency of DSM,as well as the commodity of cluster.Comparisons of current main-stream high performance computer architectures show that none of them can satisfy both HPC and utility computing.The main features of HPP are Global Address Space(GAS) and Hyper Node with single Operating System image.HPP supports the distributed global address space including both memory and I/O,but without hardware cache coherence.A Hyper Node consists of a set of application specific CPUs and a(or more) OS specific CPU.The OS CPU maintains the single system image,while the application CPUs run only lightweight run-time software.Besides the GAS interconnect network for applications,a standard SAN connects all OS CPUs and I/O devices providing system management and storage service.HPP is able to provide many opportunities of innovative research in High Performance and Utility Computing areas,including communication,synchronization,programming model,node operating system,utility computing,fault isolation,CPU and system etc.According to HPP architecture,a prototype system of Dawning5000 HPC is implemented and the feasibility of HPP is proved.

Graph data mining has been a crucial as well as inevitable area of research. Large amounts of graph data are produced in many areas, such as Bioinformatics, Cheminformatics, Social Networks, etc. Scalable graph data mining methods are getting increasingly popular and necessary due to increased graph complexities. Frequent subgraph mining is one such area where the task is to find overly recurring patterns/subgraphs. To tackle this problem, many main memory-based methods were proposed, which proved to be inefficient as the data size grew exponentially over time. In the past few years, several research groups have attempted to handle the Frequent Subgraph Mining (FSM) problem in multiple ways. Many authors have tried to achieve better performance using Graphic Processing Units (GPUs) which has multi-fold improvement over in-memory while dealing with large datasets. Later, Google's MapReduce model with the Hadoop framework proved to be a major breakthrough in high performance large batch processing. Although MapReduce came with many benefits, its disk I/O and noniterative style model could not help much for FSM domain since subgraph mining process is an iterative approach. In recent years, Spark has emerged to be the De Facto industry standard with its distributed in-memory computing capability. This is a right fit solution for iterative style of programming as well. In this survey, we cover how high-performance computing has helped in improving the performance tremendously in the transactional directed and undirected aspect of graphs and performance comparisons of various FSM techniques are done based on experimental results.

Existing generic benchmarks for accelerators (e.g. Parboil and Rodinia) have focused on high performance computing (HPC) applications which have limited control flows and data irregularity. Previous available graph analytics benchmark suites include straightforward implemented workloads which do not employ up-to-date optimization techniques and thus have quite different behaviors from real-world applications. This paper first briefly presents and characterizes the Graph Analytics Repository for Designing Next-generation Accelerators (GARDENIA) 1, which is a benchmark suite for studies of irregular algorithms on various massively parallel accelerators. It includes emerging irregular big-data and machine learning applications, in which mimic massively multithreaded programs deployed on not only datacenters but also hand-on devices. Then we characterize Nvidia GPU with GARDENIA, covering a wide spectrum of metrics such as parallelization, cache locality, off-chip traffic and irregularity. Based on the characterization on Nvidia GPU, we unveil the performance bottlenecks of the current mainstream accelerator and give architectural insights for building high performance and energy-efficient domain-specific accelerators for graph applications.

For decades, HPC has established itself as an essential tool for discoveries, innovations and new insights in science, research and development, engineering and business across a wide range of application areas in academia and industry. Today High-Performance Computing is also well recognized to be of strategic and economic value – HPC matters and is transforming industries. This article will discuss new emerging technologies that are being developed for all areas of HPC: compute/processing, memory and storage, interconnect fabric, I/O and software to address the ongoing challenges in HPC such as balanced architecture, energy efficient high-performance, density, reliability, sustainability, and last but not least ease-of-use. Of specific interest are the challenges and opportunities for the next frontier in HPC envisioned around the 2020 timeframe: ExaFlops computing. We will also outline the new and emerging area of High Performance Data Analytics, Big Data Analytics using HPC, and discuss the emerging new delivery mechanism for HPC - HPC in the Cloud.

Optimizing applications simultaneously for energy and performance is a complex problem. High performance, parallel, irregular applications are notoriously hard to optimize due to their data-dependent memory accesses, lack of structured locality and complex data structures and code patterns. Irregular kernels are growing in importance in applications such as machine learning, graph analytics and combinatorial scientific computing. Performance- and energy-efficient implementation of these kernels on modern, energy efficient, many-core platforms is therefore an important and challenging problem. We present results from optimizing two irregular applications -- the Louvain method for community detection (Grappolo), and high-performance conjugate gradient (HPCCG) -- on the Tilera many-core system. We have significantly extended MIT's OpenTuner auto-tuning framework to conduct a detailed study of platform-independent and platform-specific optimizations to improve performance as well as reduce total energy consumption. We explore the optimization design space along three dimensions: memory layout schemes, compiler-based code transformations, and optimization of parallel loop schedules. Using auto-tuning, we demonstrate whole-node energy savings of up to 41% relative to a baseline instantiation, and up to 31% relative to manually optimized variants.

The theoretical concept of technological performance has emerged only recently with the publication of Jon McKenzie's Perform or Else: From Discipline to Performance in 2001. McKenzie develops a general theory of performance based around the development of three performance paradigms: cultural performance, organizational performance (or performance management), and technological performance. In his examination of the techno-performance paradigm, he focuses primarily on late twentieth and early twenty-first century 'high performance' technologies such as computers, guided missiles and space shuttles. While he acknowledges that the concept of performance does not apply only to technologies in this period, his analysis implicitly suggests that high performance technologies are a unique invention of the modern age. This essay confronts McKenzie's restriction of techno-performance to the post-WWII period by demonstrating how technologies performed and were seen to perform in the late nineteenth century through a techno-performance reading of Bram Stoker's Dracula (1897). The late Victorian period saw startling innovations in information and communication technology (such as the overseas telegraph, the typewriter, and the gramophone), which were marketed as high performance technologies, though not in those words. To contextualise my reading of Dracula , I examine contemporary Victorian advertisements for communication technologies to demonstrate how such technologies were viewed as high performance products by Victorian advertisers and consumers. Technology in Dracula has usually been read as a metaphor. I employ McKenzie's concept of techno-performance to examine the performative functions of technology in Dracula that have not yet been explored by Victorianist scholarship. McKenzie notes two challenges posed by techno-performance: first, the challenge posed by a developer to his/her technological product, to perform or be classified as obsolete; and, second, the challenge posed by technology to its user to perform or be regarded as outmoded. I argue that Stoker's Dracula takes up both of these challenges. Emergent technologies sometimes perform in unexpected and potentially disruptive ways, much like the space shuttle Challenger cited by McKenzie; at the same time, such technologies oblige their users to perform in unexpected and disruptive ways. This essay examines emergent technologies in Dracula to highlight the relationship between individual and technological performance in the late nineteenth century.

This document presents key principles and practical considerations for optimizing CPU-bound code in the context of high-performance scientific computing. While classical algorithmic complexity provides a hardware-independent framework for evaluating scalability, real-world performance is largely determined by low-level factors such as memory access patterns, cache behavior, data locality, and compiler optimizations. The discussion highlights how modern architectures rely on hierarchical memory systems, making data movement often more critical than computation. Particular attention is given to issues such as aliasing, false sharing, and cache conflicts, which can significantly impact performance despite unchanged algorithms. The role of the compiler is emphasized as central in exploiting hardware capabilities, provided that code is well-structured and free from undefined behavior. Finally, the document underlines the limitations of general optimization rules and stresses the importance of empirical testing, as performance depends strongly on hardware characteristics and data layout.

Graph data management and mining in HPC environments has been a widely discussed issue in recent times. In this talk I will describe the use of Partitioned Global Address Space languages for graph data mining and management. I will first discuss the rationale behind X10 based graph libraries and graph database benchmarks using ScaleGraph and XGDBench as examples. Next, I will take Acacia which is completely developed with X10 language as an example system and describe our experience with implementing such high performance system with X10. In this talk I will describe how RDF processing and Streaming extensions have been implemented in Acacia. Finally, I will highlight some of the notable areas which need further attention in future.

Modern high-performance computing (HPC) storage systems use heterogeneous storage technologies organized in tiers to find a compromise between capacity, performance, and cost. In these systems, prefetching is a common technique used to move the right data at the right moment from a slow to a fast tier to improve overall performance while using the costly high-performance tier only when needed. Effective prefetching requires precise knowledge of the application I/O patterns. This knowledge can be extracted through the source code, I/O tracing tools or I/O functions call stacks. State-of-the-art solutions based on the latter approach mainly focus on applications with regular I/O profiles to avoid scalability issues due to the grammar-based techniques used. In this paper, we present an approach based on I/O call stacks that models POSIX and STDIO I/O patterns for both regular and irregular applications, thanks to the use of directed graphs. We present different models usable for prefetching. Our models were used to predict the next I/O call stack on five real HPC applications with a prediction accuracy of up to 98%. Compared to the state-of-the-art Omnisc’IO, they incurred up to 120x lower model overhead (334 ns vs. 45 μ s on LAMMPS) and had a model size 10x to 15x smaller (463 B vs. 7 kB on LQCD). • A graph-based prediction model for HPC I/Os for predictive prefetch. • The overhead in terms of model size and time is much lower than in previous work. • Three new modeling granularities are introduced.

At first thought, one may not realize the need for a special issue of Parallel and Distributed Computing Practices focused on Cluster Computing when our community has already invested years of research and spent a wealth of resources on traditional Big Iron supercomputers, for instance, the SGI/Cray Origin and IBM SP. Academia, industry, and national laboratories are still using, and for the foreseeable future, will continue to use, supercomputers to solve both grand-challenge scale and high-throughput applications. Then why the interest and new popularity of research into clusters for high-performance parallel and distributed applications? Clusters incorporate the tremendous successes that are attributable to our parallel and distributed processing community. Over the next decade, clusters will span the entire range of high-performance computing platforms. Instead of being restricted solely to supercomputers for achieving high-performance computing, the spectrum of available platforms will spread out to Beowulf-class clusters, to superclusters, and on to supercomputers. Moving along this continuum represents increasing computational resources, which in turn allows the capability for solving larger classes of applications or instances of problems. Compute-bound applications on workstations often may take advantage of the extra horsepower provided by symmetric multiprocessors. Applications with even larger computational requirements and a high-degree of concurrency may use Beowulf clusters, that is, clusters that are home-built from mass-market, commercial off-the-shelf workstations and networks, for instance, PC's interconnected by Fast Ethernet. A supercluster may contain commodity SMP nodes interconnected with scalable, low-latency, high-bandwidth networks, and is integrated by a system vendor rather than the computational scientist. Applications with greater complexity, for example, grand challenges from the sciences, demand scalable systems with high-performance, interconnects, perhaps coupled with terascale I/O subsystems, and thus, are appropriate for these superclusters. Finally, supercomputers, the largest of the clusters, may provide the ultimate platform for the highest-fidelity models and simulations and data-intensive applications. At each step up this ladder of cluster architectures, the system cost increases approximately by an order of magnitude, but having the high-performance capability to solve the problem of interest may require an entry point at any of these levels, the Beowulf, or supercluster, or even supercomputer platform. Cluster computing with Beowulf- or superclusters, thus, may be one or two orders of magnitude less expensive than a supercomputer, and provide a cost-effective solution and capability that was not available on a workstation. But this cost comparison measures just the basic system price. What about the applications? As recently as a few years ago, the lack of a canonical high-performance architectural paradigm meant that migrating to a new computer system, even from the same vendor, typically required redesigning applications to efficiently use the high-performance system. Thus, the application must be redesigned with parallel algorithms and libraries that are optimized for each high-performance system. This process of porting, redesigning, optimizing, debugging, and analyzing, the application may in fact be prohibitively expensive, and certainly frustrating when the process is completed just in time to greet the next generation system! That was then, this is now. With the successful standardization of system-level interfaces, many of these issues no longer hinder our expedited use of these new, high-performance clusters. The message-passing interface (MPI) is a clear example of a standard that has contributed significantly to the success of cluster computing. Supercomputer vendors working together with the high-performance computing community embarked on a process to standardize the user-level interface and behavior for passing messages between concurrent tasks. Now, using ordinary sequential compilers and vendor-supplied or freely-available implementations of this message-passing interface, applications can be implemented efficiently on one supercomputer and ported with little or no effort to other parallel machines. The message-passing paradigm, thus, provides a reasonably good canonical high-performance model and methodology for implementing high-performance applications. An algorithm that exhibits efficient execution and scaling on one machine then is likely to perform well on another with little or no additional programming effort. Using this cluster computing paradigm, algorithms typically are optimized by minimizing the number of remote memory accesses (reducing the amount of messages passed), and carefully organizing the data placement to increase memory locality. Thus, high-performance algorithms and applications that have been optimized for supercomputers, using the same cluster paradigm, may be appropriate for Beowulf and superclusters. Cluster computing, realizing the importance of a standard high-performance programming environment that applies equally well to both ends of the architectural spectrum, highly leverages these and other successes from parallel and distributed processing research and development. This is in an exciting, new era for high-performance computing where the canonical architectural paradigm stretches from home-built Beowulf clusters, to superclusters, and on to supercomputers. But cluster computing faces some challenges ahead of it. Continued research is required for designing 1) scalable, high-performance algorithms, applications, and methodologies, that can tolerate the communication overheads, and adapt to hybrid, SMP, and heterogeneous, nodes; 2) high-performance computer architectures, scalable interconnection networks, and I/O subsystems; and 3) scalable systems software, compilers, and monitoring tools. Cluster computing has come of age, and it is high time for the parallel and distributed processing community to lay the groundwork for our own legacy of success stories. David A. Bader University of New Mexico

R is a robust open-source programming language mainly used for statistical computing . Many areas of statistical research are experiencing rapid growth in the size of data sets. Methodological advances drive increased use of simulations. A common approach is to use parallel/concurrent computing. This paper presents an overview of techniques for parallel computing with R on ACI (a PSU Infrastructure) and benchmark it with C/C++. We review the scalabilty concern of R, and look at the simplicity of using R as a primary language in Coding for HPC. We will look at the various R packages for HPC like Rmpi, Rcpp, snow and snowfall. We utilize a series of algorithms to benchmark and will illustrate each benchmark with a representative graph for ease of understanding. The paper concludes with a better understanding of which language to use when in high performance computing .

The transition towards net-zero emissions is inevitable for humanity’s future. Of all the sectors, electrical energy systems emit the most emissions. This urgently requires the witnessed accelerating technological landscape to transition towards an emission-free smart grid. It involves massive integration of intermittent wind and solar-powered resources into future power grids. Additionally, new paradigms such as large-scale integration of distributed resources into the grid, proliferation of Internet of Things (IoT) technologies, and electrification of different sectors are envisioned as essential enablers for a net-zero future. However, these changes will lead to unprecedented size, complexity and data of the planning and operation problems of future grids. It is thus important to discuss and consider High Performance Computing (HPC), parallel computing, and cloud computing prospects in any future electrical energy studies. This article recounts the dawn of parallel computation in power system studies, providing a thorough history and paradigm background for the reader, leading to the most impactful recent contributions. The reviews are split into Central Processing Unit (CPU) based, Graphical Processing Unit (GPU) based, and Cloud-based studies and smart grid applications. The state-of-the-art is also discussed, highlighting the issue of standardization and the future of the field. The reviewed papers are predominantly focused on classical imperishable electrical system problems. This indicates the need for further research on parallel and HPC approaches applied to future smarter grid challenges, particularly to the integration of renewable energy into the smart grid.

The main focus of this article is the design of embedded signal processing (ESP) application software. We identify the characteristics of such applications in terms of their computational requirements, data layouts, and latency and throughput constraints. We describe an ESP application, an adaptive sonar beamformer. Then, we briefly survey the state-of-the-art in high performance computing (HPC) technology and address the advantages and challenges in using HPC technology for implementing ESP applications. To describe the software design issues in this context, we define a task model to capture the features of ESP applications. This model specifies the independent activities in each processing stage. We also identify various optimization problems in parallelizing ESP applications. We address the key issues in developing scalable and portable algorithms for ESP applications. We focus on the algorithmic issues in exploiting coarse-grain parallelism. These issues include data layout design and task mapping. We show a task mapping methodology for application software development based on our execution model (Lee et al., 1998). This uses a novel stage partitioning technique to exploit the independent activities in a processing stage. We use our methodology to maximize the throughput of an ESP application for a given platform size. The resulting application software using this methodology is called a software task pipeline. An adaptive sonar beamformer has been implemented using this design methodology.

Designing irregular parallel algorithms with mutual exclusion and lock-free protocols

From its founding in 1952 until today, Lawrence Livermore National Laboratory (LLNL) has made significant strategic investments to develop high performance computing (HPC) and its application to national security and basic science. Now, 60 years later, the Computation Directorate and its myriad resources and capabilities have become a key enabler for LLNL programs and an integral part of the effort to support our nation's nuclear deterrent and, more broadly, national security. In addition, the technological innovation HPC makes possible is seen as vital to the nation's economic vitality. LLNL, along with other national laboratories, is working to make supercomputing capabilities and expertise available to industry to boost the nation's global competitiveness. LLNL is on the brink of an exciting milestone with the 2012 deployment of Sequoia, the National Nuclear Security Administration's (NNSA's) 20-petaFLOP/s resource that will apply uncertainty quantification to weapons science. Sequoia will bring LLNL's total computing power to more than 23 petaFLOP/s-all brought to bear on basic science and national security needs. The computing systems at LLNL provide game-changing capabilities. Sequoia and other next-generation platforms will enable predictive simulation in the coming decade and leverage industry trends, such as massively parallel and multicore processors, to run petascale applications. Efficient petascale computing necessitates refining accuracy in materials property data, improving models for known physical processes, identifying and then modeling for missing physics, quantifying uncertainty, and enhancing the performance of complex models and algorithms in macroscale simulation codes. Nearly 15 years ago, NNSA's Accelerated Strategic Computing Initiative (ASCI), now called the Advanced Simulation and Computing (ASC) Program, was the critical element needed to shift from test-based confidence to science-based confidence. Specifically, ASCI/ASC accelerated the development of simulation capabilities necessary to ensure confidence in the nuclear stockpile-far exceeding what might have been achieved in the absence of a focused initiative. While stockpile stewardship research pushed LLNL scientists to develop new computer codes, better simulation methods, and improved visualization technologies, this work also stimulated the exploration of HPC applications beyond the standard sponsor base. As LLNL advances to a petascale platform and pursues exascale computing (1,000 times faster than Sequoia), ASC will be paramount to achieving predictive simulation and uncertainty quantification. Predictive simulation and quantifying the uncertainty of numerical predictions where little-to-no data exists demands exascale computing and represents an expanding area of scientific research important not only to nuclear weapons, but to nuclear attribution, nuclear reactor design, and understanding global climate issues, among other fields. Aside from these lofty goals and challenges, computing at LLNL is anything but 'business as usual.' International competition in supercomputing is nothing new, but the HPC community is now operating in an expanded, more aggressive climate of global competitiveness. More countries understand how science and technology research and development are inextricably linked to economic prosperity, and they are aggressively pursuing ways to integrate HPC technologies into their native industrial and consumer products. In the interest of the nation's economic security and the science and technology that underpins it, LLNL is expanding its portfolio and forging new collaborations. We must ensure that HPC remains an asymmetric engine of innovation for the Laboratory and for the U.S. and, in doing so, protect our research and development dynamism and the prosperity it makes possible. One untapped area of opportunity LLNL is pursuing is to help U.S. industry understand how supercomputing can benefit their business. Industrial investment in HPC applications has historically been limited by the prohibitive cost of entry, the inaccessibility of software to run the powerful systems, and the years it takes to grow the expertise to develop codes and run them in an optimal way. LLNL is helping industry better compete in the global market place by providing access to some of the world's most powerful computing systems, the tools to run them, and the experts who are adept at using them. Our scientists are collaborating side by side with industrial partners to develop solutions to some of industry's toughest problems. The goal of the Livermore Valley Open Campus High Performance Computing Innovation Center is to allow American industry the opportunity to harness the power of supercomputing by leveraging the scientific and computational expertise at LLNL in order to gain a competitive advantage in the global economy.