Scalable communication protocols for dynamic sparse data exchange

Many large-scale parallel programs follow a bulk synchronous parallel (BSP) structure with distinct computation and communication phases. Although the communication phase in such programs may involve all (or large numbers) of the participating processes, the actual communication operations are usually sparse in nature. As a result, communication phases are typically expressed explicitly using point-to-point communication operations or collective operations. We define the dynamic sparse data-exchange (DSDE) problem and derive bounds in the well known LogGP model. While current approaches work well with static applications, they run into limitations as modern applications grow in scale, and as the problems that are being solved become increasingly irregular and dynamic.To enable the compact and efficient expression of the communication phase, we develop suitable sparse communication protocols for irregular applications at large scale. We discuss different irregular applications and show the sparsity in the communication for real-world input data. We discuss the time and memory complexity of commonly used protocols for the DSDE problem and develop NBX--a novel fast algorithm with constant memory overhead for solving it. Algorithm NBX improves the runtime of a sparse data-exchange among 8,192 processors on BlueGene/P by a factor of 5.6. In an application study, we show improvements of up to a factor of 28.9 for a parallel breadth first search on 8,192 BlueGene/P processors.

HPC systems are widely used for accelerating calculation-intensive irregular applications, e.g., molecular dynamics (MD) simulations, astrophysics applications, and irregular grid applications. As the scalability and complexity of current HPC systems keeps growing, it is difficult to parallelize these applications in an efficient fashion due to irregular communication patterns, load imbalance issues, dynamic characteristics, and many more. This paper presents a fine granular programming scheme, on which programmers are able to implement parallel scientific applications in a fine granular and SPMD (single program multiple data) fashion. Different from current programming models starting from the global data structure, this programming scheme provides a high-level and object-oriented programming interface that supports writing applications by focusing on the finest granular elements and their interactions. Its implementation framework takes care of the implementation details e.g., the data partition, automatic EP aggregation, memory management, and data communication. The experimental results on SuperMUC show that the OOP implementations of multi-body and irregular applications have little overhead compared to the manual implementations using C++ with OpenMP or MPI. However, it improves the programming productivity in terms of the source code size, the coding method, and the implementation difficulty.

Testing parallel applications on a large number of processors is often impractical. Not only does it require access to scarce compute resources, but tracking down defects with the available debugging tools can often be very time consuming. Highly parallel codes should be testable on one processor at a time, so that a developer’s workstation is sufficient for executing and debugging test cases on millions of processes. Thanks to its supersteps, Bulk Synchronous Parallel programs are well suited for this kind of testing. This paper presents a mocking library for BSPlib which enables testing of fast and complex parallel algorithms at scale.

While regular algorithms, characterized by operations on dense matrices and arrays, have long been the mainstay of scientific, high-performance computing, irregular algorithms, which feature unpredictable accesses to pointer-based data structures, are becoming increasingly common in high performance computing, arising in graph analysis, data mining and visualization, among other domains. Unfortunately, the defining characteristics of irregular applications, their dynamic, unpredictable, data-dependent access patterns and data layouts, make achieving high performance on large scale systems difficult. Scaling applications to peta- and exa-scale requires carefully controlling communication and data movement and placement, an inherently difficult task when access patterns and data layouts are unpredictable! Most irregular applications that attain high performance must be painstakingly hand-written and hand-tuned, with few common principles or paradigms uniting various implementations and easing future development. Despite the increasing importance of irregular applications, there is little programmer knowledge, and even less compiler ability, devoted to optimizing them. This project aims to solve these problems. By allowing programmers to write irregular applications in high level forms, with at most a few annotations highlighting key structural properties, programmers can focus on developing their algorithms and methods. The compiler and run-time system can take on the tedious task of optimizing the application for execution at large scales, and can automatically provide efficient implementations. This will provide portability and ease maintenance for existing irregular applications, but, more importantly, open up whole new domains of computational science to large-scale, high-performance simulation codes.

In this paper, we present the Data Movement and Control Substrate (DMCS), a library which implements low‐latency one‐sided communication primitives for use in parallel adaptive and irregular applications. DMCS is built on top of low‐level, vendor‐specific communication subsystems such as LAPI (Low‐level Application Programme Interface) for IBM SP machines, as well as on widely available message‐passing libraries like MPI for clusters of workstations and PCs. DMCS adds a small overhead to the communication operations provided by the lower communication system. In return, DMCS provides a flexible and easy to understand application program interface for one‐sided communication operations. Furthermore, DMCS is designed so that it can be easily ported and maintained by non‐experts. Copyright © 2002 John Wiley & Sons, Ltd.

In order to achieve exascale performance it is important to detect potential bottlenecks and identify strategies to overcome them. For this, both applications and system software must be analysed and potentially improved. The EU FP7 project Collaborative Research into Exascale Systemware, Tools & Applications (CRESTA) chose the approach to co-design advanced simulation applications and system software as well as development tools. In this paper, we present the results of a co-design activity focused on the simulation code NEK5000 that aims at performance improvements of collective communication operations. We have analysed the algorithms that form the core of NEK5000’s communication module in order to assess its viability on recent computer architectures before starting to improve its performance. Our results show that the crystal router algorithm performs well in sparse, irregular collective operations for medium and large processor number but improvements for even larger system sizes of the future will be needed. We sketch the needed improvements, which will make the communication algorithms also beneficial for other applications that need to implement latency-dominated communication schemes with short messages. The latency-optimised communication operations will also become used in a runtime-system providing dynamic load balancing, under development within CRESTA.

Analysis of massive graphs has emerged as an important area for massively parallel computation. In this paper, it is shown how the Fresh Breeze trees-of-chunks memory model may be used to perform breadth-first search of large undirected graphs. Overall, the computation can be expressed as a data flow process wherein a set of vertices to be searched is partitioned into a set of sub-domains and processed independently by many concurrent tasks.The main contributions of the paper are listed below.• We present the first case study demonstrating the power of the Fresh Breeze program execution model (PXM) in the exploitation of fine-grain parallelism found in irregular applications such as graph algorithms.• We present a novel parallel breadth-first search algorithm which is fully determinate.• We describe a unique sparse vector representation that represents the set of adjacencies for each vertex.• We provide an experimental study and analysis of our implementation. An estimate is also made of the performance that might be achieved with a massively parallel system built according to Fresh Breeze principles.

For large scale computational grids, where the resources are distributed over areas spanning thousands of miles, achieving efficiency of collective communication operations such as broadcast becomes of paramount importance. We propose a broadcast algorithm constructed in terms of point-to-point communication operations that occur according to a topology determined using a generalization of the single source shortest path algorithm such that the point-to-point operations are ordered according to a heuristic. We show that the proposed approach is competitive with, and in some cases exceeds, the performance of the broadcast operation implemented in MPICH-G2, the most used grid-enabled implementation of MPI.

Modern high performance computing systems are being increasingly deployed in a hierarchical fashion with multi-core computing platforms forming the base of the hierarchy. These systems are usually comprised of multiple racks, with each rack consisting of a finite number of chassis, and each chassis having multiple compute nodes or blades, based on multi-core architectures. The networks are also hierarchical with multiple levels of switches. Message exchange operations between processes that belong to different racks involve multiple hops across different switches and this directly affects the performance of collective operations. In this paper, we take on the challenges involved in detecting the topology of large scale InfiniBand clusters and leveraging this knowledge to design efficient topology-aware algorithms for collective operations. We also propose a communication model to analyze the communication costs involved in collective operations on large scale supercomputing systems. We have analyzed the performance characteristics of two collectives, MPI_Gather and MPI_Scatter, on such systems and we have proposed topology-aware algorithms for these operations. Our experimental results have shown that the proposed algorithms can improve the performance of these collective operations by almost 54% at the micro-benchmark level.

This paper describes a fast associative search algorithm based on parallel searching by pattern-splitting. The text is read as a sequence of substrings and searching is parallel within each substring. Substring length can be arbitrarily chosen and this division is independent of the logical partitioning of the data, such as tuple and domain. This provides a flexible storage structure across tracks of the secondary storage. The algorithm developed is used in the design of a hardware associative search. It works directly on the secondary storage and is a basis for database machine design. The design is cellular in structure and can be implemented by using LSI technology.

A comparative study of MPI implementations on a cluster of SMP workstations

Parallel computing utilizes heterogeneous parallel computing environments in which multiple processors of various kinds are connected by communication links. Efficient collective communication operations are needed to realize efficient parallel computing in parallel computing environments. Although methods for realizing efficient collective communication operations have previously been proposed for homogeneous parallel computing environments, these methods applied as they are to heterogeneous parallel computing environments cannot realize efficient collective communication operations. In this paper, we consider design of an efficient broadcast, which is one of the most important collective communication operations. We show that we can extend the known sufficient condition for Fastest‐Node‐First (FNF) to output an optimal schedule, where FNF is an efficient heuristic algorithm for the broadcast. We show that we can reduce the time cost of the branch‐and‐bound search for the optimal schedule. © 2004 Wiley Periodicals, Inc. Syst Comp Jpn, 35(5): 44–54, 2004; Published online in Wiley InterScience (www.interscience.wiley.com). DOI 10.1002/scj.10533

Fast Dimension-based Partitioning and Merging clustering algorithm

Wireless sensor network (WSN) applications have been studied extensively in recent years. Such applications involve resource-limited embedded sensor nodes that have small size and low power requirements. Based on the need for extended network lifetimes in WSNs in terms of energy use, the energy efficiency of computation and communication operations in the sensor nodes becomes critical. Digital Signal Processing (DSP) applications typically require intensive data processing operations and as a result are difficult to implement directly in resource-limited WSNs. In this article, we present a novel design methodology for modeling and implementing computationally intensive DSP applications applied to wireless sensor networks. This methodology explores efficient modeling techniques for DSP applications, including data sensing and processing; derives formulations of Energy-Driven Partitioning (EDP) for distributing such applications across wireless sensor networks; and develops efficient heuristic algorithms for finding partitioning results that maximize the network lifetime. To address such an energy-driven partitioning problem, this article provides a new way of aggregating data and reducing communication traffic among nodes based on application analysis. By considering low data token delivery points and the distribution of computation in the application, our approach finds energy-efficient trade-offs between data communication and computation.

The use of multimedia in education has become an important element to improve the quality of education and to reduce the cost of the educational system. In addition, the students got the benefit of learning and understanding the material better than in the conventional way. In this work, a learning package called "Data Communications Learning Center" (DCLC) has been developed and tested. The main objective of this center is to help university students and others to learn data communication concepts, architectures and operations. The center is a world wide web (www) based, and it allows any student that uses a standard modem for the Internet access to use our center. The center has been developed to be easy to use. The initial evaluation of the center examined by a few students have complemented that the center has improved their understanding to some data communication concepts which the center already supports. Although our intention is to support one topic at the first stage of this project, one can easily add other topics to the center. Any professor who is willing to put his course in the center can achieve this without any prior knowledge about the internal design of the center.