A software architecture for global address space communication on clusters: put/get on fast messages

We are proposing a shared-memory communication infrastructure that provides a common parallel programming interface for FPGA and CPU components in a heterogeneous system. Our intent is to ease the integration of reconfigurable hardware into parallel programming models like Partitioned Global Address Space (PGAS). For this purpose, we introduce a remote memory access component based on Active Messages that implements the core API of the Berkeley GASNet communication library, and a simple controller that manages communication and synchronization for custom FPGA cores. We demonstrate how these components deliver a simple and easily configurable communication mechanism between distributed memories in a multi-FPGA system with processors as well as custom hardware nodes.

The first exascale supercomputers are expected by the end of this decade and will presumably feature an increase in core count, but a decrease in the amount of memory available per core. As of now, it is still unclear if the current programming models will provide high performance on exascale systems. One programming model considered to be an alternative to MPI is the so-called partitioned global address space (PGAS) model. Within this paper we evaluate a relatively new PGAS API: the Global Address Space Programming Interface (GASPI) and compare it to MPI on the basis of microbenchmarks. These benchmarks show that GASPI provides about the same level of performance for single-threaded communication, but is up to an order of magnitude faster than both Intel and IBM MPI for multi-threaded communication. Hereafter, we discuss the different features of GASPI in comparison to two main PGAS languages, namely UPC and CAF. In addition, we present a basic numerical algorithm, a dense matrix-matrix multiplication, as an example on how an implementation can make efficient use of GASPI's features, especially the asynchronous and one-sided communication mechanisms.

Dense LU factorization has a high ratio of computation to communication and, as evidenced by the High Performance Linpack (HPL) benchmark, this property makes it scale well on most parallel machines. Nevertheless, the standard algorithm for this problem has non-trivial dependence patterns which limit parallelism, and local computations require large matrices in order to achieve good single processor performance. We present an alternative programming model for this type of problem, which combines UPC's global address space with lightweight multithreading. We introduce the concept of memory-constrained lookahead where the amount of concurrency managed by each processor is controlled by the amount of memory available. We implement novel techniques for steering the computation to optimize for high performance and demonstrate the scalability and portability of UPC with Teraflop level performance on some machines, comparing favourably to other state-of-the-art MPI codes.

The Partitioned Global Address Space (PGAS) programming paradigm offers ease-of-use in expressing parallelism through a global shared address space while emphasizing performance by providing locality awareness through the partitioning of this address space. Therefore, the interest in PGAS programming languages is growing and many new languages have emerged and are becoming ubiquitously available on nearly all modern parallel architectures. Recently, new parallel machines with multiple cores are designed for targeting high performance applications. Most of the efforts have gone into benchmarking but there are a few examples of real high performance applications running on multicore machines. In this paper, we present and evaluate a parallelization technique for implementing a local DNA sequence alignment algorithm using a PGAS based language, UPC (Unified Parallel C) on a chip multithreading architecture, the UltraSPARC T1.

An increasing number of supercomputers adopt a heterogeneous architecture, consisting of both general purpose CPUs and specialized accelerators. Such design is beneficial for scalability and power, but on the other hand, heterogeneity brings new challenges in communication systems to connect heterogeneous components and provide support for programming. The communication system of the Dawning 6000 connects two kinds of heterogeneous processors, Loongson and AMD, and adopts a three layer architecture with an intranode layer between heterogeneous components. To efficiently connect heterogeneous components, the system forms a global address space and provides a mechanism for message transmission via an in-node global store; and employing Infiniband network, provides an OS-bypassing virtualization method to share an Infiniband card between nodes. To facilitate programming on heterogeneous processors, it supports unified parallel C (UPC), with a modified complier based on global address space. Also, a special collective network is implemented for collective operations. Results obtained from a prototype system prove these features to be both feasible and efficient.

UPC++ is a C++ library that provides Partitioned Global Address Space (PGAS) programming. It is designed for writing parallel programs that run efficiently and scale well on distributed-memory parallel computers. The PGAS model is single program, multiple-data (SPMD), with each separate constituent process having access to local memory as it would in C++. PGAS additionally provides one-sided Remote Memory Access (RMA) to a global address space, which is allocated in shared segments that are distributed over the processes. UPC++ also features Remote Procedure Call (RPC) communication, making it easy to move computation to operate on data that resides on remote processes. In UPC++, all communication operations are explicit, which encourages programmers to be aware of the cost of communication and data movement. Moreover, all communication operations are asynchronous by default, to enable programmers to write code that scales well even on hundreds of thousands of cores.

Distributed memory parallel machines are increasingly being used by the scientific community to provide computational power for solving problems. Building a distributed memory parallel machine offers a tremendous advantage in terms of cost and scalability over building a shared memory parallel machine. Unfortunately, distributed memory machines are much more difficult to program, mainly due to the lack of a single global address space. Communication between processors in a distributed memory machine is accomplished by message passing. The time cost for acquiring data from another processor's memory is significantly greater than the time cost of acquiring data from local memory. To assist the programmer in using a distributed memory machine, compilers have been developed which alleviate the need for the programmer to manage the lack of a global address space. These compilers will generate semantically correct code containing communication routines to produce a logically shared memory machine on a physically distributed memory machine. However, the runtime performance of the code emitted from some of these compilers can be further improved. This thesis is primarily concerned with advanced compilation techniques which will provide better runtime performance of programs written in imperative languages for regular scientific programs. The optimization of communication generation for distributed memory machines is used to produce this result. We look at two "state of the art" compilation systems and find that improvement in the communication code generated by these compilers is still possible. We have simplified existing descriptions of the various formalisms found in these compilers, while still providing enough formalism to study the problems in depth. We have extended some existing optimization techniques by enhancing the elimination of masks, by extending the notion of relaxing the owner computes rule, by extending the notion of replicating computation, and by extending the notion of replicating data. This thesis has discussed a novel technique to split loops so that they do not contain any guarded statements. Finally, we have suggested avenues for future research, as it is clear from this thesis that more work is required before compilation systems for distributed memory parallel machines will be widely used for general purpose computing.

UPC++ is a C++11 library that provides Partitioned Global Address Space (PGAS) programming. It is designed for writing parallel programs that run efficiently and scale well on distributed-memory parallel computers. The PGAS model is single program, multiple-data (SPMD), with each separate constituent process having access to local memory as it would in C++. However, PGAS also provides access to a global address space, which is allocated in shared segments that are distributed over the processes. UPC++ provides numerous methods for accessing and using global memory. In UPC++, all operations that access remote memory are explicit, which encourages programmers to be aware of the cost of communication and data movement. Moreover, all remote-memory access operations are by default asynchronous, to enable programmers to write code that scales well even on hundreds of thousands of cores.

We present DRASync, a region-based allocator that implements a global address space abstraction for MPI programs with pointer-based data structures. The main features of DRASync are: (a) it amortizes communication among nodes to allow efficient parallel allocation in a global address space; (b) it takes advantage of bulk deallocation and good locality with pointer-based data structures. (c) it supports ownership semantics of regions by nodes akin to reader-writer locks, which makes for a high-level, intuitive synchronization tool in MPI programs, without sacrificing message-passing performance. We evaluate DRASync against a state-of-the-art distributed allocator and find that it produces comparable performance while offering a higher level abstraction to programmers.

Recent technology advances in high-density, byte-addressable non-volatile memory (NVM) and low-latency interconnects have enabled building large-scale systems with a large disaggregated fabric-attached memory (FAM) pool shared across decentralized compute nodes. The OpenFAM API is an API for programming with persistent FAM that is inspired by the OpenSHMEM partitioned global address space (PGAS) model. Unlike OpenSHMEM, where each node contributes local memory toward a logically shared global address space, FAM isn’t associated with a particular node and can be addressed directly from any node without the cooperation or involvement of another node. The OpenFAM API enables programmers to manage FAM allocations, access FAM-resident data structures, and order FAM operations. Because state in FAM can survive program termination, the API also provides interfaces for naming and managing data beyond the lifetime of a single program invocation.

Using large-scale multicore systems to get the maximum performance and energy efficiency with manageable programmability is a major challenge. The partitioned global address space (PGAS) programming model enhances programmability by providing a global address space over large-scale computing systems. However, so far the performance and energy efficiency of the PGAS model on multicore-based parallel architectures have not been investigated thoroughly. In this paper we use a set of selected kernels from the well-known NAS Parallel Benchmarks to evaluate the performance and energy efficiency of the UPC programming language, which is a widely used implementation of the PGAS model. In addition, the MPI and OpenMP versions of the same parallel kernels are used for comparison with their UPC counterparts. The investigated hardware platforms are based on multicore CPUs, both within a single 16-core node and across multiple nodes involving up to 1024 physical cores. On the multi-node platform we used the hardware measurement solution called High definition Energy Efficiency Monitoring tool in order to measure energy. On the single-node system we used the hybrid measurement solution to make an effort into understanding the observed performance differences, we use the Intel Performance Counter Monitor to quantify in detail the communication time, cache hit/miss ratio and memory usage. Our experiments show that UPC is competitive with OpenMP and MPI on single and multiple nodes, with respect to both the performance and energy efficiency.

UPC++ is a C++11 library that provides Asynchronous Partitioned Global Address Space (APGAS) programming. It is designed for writing parallel programs that run efficiently and scale well on distributed-memory parallel computers. The APGAS model is single program, multiple-data (SPMD), with each separate thread of execution (referred to as a rank, a term borrowed from MPI) having access to local memory as it would in C++. However, APGAS also provides access to a global address space, which is allocated in shared segments that are distributed over the ranks. UPC++ provides numerous methods for accessing and using global memory. In UPC++, all operations that access remote memory are explicit, which encourages programmers to be aware of the cost of communication and data movement. Moreover, all remote-memory access operations are by default asynchronous, to enable programmers to write code that scales well even on hundreds of thousands of cores.

An architecture of high performance computer,called Hyper Parallel Processing(HPP),is proposed to satisfy the requirements of both High Performance Computing,and Utility Computing which will be the application model of data centers.HPP combines the benefits of the scalability of MPP,the communication efficiency of DSM,as well as the commodity of cluster.Comparisons of current main-stream high performance computer architectures show that none of them can satisfy both HPC and utility computing.The main features of HPP are Global Address Space(GAS) and Hyper Node with single Operating System image.HPP supports the distributed global address space including both memory and I/O,but without hardware cache coherence.A Hyper Node consists of a set of application specific CPUs and a(or more) OS specific CPU.The OS CPU maintains the single system image,while the application CPUs run only lightweight run-time software.Besides the GAS interconnect network for applications,a standard SAN connects all OS CPUs and I/O devices providing system management and storage service.HPP is able to provide many opportunities of innovative research in High Performance and Utility Computing areas,including communication,synchronization,programming model,node operating system,utility computing,fault isolation,CPU and system etc.According to HPP architecture,a prototype system of Dawning5000 HPC is implemented and the feasibility of HPP is proved.

Communication performance comparisons, in terms of message latency, between mesh and hierarchical ring interconnection networks for shared-memory multiprocessors are developed. Traffic in the networks is assumed to consist of the short, fixed-length messages needed for remote-memory read/write operations and cache coherency control. Square-mesh networks with bidirectional links and no end-around connections are compared to two- and three-level hierarchical rings. Wormhole routing is used in the meshes, with messages consisting of / flits, with each flit containing b bits. In the ring networks, each ring segment can contain a complete message of f × b bits. Meshes and three-level rings provide comparable delays for system sizes of up to 200 or 300 processor clusters. The delay comparisons are valid only for the lightly loaded traffic case in which contention and blocking are low. This is the situation in practical shared-memory multiprocessor systems with a low cache miss rate and a high physical locality of reference in the global address space.

Power consumption increasingly presents an upper bound on sustainable large scale computing performance and reliability. The Partitioned Global Address Space (PGAS) programming model is a family of parallel programming paradigms with a global address space for ease-of-use while providing locality awareness for efficient execution. Very little exploration has been done to determine the potential of PGAS programming models in improving scalable energy efficient computation for high performance computing (HPC) clusters. This paper examines features of the PGAS programming model that may support predictively reducing power consumption in distributed clusters via dynamic voltage frequency scaling (DVFS). These concepts are tested with Unified Parallel C (UPC) codes running on a cluster of commodity PCs which have been instrumented to measure power at the CPU socket level. We have also explored approaches to automating these power optimization techniques at compile time. Benchmarking results show a tangible reduction in power consumption without impacting the overall execution time of the program.