Multiple and single address spaces: towards a middle ground

Two broad classes of memory models are available today: models with hardware cache coherence, used in conventional chip multiprocessors, and models that rely upon software to manage coherence, found in compute accelerators. In some systems, both types of models are supported using disjoint address spaces and/or physical memories. In this paper we present Cohesion, a hybrid memory model that enables fine-grained temporal reassignment of data between hardware-managed and software-managed coherence domains, allowing a system to support both. Cohesion can be used to dynamically adapt to the sharing needs of both applications and runtimes. Cohesion requires neither copy operations nor multiple address spaces. Cohesion offers the benefits of reduced message traffic and on-die directory overhead when software-managed coherence can be used and the advantages of hardware coherence for cases in which software-managed coherence is impractical. We demonstrate our protocol using a hierarchical, cached 1024-core processor with a single address space that supports both software-enforced coherence and a directory-based hardware coherence protocol. Relative to an optimistic, hardware-coherent baseline, a realizable Cohesion design achieves competitive performance with a 2× reduction in message traffic, 2.1× reduction in directory utilization, and greater robustness to on-die directory capacity.

Some aspects of research into distributed shared memory (DSM) systems which concern reliability are presented. The research is concerned with applications that use shared memory in single-site systems, and their extension to operation in a distributed environment. A DSM system consists of a shared address space that is accessed by logical operations. In the author's model processes create shared memory by using a segment; the segment's size, name, and access protection are defined when the memory is created. Processes locate and attach the segment into their virtual memory address space by name. When a process is finished with the segment, it may be detached. The last detach of a segment destroys the data. The shared portion of a process's address space is partitioned into data pages, which are replicated. System implementation, distributed architecture, and issues affecting DSM reliability are discussed. >

Next-generation workstations will have hardware support for digital continuous media (CM) such as audio and video. CM applications handle data at high rates, with strict timing requirements, and often in small chunks. If such applications are to run efficiently and predictably as user-level programs, an operating system must provide scheduling and IPC mechanisms that reflect these needs. We propose two such mechanisms: split-level CPU scheduling of lightweight processes in multiple address spaces, and memory-mapped streams for data movement between address spaces. These techniques reduce the the number of user/kernel interactions (system calls, signals, and preemptions). Compared with existing mechanisms, they can reduce scheduling and I/O overhead by a factor of 4 to 6.

Modern computers are built with increasingly complex software stack crossing multiple layers (i.e., worlds), where cross-world call has been a necessity for various important purposes like security, reliability, and reduced complexity. Unfortunately, there is currently limited cross-world call support (e.g., syscall, vmcall), and thus other calls need to be emulated by detouring multiple times to the privileged software layer (i.e., OS kernel and hypervisor). This causes not only significant performance degradation, but also unnecessary implementation complexity.

With the advent of the 64-bit microprocessor, the virtual address space supported by a workstation will be large enough to permit the use of a single shared address space spanning a network of workstations as the primary abstraction provided by a distributed operating system. In such a system, built upon a software distributed shared memory, the programmer has considerable flexibility when choosing a mechanism for interprocess communication. This flexibility permits the programmer to make a case-by-case choice between simplicity and performance when both goals are not simultaneously achievable. With the inclusion of mechanisms supporting protection and fault tolerance, it is believed that such a system can provide the advantages of conventional message-based distributed operating systems, in addition to several other benefits e.g. easy sharing of complex data structures between processes, transparent replication of server functions, and a uniform interface for all communication. >

Memory-centric computing demands careful organization of the virtual address space, but traditional methods for doing so are inflexible and inefficient. If an application wishes to address larger physical memory than virtual address bits allow, if it wishes to maintain pointer-based data structures beyond process lifetimes, or if it wishes to share large amounts of memory across simultaneously executing processes, legacy interfaces for managing the address space are cumbersome and often incur excessive overheads. We propose a new operating system design that promotes virtual address spaces to first-class citizens, enabling process threads to attach to, detach from, and switch between multiple virtual address spaces. Our work enables data-centric applications to utilize vast physical memory beyond the virtual range, represent persistent pointer-rich data structures without special pointer representations, and share large amounts of memory between processes efficiently.

Irregular memory access patterns pose performance and user productivity challenges on distributed-memory systems. They can lead to fine-grained remote communication and the data access patterns are often not known until runtime. The Partitioned Global Address Space (PGAS) programming model addresses these challenges by providing users with a view of a distributed-memory system that resembles a single shared address space. However, this view often leads programmers to write code that causes fine-grained remote communication, which can result in poor performance. Prior work has shown that the performance of irregular applications written in Chapel, a high-level PGAS language, can be improved by manually applying optimizations. However, applying such optimizations by hand reduces the productivity advantages provided by Chapel and the PGAS model. We present an inspector-executor based compiler optimization for Chapel programs that automatically performs remote data replication. While there have been similar compiler optimizations implemented for other PGAS languages, high-level features in Chapel such as implicit processor affinity lead to new challenges for compiler optimization. We evaluate the performance of our optimization across two irregular applications. Our results show that the total runtime can be improved by as much as 52x on a Cray XC system with a low-latency interconnect and 364x on a standard Linux cluster with an Infiniband interconnect, demonstrating that significant performance gains can be achieved without sacrificing user productivity.

The challenge of verification of a multi-processor design is growing rapidly due to the increase of complexity in the design, bus protocols, weakly ordered memory and the presence of multiple hierarchies of caches in the memory subsystem. Adding to this complexity, ARM architecture also defines a set of attributes and characteristics required to support memory and devices in the system memory map. One of the key issues that need to be addressed, under these scenarios, is making sure that memory is consistent and coherent as different processors access, cache and update a single shared address space. In this work we present an ARM internal random instruction sequence (RIS) tool, MEMRIS (Memory Random Instruction Sequence), which provides a scalable approach to verify and validate memory coherence in multi-processor/cluster systems with up to 8-processors in both AARCH32/AARCH64. The output format of the generator is native ARM executables intended to run in a top-level simulation based environment. The MEMRIS generator allows the user to control many aspects of the test such as the number, size and location of address spaces as well as the mix of instructions that will operate on the defined memory regions. These memory regions are then configured to be shared between the Processing Elements (PEs), using defined address constraints, to hence trigger various sharing scenarios and invoke snoops and cache-line migrations. These scenarios are built using assembly sequences bounded into discrete zones. The zones are synchronized with various primitives throughout the test, i.e. Using Bakers algorithm and exclusives. The tool also internally models the code execution threads as to provide an optional self-check memory sequence at the end of the test.

Data parallel languages designed for distributed memory computing environments provide a single global address space to the programmer. The mapping from this global address space to the distributed local address space is performed by a compiler, which does this mapping based on the array distribution format. Thus, each array in a data parallel language program has its own distribution format. The Reshape function changes the array distribution format as well as the array shape. However, the changed distribution format cannot be represented by any distribution format supported in current languages. Because there is no suitable distribution format, it is necessary to change the reshaped distribution format to an existing distribution format, with heavy overhead due to the redistribution function. To eliminate the redistribution step in Reshape function, we have proposed a new distribution format, HIER-CYCLIC, which can represent the reshaped distribution format. We have also proposed a language syntax to use HIER-CYCLIC and a compiling mechanism. Finally, we performed an experiment on an IBM-SP2 machine using a shift function.

The X10 programming language is intended to ease the programming of scalable concurrent and distributed applications. X10 augments a familiar imperative object-oriented programming model with constructs to support light-weight asynchronous tasks as well as execution across multiple address spaces. A crucial aspect of X10's runtime system is the scheduling of concurrent tasks. Work-stealing schedulers have been shown to efficiently load balance fine-grain divide-and-conquer task-parallel program on SMPs and multicores. But X10 is not limited to shared-memory fork-join parallelism. X10 permits tasks to suspend and synchronize by means of conditional atomic blocks and remote task invocations.In this paper, we demonstrate that work-stealing scheduling principles are applicable to a rich programming language such as X10, achieving performance at scale without compromising expressivity, ease of use, or portability. We design and implement a portable work-stealing execution engine for X10. While this engine is biased toward the efficient execution of fork-join parallelism in shared memory, it handles the full X10 language, especially conditional atomic blocks and distribution.We show that this engine improves the run time of a series of benchmark programs by several orders of magnitude when used in combination with the C++ backend compiler and runtime for X10. It achieves scaling comparable to state-of-the art work-stealing scheduler implementations---the Cilk++ compiler and the Java fork/join framework---despite the dramatic increase in generality.

No abstract available.

With reference to a distributed context consisting of computers connected by a local area network, we present the organization of a memory management system giving physical support to a uniform, persistent vision of storage according to a single address space paradigm. Our system implements a two-layer storage hierarchy in which the distributed secondary memory stores the valid data items and the primary memory supports a form of data caching, for fast processor access. The proposed system defines a small, powerful set of operations that allow application programs to exert explicit control over the memory management activities at the levels of physical storage allocation, data migration across the network, and the data movements between the secondary memory and the primary memory. The system, that has been implemented in prototype form, is assessed from a number of viewpoints. We show that the storage requirements of the information for memory management are negligible. Moreover, the number of messages necessary to determine the network location of a given data item is low and independent of both the network size and the past movements of this data item in the distributed storage.Copyright © 2000 John Wiley & Sons, Ltd.

The significant performance-to-cost ratio advantage of clusters, combined with recent advances in middleware (programming environment) and networking technologies, has made them the single most popular and fastest growing platform for high performance computing in recent years. While the message passing interface (MPI) still dominates as a means of parallel programming in clusters, it is nevertheless desirable for programmers to program in a single address space, not only across a cluster but also among multiple, likely heterogeneous, clusters so as to significantly extend the computing power of a single cluster. In this paper we propose a distributed shared object (DSO) model based on a distributed hierarchical consistency model (DHCM) protocol for heterogeneous clusters. DHCM, inspired by but significantly improved over the local consistency, is designed to help maintain coherence and consistency in a DSO programming environment and to adapt to different levels of consistency. The notion of adaptive consistency is proposed and partially implemented to improve the efficiency in consistency control, and scalability is addressed as well through the hierarchical structure of the protocol design. We implemented this model purely in Java for portability and heterogeneity. The performance of DHCM is evaluated by executing the LU application chosen from the SPLASH-2 benchmark suite on a 128-node Linux cluster. The experimental results show that the protocol with a hierarchical structure significantly outperforms the protocol with a single-tier in terms of execution time, indicating higher scalability.

This paper presents Weaves, a language neutral framework for scalable execution of legacy parallel scientific codes. Weaves supports scalable threads of control and multiple namespaces with selective sharing of state within a single address space. We resort to two examples for illustration of different aspects of the framework and to stress the diversity of its application domains. The more expressive collaborating partial differential equation (PDE) solvers are used to exemplify developmental aspects, while freely available Sweep3D is used for performance results. We outline the framework in the context of shared memory systems, where its benefits are apparent. We also contrast Weaves against existing programming paradigms, present use cases, and outline its implementation. Preliminary performance tests show significant scalability over process-based implementations of Sweep3D.

CORBA is a commercial standard for distributed object computing which shows great promise in the development of distributed programs. Its interface description language (IDL) enables objects to be developed independently of the underlying programming language, operating system, or computer architecture on which they will execute. While this is sufficient in many environments, programs deployed in a wide-area distributed system encounter conditions which are much more hostile and varying than those operating in a single address space or within a single local area network. In this paper we discuss four major problems we have observed in our developing and deploying wide-area distributed object applications and middleware. First, most programs are developed ignoring the variable wide area conditions. Second, when application programmers do try to handle these conditions, they have great difficulty because these harsh conditions are different from those of the local objects they are used to dealing with. Third, IDL hides information about the tradeoffs any implementation of an object must make. Fourth, there is presently no way to systematically reuse current technology components which deal with these conditions, so code sharing becomes impractical. In this paper we also describe our architecture, Quality of Service for CORBA Objects (QuO), which we have developed to overcome these limitations and integrate their solution by providing QoS abstractions to CORBA objects. First, it makes these conditions first class entities and integrates knowledge of them over time, space, and source. Second, it reduces their variance by masking. Third, it exposes key design decisions of an object's implementation and how it will be used. Fourth, it supports reuse of various architectural components and automatically generates others. © 1997 John Wiley & Sons, Inc.