M-PVM: An Implementation of PVM for Multithreaded and Shared-Memory Environments

In an often-cited paper, A. Karp [1] commented on the “sorry state” of parallel programming, lamenting the difficulties in programming and debugging parallel programs on existing programming models. Indeed, a variety of programming models have been proposed with not entirely satisfactory results. In this note, we focus on the world of massively parallel shared memory machines and examine programming models for these machines. A shared memory parallel computer is one in which the processors communicate by reading and writing data words to a shared memory. We consider shared memory parallel computers with the following characteristics. Each processor has some fast, relatively small, private memory (e.g. registers, cache). There is a large shared memory composed of a series of memory banks that are accessible by all processors via an interconnection network. Each processor has one such memory bank which is local to it, i.e. it can access the bank without going through the interconnection network. The access time to a memory bank for a processor depends in part on the lengths of the links traversed and the number of hops in the path from the processor to the bank. Special coprocessors take care of interprocessor communication: both programmers and processors are relieved of the burden of routing the parallel requests to shared memory, including handling intermediate hops of requests traveling to their destinations. In order to overcome the high latency of accessing global memory, the parallel computer supports the pipelining of global memory accesses through the network. This implies that the interconnection network has sufficient bandwidth to handle the multiple requests per processor.

Microtasking versus Message Passing Parallelisation of the 3D-Combustion Code AIOLOS on the NEC SX-4

Distributed shared memory (DSM) systems allow programmers to use a shared memory programming model on distributed systems. However, DSM systems present new challenges to performance evaluation and debugging. While programmers write programs with a shared memory model using shared variables and shared memory synchronization primitives to coordinate processes, most performance problems arise from the implementation of these abstractions with messages. Because the DSM abstraction hides the underlying message passing, it is difficult to identify the sources of performance problems. We have developed an instrumentation system and performance evaluation techniques for DSM systems that correlate message passing behavior in the lower levels of a DSM system with the higher-level shared memory programming abstractions. We have developed a prototype tool called Xunify to demonstrate the feasibility of such an approach. Xunify provides graphical animations of DSM programs at three distinct levels and offers a powerful event filtering facility capable to help show system behavior corresponding to specific DSM constructs.

Nowadays clusters are one of the most used platforms in High Performance Computing and most programmers use the Message Passing Interface (MPI) library to program their applications in these distributed platforms getting their maximum performance, although it is a complex task. On the other side, OpenMP has been established as the de facto standard to program applications on shared memory platforms because it is easy to use and obtains good performance without too much effort. So, could it be possible to join both worlds? Could programmers use the easiness of OpenMP in distributed platforms? A lot of researchers think so. And one of the developed ideas is the distributed shared memory (DSM), a software layer on top of a distributed platform giving an abstract shared memory view to the applications. Even though it seems a good solution it also has some inconveniences. The memory coherence between the nodes in the platform is difficult to maintain (complex management, scalability issues, high overhead and others) and the latency of the remote-memory accesses which can be orders of magnitude greater than on a shared bus due to the interconnection network. Therefore this research improves the performance of OpenMP applications being executed on distributed memory platforms using a DSM with sequential consistency evaluating thoroughly the results from the NAS parallel benchmarks. The vast majority of designed DSMs use a relaxed consistency model because it avoids some major problems in the area. In contrast, we use a sequential consistency model because we think that showing these potential problems that otherwise are hidden may allow the finding of some solutions and, therefore, apply them to both models. The main idea behind this work is that both runtimes, the OpenMP and the DSM layer, should cooperate to achieve good performance, otherwise they interfere one each other trashing the final performance of applications. We develop three different contributions to improve the performance of these applications: (a) a technique to avoid false sharing at runtime, (b) a technique to mimic the MPI behaviour, where produced data is forwarded to their consumers and, finally, (c) a mechanism to avoid the network congestion due to the DSM coherence messages. The NAS Parallel Benchmarks are used to test the contributions. The results of this work shows that the false-sharing problem is a relative problem depending on each application. Another result is the importance to move the data flow outside of the critical path and to use techniques that forwards data as early as possible, similar to MPI, benefits the final application performance. Additionally, this data movement is usually concentrated at single points and affects the application performance due to the limited bandwidth of the network. Therefore it is necessary to provide mechanisms that allows the distribution of this data through the computation time using an otherwise idle network. Finally, results shows that the proposed contributions improve the performance of OpenMP applications on this kind of environments.

Parallel programming has been widely used in many scientific and technical areas to solve large problems. While general-purpose processors have rich infrastructure to support parallel programming on shared memory, such as coherent caches and synchronization libraries, parallel programming infrastructure for FPGAs is limited. Thus, development of FPGA-based parallel algorithms remains difficult. In this work, we seek to simplify parallel programming on FPGAs. We provide a set of easy-to-use declarative primitives to maintain coherency and consistency of accesses to shared memory resources. We propose a shared-memory service that automatically manages coherent caches on multiple FPGAs. Experimental results of a 2-dimensional heat transfer equation show that the shared memory service with our distributed coherent caches outperforms a centralized cache by 2.6x. To handle synchronization, we provide new lock and barrier primitives that leverage native FPGA communication capabilities and outperform traditional through-memory primitives by 1.8x.

The MULTIPLUS project aims at the development of a modular parallel architecture suitable for the study of several aspects of parallelism in both true shared memory and virtual shared memory environments. The MULTIPLUS architecture is able to support up to 1024 Processing Elements based on SPARC microprocessors. The MULPLIX Unix-like operating system offers a suitable parallel programming environment for the MULTIPLUS architecture by providing facilities for the creation of threads, the allocation of private and shared memory space and the efficient use of synchronization primitives. After presenting the main features of the MULTIPLUS architecture and of the MULPLIX operating system, the paper describes in detail the design and the implementation of the three MULTIPLUS architecture basic hardware modules: the Processing Element, the Multistage Interconnection Network and the I/O Processor. In addition, the definition of the MULPLIX parallel programming primitives is discussed and their use is illustrated through an example. Finally, future directions in the development of the MULTIPLUS research project are commented.

OpenMP is an architecture-independent language for programming in the shared memory model. OpenMP is designed to be simple and powerful in terms of programming abstractions. Unfortunately, the architecture-independent abstractions sometimes come with the price of low parallel performance. This is especially true for applications with an unstructured data access pattern running on distributed shared memory systems (DSM). Here, proper data distribution and algorithmic optimizations play a vital role for performance. In this article, we have investigated ways of improving the performance of an industrial class conjugate gradient (CG) solver, implemented in OpenMP running on two types of shared memory systems. We have evaluated bandwidth minimization, graph partitioning and reformulations of the original algorithm reducing global barriers. By a detailed analysis of barrier time and memory system performance, we found that bandwidth minimization is the most important optimization reducing both L2 misses and remote memory accesses. On a uniform memory system, we get perfect scaling. On a NUMA system, the performance is significantly improved with the algorithmic optimizations leaving the system dependent global reduction operations as a bottleneck.

CHAPTER 4 - Processes, Shared Memory, and Simple Parallel Programs

In this paper, we propos a new shared memory model: Transactionalmemory Coherence and Consistency (TCC).TCC providesa model in which atomic transactions are always the basicunit of parallel work, communication, memory coherence, andmemory reference consistency.TCC greatly simplifies parallelsoftware by eliminating the need for synchronization using conventionallocks and semaphores, along with their complexities.TCC hardware must combine all writes from each transaction regionin a program into a single packet and broadcast this packetto the permanent shared memory state atomically as a large block.This simplifies the coherence hardware because it reduces theneed for small, low-latency messages and completely eliminatesthe need for conventional snoopy cache coherence protocols, asmultiple speculatively written versions of a cache line may safelycoexist within the system.Meanwhile, automatic, hardware-controlledrollback of speculative transactions resolves any correctnessviolations that may occur when several processors attemptto read and write the same data simultaneously.The cost of thissimplified scheme is higher interprocessor bandwidth.To explore the costs and benefits of TCC, we study the characterisitcsof an optimal transaction-based memory system, and examinehow different design parameters could affect the performanceof real systems.Across a spectrum of applications, the TCC modelitself did not limit available parallelism.Most applications areeasily divided into transactions requiring only small write buffers,on the order of 4-8 KB.The broadcast requirements of TCCare high, but are well within the capabilities of CMPs and small-scaleSMPs with high-speed interconnects.

This paper addresses the problem of the 3D game design aided by multi-dimensional collision detection algorithm with GPU optimization. For the designed model, our task will be focused on the 3D game design, the GPU performance on the image computing and neural network computing will be essential, hence, the graph neural network algorithm is considered. This study firstly considers the GPU optimization with shared memory model. Shared memory can be used to design the memory access mode in order to realize the combination of the memory access which improves the efficiency. Then, this study considers the multi-dimensional collision detection algorithm design. In order to achieve the unique representation of matrix translation and also spatial points, this paper then adopts the homogeneous coordinate system in matrix transformation. That improves the traditional model. Furthermore, the 3D game design steps are clarified. Based on the experiment under different scenarios, the game design sample code, the GPU performance and algorithm performance are all tested.

We consider a set of users wishing to receive a service in an asynchronous distributed system. Such users declare their wishes and then wait to gain admittance to be served. Except for the initial transient period, at least one user must be waiting to be served, and the system should be as fair as possible for users. A procedure that ensures such a situation is called a turn function. It can be easily implemented in the asynchronous multi-writer/reader shared memory model. The turn function is useful to solve some problems concerning temporal orders within an asynchronous distributed system (e.g., the mutual exclusion problem and the k-exclusion problem). In this paper we propose an algorithm for the turn function that can be implemented in the single-writer/multi-reader shared memory model. Its implementation is simpler than the general method of simulating operations of multi-writer/reader shared variables by using a bounded concurrent time-stamp scheme in the single-writer/multi-reader shared memory model. We show the correctness of our algorithm and discuss its efficiency.KeywordsShared VariableService RegionMutual ExclusionRead OperationRelease RelationshipThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

GPUs provide high-bandwidth/low-latency on-chip shared memory and L1 cache to efficiently service a large number of concurrent memory requests. Specifically, concurrent memory requests accessing contiguous memory space are coalesced into warp-wide accesses. To support such large accesses to L1 cache with low latency, the size of L1 cache line is no smaller than that of warp-wide accesses. However, such L1 cache architecture cannot always be efficiently utilized when applications generate many memory requests with irregular access patterns especially due to branch and memory divergences that make requests uncoalesced and small. Furthermore, unlike L1 cache, the shared memory of GPUs is not often used in many applications, which essentially depends on programmers. In this article, we propose Elastic-Cache, which can efficiently support both fine- and coarse-grained L1 cache line management for applications with both regular and irregular memory access patterns to improve the L1 cache efficiency. Specifically, it can store 32- or 64-byte words in non-contiguous memory space to a single 128-byte cache line. Furthermore, it neither requires an extra memory structure nor reduces the capacity of L1 cache for tag storage, since it stores auxiliary tags for fine-grained L1 cache line managements in the shared memory space that is not fully used in many applications. To improve the bandwidth utilization of L1 cache with Elastic-Cache for fine-grained accesses, we further propose Elastic-Plus to issue 32-byte memory requests in parallel, which can reduce the processing latency of memory instructions and improve the throughput of GPUs. Our experiment result shows that Elastic-Cache improves the geometric-mean performance of applications with irregular memory access patterns by 104% without degrading the performance of applications with regular memory access patterns. Elastic-Plus outperforms Elastic-Cache and improves the performance of applications with irregular memory access patterns by 131%.

The currently dominant programming models to write software for multicore processors use threads that run over shared memory. However, as the core count increases, cache coherency protocols get very complex and ineffective, and maintaining a shared memory abstraction becomes expensive and impractical. Moreover, writing multithreaded programs is notoriously difficult, as the programmer needs to reason about all the possible thread interleavings and interactions, including the myriad of implicit, non-obvious, and often unpredictable thread interactions through shared memory. Overall, as processors get more cores and parallel software becomes mainstream, the shared memory model reaches its limits regarding ease of programming and efficiency.This position paper presents two ideas aiming to solve the problem. First, we restrict the way the programmer expresses parallelism: The program is a collection of possibly recursive tasks, where each task is atomic and cannot communicate with any other task during its execution. Second, we relax the requirement for coherent shared memory: Each task defines its memory footprint, and is guaranteed to have exclusive access to that memory during its execution. Using this model, we can then define a runtime system that transparently performs the data transfers required among cores without cache coherency, and also produces a deterministic execution of the program, provably equivalent to its sequential elision.

: One of the most difficult issues that must be addressed when studying a class of parallel algorithms is the problem of choosing a model that captures the inherent difficulty of implementing these algorithms on a multiprocessor architecture. Shared memory models have proven to be an effective tool for deriving lower bounds on the complexity of comparison problems. In particular, a speed-up of lg(P) is possible for the problem of finding an element in an N-element sorted list, and speed-ups of PlglgP and P are possible for merging N-element sorted lists of P processors for the cases N=P and PN respectively. In practice, these speed-ups are not attainable since the shared memory models ignore many practical considerations in multiprocessor systems, such as interprocessor communications, distribution of data on local memories and limited fan-out of memory locations. In this paper we introduce a model for parallel computation that is strictly weaker than the shared memory models. The model is based on an actual machine currently being constructed (ZMOB). We examine the communication facilities available in the model and show that lower bounds for merging and searching on shared memory models are attainable (within a constant).

Parallel programs have traditionally been implemented on distributed systems using conventional programming languages, such as Fortran or C, plus a library with support for communication, such as, for instance, PVM [11]. Also, some effort has been put into developing parallel programming languages [13]. Usually, the different development platforms are compared as to application execution time or parallel efficiency. However, as pointed out in [4], when measuring the performance of a parallel application or programming tool, different metrics may need to be taken into consideration. The ease of construction of an application and the time a programmer spends to implement it may in some cases be more important than the final execution time. Working under this assumption, this work argues that the use of interpreted languages as a basis for parallel programming tools should be further investigated. As support for our argument, we show an example of use of an interpreted language in a classical application in parallel distributed programming, namely, that of detecting program termination. Two different algorithms for termination detection are implemented in Lua, an extension language developed at PUC-Rio [5].