Algorithmic optimizations of a conjugate gradient solver on shared memory architectures

Distributed shared memory (DSM) systems allow programmers to use a shared memory programming model on distributed systems. However, DSM systems present new challenges to performance evaluation and debugging. While programmers write programs with a shared memory model using shared variables and shared memory synchronization primitives to coordinate processes, most performance problems arise from the implementation of these abstractions with messages. Because the DSM abstraction hides the underlying message passing, it is difficult to identify the sources of performance problems. We have developed an instrumentation system and performance evaluation techniques for DSM systems that correlate message passing behavior in the lower levels of a DSM system with the higher-level shared memory programming abstractions. We have developed a prototype tool called Xunify to demonstrate the feasibility of such an approach. Xunify provides graphical animations of DSM programs at three distinct levels and offers a powerful event filtering facility capable to help show system behavior corresponding to specific DSM constructs.

Data prefetching is a technique where a processing unit issues one or more non-blocking load operations before the actual data items are required. The access latency of prefetching can be alleviated by overlapping it with other executions which are independent of the prefetched data. In distributed shared memory (DSM) systems, remote memory accesses take much longer than local ones, and hence data prefetching should be effective for such systems. However, to our knowledge, relatively little research has been done for data prefetching on DSM systems. This paper is concerned with issues of supporting data prefetching on DSM systems. Our approach is to develop a new memory consistency semantic (MCS) model under which the prefetchable shared data objects, as well as the best moment to launch a prefetching operation, can be easily identified. Our new MCS, called aggressive consistency, utilizes the coherence-on-demand concept and supports a special synchronization operation called SYNC, which also acts as the prefetching indicator. Preliminary simulation results show that our prefetching approach, combined with aggressive consistency, can substantially improve the performance of DSM systems.

The performance of page-based software shared virtual memory (SVM) is still far from that achieved on hardware-coherent distributed shared memory (DSM) systems. The interrupt cost for asynchronous protocol processing has been found to be a key source of performance loss and complexity.This paper shows that by providing simple and general support for asynchronous message handling in a commodity network interface (NI), and by altering SVM protocols appropriately, protocol activity can be decoupled from asynchronous message handling and the need for interrupts or polling can be eliminated. The NI mechanisms needed are generic, not SVM-dependent. They also require neither visibility into the node memory system nor code instrumentation to identify memory operations. We prototype the mechanisms and such a synchronous home-based LRC protocol, called GeNIMA (GEneral-purpose Network Interface support in a shared Memory Abstraction), on a cluster of SMPs with a programmable NI, though the mechanisms are simple and do not require programmability.We find that the performance improvements are substantial, bringing performance on a small-scale SMP cluster much closer to that of hardware-coherent shared memory for many applications, and we show the value of each of the mechanisms in different applications. Application performance improves by about 37% on average for reasonably well performing applications, even on our relatively slow programmable NI, and more for others. We discuss the key remaining bottlenecks at the protocol level and use a firmware performance monitor in the NI to understand the interactions with and the implications for the communication layer.

Distributed Shared Memory (DSM) systems have gained popular acceptance by combining the scalability and low cost of distributed system with the ease of use of single address space. Many new hardware DSM and software DSM systems have been proposed in recent years. In general, benchmarking is widely used to demonstrate the performance advantages of new systems. However, the common method used to summarize the measured results is the arithmetic mean of ratios, which is incorrect in some cases. Furthermore, many published papers list a lot of data only, and do not summarize them effectively, which confuse users greatly. In fact, many users want to get a single number as conclusion, which is not provided in old summarizing techniques. Therefore, a new data-summarizing technique based on confidence interval is proposed in this paper. The new technique includes two data-summarizing methods: (1) paired confidence interval method; (2) unpaired confidence interval method. With this new technique, it is concluded that at some confidence one system is better than others. Four examples are shown to demonstrate the advantages of this new technique. Furthermore, with the help of confidence level, it is proposed to standardize the benchmarks used for evaluating DSM systems so that a convincing result can be got. In addition, the new summarizing technique fits not only for evaluating DSM systems, but also for evaluating other systems, such as memory system and communication systems.

One of the factors that can influence the performance of a DSM system is the efficiency of multipoint access on the interconnection network. This work presents an evaluation of the Brazos system, a software implemented distributed shared memory (DSM) system designed for x86 SMP nodes running Windows NT and that takes advantage of multicast communication, with regard to different configurations of the communication network: broadcast fast Ethernet, switched fast Ethernet and LANE-ATM. In this evaluation, although the bandwidth of the ATM is larger than the fast Ethernet, low performance using LANE-ATM was observed, compared to the Brazos system performance using broadcast or switched Fast Ethernet. It shows the importance of the communication subsystem and the necessity of improvements on aspects like multipoint access, which has been not much explored on networks like ATM.

Summary form only given. We describe the design and analysis of a scalable architecture suitable for large-scale DSMs (distributed shared memory) systems. The approach is based on an interconnect technology which combines optical components and a novel architecture design. In DSM systems, as the network size increases, network contention results in increasing the critical remote memory access latency, which significantly penalizes the performance of DSM systems. In our proposed architecture called RAPID (reconfigurable and scalable all-photonic interconnect for distributed-shared memory), we provide high connectivity by maximizing the channel availability for remote communication to reduce the remote memory access latency. RAPID also provides fast and efficient unicast, multicast and broadcast capabilities using a combination of aggressively designed wavelength, time and space-division multiplexing techniques. We evaluated RAPID based on network characteristics, power budget criteria and simulation using synthetic traffic workloads and compared it against other scalable electrical networks. We found that RAPID, not only outperforms other networks, but also, satisfies most of the requirements of shared memory multiprocessor design such as low latency, high bandwidth, high connectivity, and easy scalability.

Microtasking versus Message Passing Parallelisation of the 3D-Combustion Code AIOLOS on the NEC SX-4

High speed networks and rapidly improving microprocessor performance make the network of workstations an extremely important tool for parallel computing in order to speedup the execution of scientific applications. Shared memory is an attractive programming model for designing parallel and distributed applications, where the programmer can focus on algorithmic development rather than data partition and communication. Based on this important characteristic, the design of systems to provide the shared memory abstraction on physically distributed memory machines has been developed, known as Distributed Shared Memory (DSM). DSM is built using specific software to combine a number of computer hardware resources into one computing environment. Such an environment not only provides an easy way to execute parallel applications, but also combines available computational resources with the purpose of speeding up execution of these applications. DSM systems need to maintain data consistency in memory, which usually leads to communication overhead. Therefore, there exists a number of strategies that can be used to overcome this overhead issue and improve overall performance. Strategies as prefetching have been proven to show great performance in DSM systems, since they can reduce data access communication latencies from remote nodes. On the other hand, these strategies also transfer unnecessary prefetching pages to remote nodes. In this research paper, we focus on the access pattern during execution of a parallel application, and then analyze the data type and behavior of parallel applications. We propose an adaptive data classification scheme to improve prefetching strategy with the goal to improve overall performance. Adaptive data classification scheme classifies data according to the accessing sequence of pages, so that the home node uses past history access patterns of remote nodes to decide whether it needs to transfer related pages to remote nodes. From experimental results, we can observe that our proposed method can increase the accuracy of data access in effective prefetch strategy by reducing the number of page faults and misprefetching. Experimental results using our proposed classification scheme show a performance improvement of about 9---25% over the same benchmark applications running on top of an original JIAJIA DSM system.

Two most commonly used classifications of reference locality are: temporal locality and spatial locality. This paper introduces a new class of reference locality, called Regional Locality, which is the program behavior that a set of addresses which are accessed in one critical or non-critical region will be very likely accessed as a whole in the same critical region or other non-critical regions. We proposed three updates propagation protocols based on Regional Locality in Distributed Shared Memory systems. These protocols include: Selective Lazy/Eager Updates Propagation protocol, First Hit Updates Propagation protocol, and Second Hit Updates Propagation protocol. Our experimental results indicate that Regional Locality exists in executions of many Distributed Shared Memory concurrent programs. We have shown that the proposed protocols outperform the existing updates propagation protocols based on temporal locality. Exploring Regional Locality in other shared memory systems would be an interesting future research direction

Increasing the core count of CPUs to increase computational performance has been a significant trend for the better part of a decade. This has led to an unprecedented availability of large shared memory machines. Programming paradigms and systems are shifting to take advantage of this architectural change, so that intra-node parallelism can be fully utilized. Algorithms designed for parallel execution on distributed systems will also need to be modified to scale in these new shared and hybrid memory systems. In this paper, we reinvestigate parallel rendering algorithms with the goal of finding one that achieves favorable performance in this new environment. We test and analyze various methods, including sort-first, sort-last, and a hybrid scheme, to find an optimal parallel algorithm that maximizes shared memory performance.

Despite the fast evolution of Distributed Shared Memory (DSM) systems, the load balance problem has not received enough attentions. This problem arises naturally after multithreading was introduced to DSM systems few years ago. The cognizance of it would bring us a significant improvement in system performance. In this paper, we address it by proposing and experimentally evaluating a load balancing method called Dependence-Driven Load Balancing (DDLB) that is dedicated for multithreaded DSM systems. The most attractive feature of this method is to take thread dependence into account in making decisions for migration. In contrast to existing thread scheduling works which for the most part have relied on the information supplied before execution, we require no a priori information. Typically, DDLB embraces three policies, i.e. transfer policy, location policy and selection policy, and applies affinity scheduling. Finally, from the experimental results, the performance of the system with load balancing is improving evidently.

This paper presents the design and implementation of our novel home-based software Distributed Shared Memory (DSM) system. We call our system Lazy Home-based Scope Consistency (LHScC) since the system implements scope consistency lazily based on the home-based protocol. Our technique is different from others in that an optimum home node is decided before updating a home node. The performance evaluation shows that LHScC is more scalable than other DSM systems and removes the home assignment problem in the usual home-based protocol.

Software based Distributed Shared Memory (DSM) systems have been the focus of considerable research effort, primarily in improving performance and consistency protocols. Unfortunately, computer clusters present a number of challenges for any DSM systems that are not solvable through consistency protocols alone. These challenges relate to the ability of DSM systems to adjust to load fluctuations, computers being added/removed from the cluster, to deal with faults, and the ability to use DSM objects larger than the available physical memory. We present here a proposal for the Synergy Distributed Shared Memory System and its integration with the virtual memory, group communication and process migration services of the Genesis Cluster Operating System.

Coherence protocols and memory consistency models are two improtant issues in hardware coherent shared memory multiprocessors and softare distributed shared memory(DSM) systems. Over the years, many researchers have made extensive study on these two issues repectively. However, the interaction between them has not been studied in the literature. In this paper, we study the coherence protocols and memory consistency models used by hardware and software DSM systems in detail. Based on our analysis, we draw a general definition for memory consistency model, i.e., memory consistency model is the logical sum of the ordering of events in each processor and coherence protocol. We also point that in hardware DSM system the emphasis of memory consistency model is relaxing the restriction of event ordering, while in software DSM system, memory consistency model focuses mainly on relaxing coherence protocol. Taking Lazy Release Consistency(LRC) as an example, we analyze the relationship between coherence protocols and memory consistency models in software DSM systems, and find that whether the advantages of LRC can be exploited or not depends greatly on it's corresponding protocol. We draw the conclusion that the more relaxed consistency model is, the more relaxed coherence protocol needed to support it. This conclusion is very useful when we design a new consistency model. Furthermore, we make some improvements on traditional multiple writer protocol, and as far as we aware, we describe the complex state transition for multiple writer protocol for the first time. In the end, we list the main research directions for memory consistency models in hardware and software DSM systems.

Shared memory architectures have recently experienced a large increase in thread-level parallelism, leading to complex memory hierarchies with multiple cache memory levels and memory controllers. These new designs created a Non-Uniform Memory Access (NUMA) behavior, where the performance and energy consumption of memory accesses depend on the place where the data is located in the memory hierarchy. Accesses to local caches or memory controllers are generally more efficient than accesses to remote ones. A common way to improve the locality and balance of memory accesses is to determine the mapping of threads to cores and data to memory controllers based on the affinity between threads and data. Such mapping techniques can operate at different hardware and software levels, which impacts their complexity, applicability, and the resulting performance and energy consumption gains. In this article, we introduce a taxonomy to classify different mapping mechanisms and provide a comprehensive overview of existing solutions.