Performance of MPI Parallel Applications

Grids provide infrastructure for intensive computations and storage of shared large scale database across a distributed environment. Although grid technologies enable the sharing and utilization of widespread resources, the performance of parallel applications on the Grid is sensitive to the effectiveness of the scheduling algorithms used. Scheduling is the decision process by which application components are assigned to available resources to optimize various performance metrics. In this paper we discuss how economy and quality of service plays important role in scheduling resources. Various Grid scheduling algorithms are discussed from different points of view, such as static vs. dynamic policies, objective functions, Quality of service constraints, strategies dealing with dynamic behavior of resources and so on. After discussing existing algorithms a new Failure rate-cost and time optimization algorithm is been proposed which will guarantee quality of service.

We evaluate the impact of programming language features on the performance of parallel applications on modern parallel architectures, particularly for the demanding case of sparse integer codes. We compare a number of programming languages (Pthreads, OpenMP, MPI, UPC) on both shared and distributed-memory architectures. We find that language features can make parallel programs easier to write, but cannot hide the underlying communication costs for the target parallel architecture. Powerful compiler analysis and optimization can help reduce software overhead, but features such as fine-grain remote accesses are inherently expensive on clusters. To avoid large reductions in performance, language features must avoid degrading the performance of local computations.

Cloud computing and big data: Technologies and applications

Parallel machines are typically space shared, or time shared such that only one application executes on a group of nodes at any given time. It is generally assumed that executing multiple parallel applications simultaneously on a group of independently scheduled nodes is not efficient because of synchronization requirements. The central contribution of this paper is to demonstrate that performance of parallel applications with sharing is typically competitive for independent and coordinated (gang) scheduling on small compute clusters. There is a modest overhead due to uncoordinated scheduling but it is often compensated by better sharing of resources. The impact of sharing was studied for different numbers of nodes and threads and different memory and CPU requirements of competing applications. The significance of the CPU time slice, a key parameter in CPU scheduling, was also studied. Application characteristics and operating system scheduling policies are identified as the main factors that influence performance with node sharing. All experiments are performed with NAS benchmarks on a Linux cluster. The significance of this research is that it provides evidence to support flexible and decentralized scheduling and resource selection policies for cluster and grid environments.

InfiniBand continues to become more and more important in High Performance Computing world. This talk discusses the impact of DDR (i.e., 20 GigE) switches and HCA's on the creation of low latency, high bandwidth InfiniBand fabrics. When combined with low latency HCA's, such as those from Qlogic, the fabrics discussed can make as much as a 40% reduction in fabric latency, improving the performance of fine grain parallel applications. They also make it possible to create 3 hop low latency fabrics that provide excellent performance that can be used with clusters that have as many as 1009 nodes.There are two approaches to using DDR fabrics to improving latency. The first uses a combination of one, two and three hop fabrics and uses what we call a FasTree topology to create fabrics which can be used to create small clusters (32 to 96 nodes). FasTree's not only have lower latency, but require fewer switch components than fat trees. This does not mean that they have smaller bi-sectional bandwidths, as their links run at twice the speed of an SDR fabric. One of the features of a FasTree, that distinguishes it from other fabrics, is that it does not contain spines. The second fabric, which we call a ThinTree, uses complex single hop spines to link together different fabric sub-domains. Any node in a ThinTree is at most 3 hops away from any other node. There are, however, some compromises required to make it possible to link together up to 1008 nodes without exceeding 3 hops. These compromises result in sub-domains whose intra-domain bandwidth is full CBB while their inter-domain bandwidth typically runs around 40% of CBB. However, because of the 1.8 GB/sec bandwidth of the DDR fabrics, all the connections between any two nodes in a ThinTree fabric are adequate for virtually any HPC application and most others as well. The characteristics of both these topologies are discussed in the talk.

Computers of a non-dedicated cluster are often idle (users attend meetings, have lunch or coffee breaks) or lightly loaded (users carry out simple computations). These underutilized computers can be employed to execute parallel applications not only during weekends and at nights but also during office hours. Thus, they have to be shared by parallel and sequential applications which could lead to the improvement of their execution performance. However, there is a lack of experimental study showing the behavior and performance of parallel and sequential applications executing concurrently on clusters. We present here the result of an experimental study into load balancing based scheduling of a mixture of parallel and sequential applications on a non-dedicated cluster.

With the prevalence of multicore processors in computer systems, many soft real-time applications, such as media-based ones, use parallel programming models to utilize hardware resources better and possibly shorten response time. Meanwhile, virtualization technology is widely used in cloud data centers. More and more cloud services including such parallel soft real-time applications are running in virtualized environment. However, current hyper visors do not provide adequate support for them because of soft real-time constraints and synchronization problems, which result in frequent deadline misses and serious performance degradation. CPU schedulers in underlying hyper visors are central to these issues. In this paper, we identify and analyze CPU scheduling problems in hyper visors, and propose a novel scheduling algorithm considering both soft real-time constraints and synchronization problems. In our proposed method, real-time priority is introduced to accelerate event processing of parallel soft real-time applications, and dynamic time slice is used to schedule virtual CPUs. Besides, all runnable virtual CPUs of virtual machines running parallel soft real-time applications are scheduled simultaneously to address synchronization problems. We implement a parallel soft real-time scheduler, named Poris, based on Xen. Our evaluation shows Poris can significantly improve the performance of parallel soft real-time applications. For example, compared to the Credit scheduler, Poris improves the performance of media player by up to a factor of 1.35, and shortens the execution time of PARSEC benchmark by up to 44.12%.

Performance of parallel scientific applications on many-core processor architectures is a challenge that increases every day, especially when energy efficiency is concerned. To achieve this, it is necessary to explore architectures with high processing power composed by a network-on-chip to integrate many processing cores and other components. In this context, this paper presents a design space exploration over NoC-based manycore processor architectures with distributed and shared caches, using full-system simulations. We evaluate bottlenecks in such architectures with regard to energy efficiency, using different parallel scientific applications and considering aspects from caches and NoCs jointly. Five applications from NAS Parallel Benchmarks were executed over the proposed architectures, which vary in number of cores; in L2 cache size; and in 12 types of NoC topologies. A clustered topology was set up, in which we obtain performance gains up to 30.56% and reduction in energy consumption up to 38.53%, when compared to a traditional one.

Several manufacturers have already started to commercialize near-bank Processing-In-Memory (PIM) architectures, after decades of research efforts. Near-bank PIM architectures place simple cores close to DRAM banks. Recent research demonstrates that they can yield significant performance and energy improvements in parallel applications by alleviating data access costs. Real PIM systems can provide high levels of parallelism, large aggregate memory bandwidth and low memory access latency, thereby being a good fit to accelerate the Sparse Matrix Vector Multiplication (SpMV) kernel. SpMV has been characterized as one of the most significant and thoroughly studied scientific computation kernels. It is primarily a memory-bound kernel with intensive memory accesses due its algorithmic nature, the compressed matrix format used, and the sparsity patterns of the input matrices given. This paper provides the first comprehensive analysis of SpMV on a real-world PIM architecture, and presents SparseP, the first SpMV library for real PIM architectures. We make two key contributions. First, we design efficient SpMV algorithms to accelerate the SpMV kernel in current and future PIM systems, while covering a wide variety of sparse matrices with diverse sparsity patterns. Second, we provide the first comprehensive analysis of SpMV on a real PIM architecture. Specifically, we conduct our rigorous experimental analysis of SpMV kernels in the UPMEM PIM system, the first publicly-available real-world PIM architecture. Our extensive evaluation provides new insights and recommendations for software designers and hardware architects to efficiently accelerate the SpMV kernel on real PIM systems. For more information about our thorough characterization on the SpMV PIM execution, results, insights and the open-source SparseP software package [21], we refer the reader to the full version of the paper [3, 4]. The SparseP software package is publicly and freely available at https://github.com/CMU-SAFARI/SparseP.

Notice of Violation of IEEE Publication Principles<br><br> "Communication Generation for Irregular Parallel Applications,"<br>by Changjun Hu, Jing Li, Jue Wang, Yonghong Li, Liang Ding, and Jianjiang Li,<br>in the Proceedings of the International Symposium on Parallel Computing in Electrical Engineering, 2006 (PAR ELEC 2006) pp.263-270<br><br> After careful and considered review of the content and authorship of this paper by a duly constituted expert committee, this paper has been found to be in violation of IEEE's Publication Principles.<br><br>This paper contains portions of original text from the papers cited below. The original text was copied without appropriate attribution.<br><br> "Communication Generation for Aligned and Cyclic (K) Distributions Using Integer Lattice," <br>by Eric Hung-Yu Tseng and Jean-Luc Gaudiot<br>in IEEE Transactions on Parallel and Distributed Systems, 10(2), February 1999 <br/> Irregular computing significantly influences the performance of large scale parallel applications. How to generate local memory access sequence and communication set efficiently for irregular parallel application is an important issue in compiling a data parallel language into a single program multiple data (SPMD) code for distributed-memory machines. In this paper, we propose a hybrid approach, which combines the advantages of the algebra method and the integer lattice method. Our approach derives an algebraic solution of communication set enumeration at compile time for the situation of irregular array references in nested loops. Based on the integer lattice, we develop our method for global-to-local and local-to-global index translations in the case of alignment and cyclic (k) distribution. Then, we present our algorithm for the corresponding SPMD code generation, which adopts some communication optimization techniques. In our method, when parameters are known, the communication set generation, the global-to-local and local-to-global index translations, and the SPMD code generation can be completed at compile time without inspector phase of run time

Currently there exists little support to specify and control the execution of a large number of experiments required for performance oriented development of scientific applications. We have developed a new directive based language called ZEN for the specification of wide sets of synthetic experiments in the context of large scale parameter and performance studies of parallel applications. We report experimental results for the performance analysis of an ocean simulation application and for the parameter study of a computational finance code.

Global Computing platforms, large scale clusters and future TeraGRID systems gather thousands of nodes for computing parallel scientific applications. At this scale, node failures or disconnections are frequent events. This Volatility reduces the MTBF of the whole system in the range of hours or minutes. We present MPICH-V, an automatic Volatility tolerant MPI environment based on uncoordinated checkpoint/roll-back and distributed message logging. MPICH-V architecture relies on Channel Memories, Checkpoint servers and theoretically proven protocols to execute existing or new, SPMD and Master-Worker MPI applications on volatile nodes. To evaluate its capabilities, we run MPICH-V within a framework for which the number of nodes, Channels Memories and Checkpoint Servers can be completely configured as well as the node Volatility. We present a detailed performance evaluation of every component of MPICH-V and its global performance for non-trivial parallel applications. Experimental results demonstrate good scalability and high tolerance to node volatility.

Performance Evaluation of Adaptive Scientific Applications using TAU

Network-on-Chips (NoCs) are important components for scalable many-core processors. Because the performance of parallel applications is usually sensitive to the latency of NoCs, reducing it is a primary requirement. In this study, a compression router that hides the (de)compression-operation delay is proposed. The compression router (de)compresses the contents of the incoming packet before the switch arbitration is completed, thus shortening the packet length without latency penalty and reducing the network injection-and-ejection latency. Evaluation results show that the compression router improves up to 33% of the parallel application performance (conjugate gradients (CG), fast Fourier transform (FT), integer sort (IS), and traveling salesman problem (TSP)) and 63% of the effective network throughput by 1.8 compression ratio on NoC. The cost is an increase in router area and its energy consumption by 0.22mm2 and 1.6 times compared to the conventional virtual-channel router. Another finding is that off-loading the decompressor onto a network interface decreases the compression-router area by 57% at the expense of the moderate increase in communication latency.

The word-invalidate cache coherence protocol