On the effectiveness of OpenMP teams for cluster-based many-core accelerators

Enhancements to array processors in the form of broadcast buses have been proposed for improving the speeds of algorithms in linear algebra, image processing and computational geometry. In this paper, we consider certain practical issues in such arrays which include fewer processors connected to the broadcast buses and finite time, namely log N time for broadcasting data to N processors. We propose a modified broadcast bus VLSI architecture which consists of a √N × √N mesh connected structure and a hierarchy of broadcast buses in each row and in each column such that every bus has k PE's. That is, in any row or column, in the first level there are \(\frac{{\sqrt N }}{k}\)buses with groups of k PE's connected to each bus. One PE from each of this group is connected to second level of buses in a similar manner. This is recursively done until there are only k PE's left which are connected by a broadcast bus. With this architecture, maximum, minimum, or sum of N values can be found in O(log N), median row of a binary picture in O(log N), shortest distance between two points in O(log N). This architecture is well suited for parallel processing of applications in Linear Algebra, Image Processing, Computational Geometry and Numerical Computations. These restricted connections to buses reduces the I/O ports significantly, and therefore, is more suitable for VLSI implementation.

Membrane computing, a recent branch of natural computing, has been gaining momentum attention in the last few decades due to its massive parallelism and efficient computation. Many researchers in the field of membrane computing have proposed sophisticated techniques inspired by cell biology for computer science applications, especially when they considered cell organization in tissues, organs, and most recently, from the organization of neurons. The interdisciplinary applications of membrane computing include, but not limited to computer science, biology, biomedicine, bioinformatics and several other fields such as mathematics, artificial intelligence, automation, economics, to name but a few. Their applications are pertaining to computer graphics, approximate optimization, cryptography, parallel computing and image processing. Hence, in this paper we present an up to date comprehensive literature review pertaining to the application of membrane computing in the area of digital image analysis, especially image segmentation, comprehensively and systematically. We thoroughly investigate the recent advancement in the field of image segmentation using membrane system. Furthermore, we highlight the merits and demerits of various software tools and methods. Finally, we suggest some intuitive future directions in light of the current limitations.

Sandia National Laboratories has developed massively parallel (MP) software that provides high absolute performance and nearly perfect parallel efficiencies. We have used this software on first- and second-generation 1K-processor hypercubes with more than thousand-fold parallel speedups and supercomputer performance. As we developed MP software, we made significant research advances in parallel numerical methods, such as parallel time stepping. The hardware diagnostic tools and parallel graphics algorithms that we have developed and provided to vendors can pinpoint hardware problems and visualize performance in thousand-processor ensembles. Our distributed-computing software will provide supercomputer performance from workstation networks. In collaboration with university and industry researchers in parallel computing, we have proven the concept of distributed parallel supercomputing with parallel computations on a network of VAX® computers. (VAX is a registered trademark of Digital Equipment Corporation.)

This paper derives tradeoffs between three basic costs of a parallel algorithm: synchronization, data movement, and computational cost. These tradeoffs are lower bounds on the execution time of the algorithm which are independent of the number of processors, but dependent on the problem size. Therefore, they provide lower bounds on the parallel execution time of any algorithm computed by a system composed of any number of homogeneous components, each with associated computational, communication, and synchronization payloads. We employ a theoretical model counts the amount of work and data movement as a maximum of any execution path during the parallel computation. By considering this metric, rather than the total communication volume over the whole machine, we obtain new insights into the characteristics of parallel schedules for algorithms with non-trivial dependency structures. We also present reductions from BSP and LogP algorithms to our execution model, extending our lower bounds to these two models of parallel computation. We first develop our results for general dependency graphs and hypergraphs based on their expansion properties, then we apply the theorem to a number of specific algorithms in numerical linear algebra, namely triangular substitution, Gaussian elimination, and Krylov subspace methods. Our lower bound for LU factorization demonstrates the optimality of Tiskin's LU algorithm answering an open question posed in his paper, as well as of the 2.5D LU algorithm which has analogous costs. We treat the computations in a general manner by noting that the computations share a similar dependency hypergraph structure and analyzing the communication requirements of lattice hypergraph structures.

A fundamental mathematical field with many applications in science and engineering is linear algebra. This paper investigates the various applications of linear algebra in the fields of traffic flow analysis, electric circuits, cubic spline interpolation, and genetics. This research delves into individual applications while emphasizing cross-disciplinary insights, fostering innovative solutions through the convergence of genetics, cubic spline interpolation, circuits, and traffic flow analysis. The research employs specific methodologies in each application area to demonstrate the practical integration of linear algebra in genetics, cubic spline interpolation, electric circuits, and traffic flow analysis. In genetics, linear algebra techniques are utilized to represent genetic data using matrices, analyze genotype distributions across generations, and identify genotype-phenotype associations. For cubic spline interpolation, linear algebra is employed to construct smooth interpolating curves, involving the derivation of equations for spline functions and the determination of coefficients using boundary conditions and continuity requirements. In electric circuit analysis, linear algebra is crucial for modeling circuit elements, formulating systems of linear equations based on Kirchhoff's laws, and solving for voltage and current distributions in circuits. In traffic flow analysis, linear algebra techniques are used to represent traffic movement in networks, formulate systems of linear equations representing traffic flow dynamics, and solve for traffic flow solutions to optimize transportation networks. By addressing contemporary challenges, emerging research frontiers, and future trajectories at the intersection of linear algebra and diverse domains, this study underscores the profound impact of mathematical tools in advancing understanding and resolving complex real-world problems across multiple fields.

We use the computing technology digital image processing (DIP) to enhance the teaching of linear algebra so as to make the course more visual and interesting. Certainly, this visual approach by using technology to link linear algebra to DIP is interesting and unexpected to both students as well as many faculty.

Summary form only given. In this poster, we present our experience in implementing a high performance computing cluster for teaching parallel computing theory and development of parallel applications. In teaching parallel and high performance computing, there is often a gap between potential performance taught in the lectures and those practically experienced in exercises in the laboratory. The development of the ELIHE cluster provides us with an opportunity take a hands on approach in teaching programming environments, tools, and libraries for development of parallel applications, parallel computation, architectures, message passing and shared memory paradigms using MPI and OpenMP, etc, at both undergraduate and graduate level. The ELIHE HP cluster consists of 9 computational nodes and a master node. All the nodes in the cluster are commodity systems - PCs, running commodity software - Linux, and CLIC Mandrake. Creating the ELIHE cluster has fulfilled two important goals: to design and implement a HP cluster for teaching parallel computing architectures in the School of Science and Technology, and to promote the use of high performance computer technology for research to faculty members and students

The increasing complexity of new parallel architectures has widened the gap between adaptability and efficiency of the codes. As high performance numerical libraries tend to focus more on performance, we wish to address this issue using a C++ library called NT2. By analyzing the properties of the linear algebra domain that can be extracted from numerical libraries and combining them with architectural features, we developed a generic approach to solve dense linear systems on various architectures including CPU and GPU. We have then extended our work with an example of a least squares solver based on semi-normal equations in mixed precision that cannot be found in current libraries. For the automatically generated solvers, we report performance comparisons with state-of-the-art codes, and show that it is possible to obtain a generic code with a high-level interface (similar to Matlab) which runs either on CPU or GPU without generating a significant overhead.

A new technique for proving lower bounds for parallel computation is introduced. This technique enables us to obtain, for the first time, nontrivial tight lower bounds for shared-memory models of parallel computation that allow several processors to have simultaneous access to the same memory location. Specifically, we use a concurrent-read concurrent-write model of parallel computation. It has p processors, each has access to a common memory of size m (also called communication width or width in short). The input to the problem is located in an additional read-only portion of the common memory.For a wide variety of problems (including parity, majority and summation) we show that the time complexity T (depth) and the communication width m are related by the trade-off curve $mT^2 = \Omega (n)$, (where n is the size of the input), regardless of the number of processors. Moreover, for every point on this curve with $m = O(n/\log ^2 n)$ we give a matching upper bound with the optimal number of processors.We e...

For pt.I, see Kibernetika, vol.13, no.6, p.28 (1977). Considerable effort was devoted in the surveyed period to automatic decomposition of sequential algorithms, or rather of procedures or subprograms written in the algorithmic languages ALGOL or FORTRAN. The authors do not consider this body of research, they only note that, on the one hand, the available linear algebra subprograms included in Eispack provide convenient objects for testing various approaches to automatic construction of parallel programs and, on the other, an important state in this activity is the development of methods for fast and efficient solution of linear recurrences, which reduce to solving systems of linear equations with band-triangular matrix (in particular, of sufficiently small width). This article reflects the penetration of the parallelism ideas into the computational methods of linear algebra in recent years. 74 references.

In this paper we benchmark the performance of the Cray T3D, IBM 9076 SP/1 and Intel Paragon XP/S parallel computers, using implementations of parallel algorithms for the computation of the vector outer-product A = uvT operation. The vector outer-product operation, although very simple in nature, requires the computation of a large number of floating-point operations and its parallelization induces a great level of communication between the processors. It is thus suited to measure the relative speed of the processor, memory subsystem and network capabilities of a parallel computer. It should not be considered a ‘toy problem’, since it arises in numerical methods in the context of the solution of systems of non-linear equations – still a difficult problem to solve. We present algorithms for both the explicit shared-memory and message-passing programming models together with theoretical computation models for those algorithms. Actual experiments were run on those computers, using Fortran 77 implementations of the algorithms. The results obtained with these experiments show that due to the high degree of communication between the processors one needs a parallel computer with fast communications and carefully implemented data exchange routines. The theoretical computation model allows prediction of the speed-up to be obtained for some problem size on a given number of processors. © 1997 John Wiley & Sons, Ltd.

Grid technologies have emerged as an important area in distributed and parallel computing. An increasing amount of users among scientific communities are using grid facilities to share, manage and ...

Processing-In-Memory (PIM) is an increasingly popular architecture aimed at addressing the 'memory wall' crisis by prioritizing the integration of processors within DRAM. It promotes low data access latency, high bandwidth, massive parallelism, and low power consumption. The skyline operator is a known primitive used to identify those multi-dimensional points offering optimal trade-offs within a given dataset. For large multidimensional dataset, calculating the skyline is extensively compute and data intensive. Although, PIM systems present opportunities to mitigate this cost, their execution model relies on all processors operating in isolation with minimal data exchange. This prohibits direct application of known skyline optimizations which are inherently sequential, creating dependencies and large intermediate results that limit the maximum parallelism, throughput, and require an expensive merging phase. In this work, we address these challenges by introducing the first skyline algorithm for PIM architectures, called DSky. It is designed to be massively parallel and throughput efficient by leveraging a novel work assignment strategy that emphasizes load balancing. Our experiments demonstrate that it outperforms the state-of-the-art algorithms for CPUs and GPUs, in most cases. DSky achieves 2× to 14× higher throughput compared to the state-of-the-art solutions on competing CPU and GPU architectures. Furthermore, we showcase DSky's good scaling properties which are intertwined with PIM's ability to allocate resources with minimal added cost. In addition, we showcase an order of magnitude better energy consumption compared to CPUs and GPUs.

Current generation GPUs can accelerate high-performance, compute-intensive applications by exploiting massive thread-level parallelism. The high performance, however, comes at the cost of increased power consumption. Recently, commercial GPGPU architectures have introduced support for concurrent kernel execution to better utilize the computational/memory resources and thereby improve overall throughput. In this paper, we argue and experimentally validate the benefits of concurrent kernels towards energy-efficient execution. We design power-performance models to carefully select the appropriate kernel combinations to be executed concurrently, the relative contributions of the kernels to the thread mix, along with the frequency choices for the cores and the memory to achieve high performance per watt metric. Our experimental evaluation shows that the concurrent kernel execution in combination with DVFS can improve energy-efficiency by up to 34.5% compared to the most energy-efficient sequential execution.

Thanks to their massive computational power and their SIMT computational model, Graphics Processing Units (GPUs) have been successfully used to accelerate a wide variety of regular applications (linear algebra, stencil computations, image processing and bioinformatics algorithms, among others). However, many established and emerging problems are based on irregular data structures, such as graphs. Examples can be drawn from different application domains: networking, social networking, machine learning, electrical circuit modeling, discrete event simulation, compilers, and computational sciences. It has been shown that irregular applications based on large graphs do exhibit runtime parallelism; moreover, the amount of available parallelism tends to increase with the size of the datasets. In this work, we explore an implementation space for deploying a variety of graph algorithms on GPUs. We show that the dynamic nature of the parallelism that can be extracted from graph algorithms makes it impossible to find an optimal solution. We propose a runtime system able to dynamically transition between different implementations with minimal overhead, and investigate heuristic decisions applicable across algorithms and datasets. Our evaluation is performed on two graph algorithms: breadth-first search and single-source shortest paths. We believe that our proposed mechanisms can be extended and applied to other graph algorithms that exhibit similar computational patterns.