Operator design pattern for data parallel computation

Recent application studies have shown that many irregular applications have a generalized data parallelism that manifests itself as iterative computations over worklists of different kinds. In general, there are complex dependencies between iterations. These dependencies cannot be elucidated statically because they depend on the inputs to the program; thus, optimistic parallel execution is the only tractable approach to parallelizing these applications.We have built a system called Galois that supports this style of parallel execution. Its main features are (i) set iterators for expressing worklist-based data parallelism, and (ii) a runtime system that performs optimistic parallelization of these iterators, detecting conflicts and rolling back computations as needed.Our work builds on the Galois system, and it addresses the problem of scheduling iterations of set iterators on multiple cores. The policy used by the base Galois system is to assign an iteration to a core whenever it needs work to do, but we show in this paper that this policy is not optimal for many applications. We also argue that OpenMP-style DO-ALL loop scheduling directives such as chunked and guided self-scheduling are too simplistic for irregular programs. These difficulties led us to develop a general scheduling framework for irregular problems; OpenMP-style scheduling strategies are special cases of this general approach. We also provide hooks into our framework, allowing the programmer to leverage application knowledge to further tune a schedule for a particular application.To evaluate this framework, we implemented it as an extension of the Galois system. We then tested the system using five real-world, irregular, data-parallel applications. Our results show that (i) the optimal scheduling policy can be different for different applications and often leverages application-specific knowledge and (ii) implementing these schedules in the Galois system is relatively straightforward.

Logic synthesis

Due to the complexity of modern data parallel applications such as image processing applications, automatic approach to infer suitable and efficient hardware realizations are more and more required. Typically, the optimization of data transfer and storage micro-architecture has a key role for the data parallelism. In this paper, we propose a comprehensive method to explore the mapping of a high-level representation of an application into a customizable hardware accelerator. The high-level representation is in a language called Array-OL. The customizable architecture uses FIFO queues and double buffering mechanism to mask the latency of data transfers and external memory access. The mapping of a high-level representation onto the given architecture is performed by applying a set of loop transformations in Array-OL. A method based on integer partition is used to reduce the space of explored solutions.Keywordsdesign space explorationdata parallel applicationsimage processingArray-OLhardware architecturedata management

A multicomputer is a loosely coupled multiprocessor architecture, which is also called a cluster computer or just a cluster. It may be assembled from commercially available PCs and networking hardware as a low-cost alternative to rack mounted blades, interconnected through a high speed backplane, or to a high performance, tightly coupled, shared memory multiprocessor. Parallel software applications exploit parallelism in a cluster by distributing chunks of data among its remote nodes for simultaneous processing.Each remote node processes its chunk of data in parallel with the others and returns its partial result back to the master node for combination into a final result. When the chunk size, or system granularity, is just right, there is a balance between the network-transit time and the remote-processing time of the data; the distributed application runs at maximum efficiency and the system granularity is said to be optimal. That said, one wonders why there are no tools explicitly designed to help parallel software designers and performance analysts determine the optimal granularity size of a given cluster!We present a back-to-basics approach, application-level program named McGAT, for Multi-computer Granularity Assessment Tool. The user specifies a range of chunk sizes to analyze, along with a remote node's processing time estimate for a particular sized chunk. McGAT is not a simulation, but a parallel application that runs on the user's hardware, distributing chunks of data to each of the cluster's remote nodes. Each node responds after an amount of time necessary to process that data, based on the user's processing time estimates. McGAT reports system throughput for each chunk size over the user-specified range.The chunk size with the maximum throughput indicates the optimal granularity of the system on which it is run, which is a key parallel software design component. The user may then choose a chunk size that provides maximum throughput within the requirements of his or her specific application.Our presentation includes graphs comparing the results from runs on a 3-, 4-, and 5-node Linux-based PC cluster, interconnected through both 10 and 100 Mbps Ethernet. We plan to expand our knowledge base to include larger numbers of nodes, and a virtual ring to complement or virtual star communications protocol.McGAT may be used to establish a performance baseline, and again later to quantify the effect of any subsequent changes to the cluster. McGAT is offered as open source software, and a beta release for Linux based platforms is available for download from the DSRLab website under the GNU General Public License.

Architectural fabrics are an integral part of fabric architecture, a term used to describe tensile fabric membranes – also known as fabric structures or textile buildings. This chapter discusses architectural fabrics in the context of their physical properties, durability, aesthetics and suitability for a given application. The account is preceded by a brief background to fabric structures and related design issues. Images of samples of different types of architectural fabrics, their descriptions and examples of application in real projects are used to clarify the subject. There are numerous factors affecting the choice of architectural fabric and these include a manufacturing design stage, known as patterning – the creation of a tensile surface in three-dimensional space out of a series of two-dimensional, unstrained pieces of fabric. Patterning is presented in a concise manner, sufficient to highlight design issues, together with problems of choice of fabric and behaviour in a full-scale tension structure. The chapter ends with a brief discussion of factors influencing the choice of architectural fabric, future research trends and development of new materials.

Dedicated clusters are becoming commonly used for high performance parallel processing. Computers of a non-dedicated cluster are often idle or lightly loaded. These under utilised computers can be employed to execute parallel applications. Thus, they have to be shared by parallel and sequential applications, which could lead to the improvement of their execution performance. There is a lack of experimental study showing the behaviour and performance of executing parallel and sequential applications concurrently on a non-dedicated cluster. We present the result of an experimental study into load balancing of a mixture of parallel and sequential applications on a non-dedicated cluster.

Computers of a non-dedicated cluster are often idle (users attend meetings, have lunch or coffee breaks) or lightly loaded (users carry out simple computations). These underutilized computers can be employed to execute parallel applications not only during weekends and at nights but also during office hours. Thus, they have to be shared by parallel and sequential applications which could lead to the improvement of their execution performance. However, there is a lack of experimental study showing the behavior and performance of parallel and sequential applications executing concurrently on clusters. We present here the result of an experimental study into load balancing based scheduling of a mixture of parallel and sequential applications on a non-dedicated cluster.

The distributed file system, HDFS, is widely deployed as the bedrock for many parallel big data analysis. However, when running multiple parallel applications over the shared file system, the data requests from different processes/executors will unfortunately be served in a surprisingly imbalanced fashion on the distributed storage servers. These imbalanced access patterns among storage nodes are caused because a). unlike conventional parallel file system using striping policies to evenly distribute data among storage nodes, data-intensive file system such as HDFS store each data unit, referred to as chunk file, with several copies based on a relative random policy, which can result in an uneven data distribution among storage nodes; b). based on the data retrieval policy in HDFS, the more data a storage node contains, the higher probability the storage node could be selected to serve the data. Therefore, on the nodes serving multiple chunk files, the data requests from different processes/executors will compete for shared resources such as hard disk head and networkbandwidth, resulting in a degraded I/O performance. In this paper, we first conduct a complete analysis on how remote and imbalanced read/write patterns occur and how they are affected by the size of the cluster. We then propose novel methods, referred to as Opass, to optimize parallel data reads, as well as to reduce the imbalance of parallel writes on distributed file systems. Our proposed methods can benefit parallel data-intensive analysis with various parallel data access strategies. Opass adopts new matching-based algorithms to match processes to data so as to compute the maximum degree of data locality and balanced data access. Furthermore, to reduce the imbalance of parallel writes, Opass employs a heatmap for monitoring the I/O statuses of storage nodes and performs HM-LRU policy to select a local optimal storage node for serving write requests. Experiments are conducted on PRObE's Marmot 128-node cluster testbed and the results from both benchmark and well-known parallel applications show the performance benefits and scalability of Opass.

This paper presents HyperFlow: an approach to workflow programming which combines the advantages of a declarative workflow description and low-level scripting programming. The workflow execution model of HyperFlow is based on a formal model of computation - Process Networks. The execution environment is implemented on the basis of a widely adopted runtime platform node.js. Workflow programming benefits from such an approach in multiple ways, including leveraging a large programming ecosystem with many developers, reusable software packages and learning resources; elimination of shim nodes from the workflow graph; and increased reusability of workflow processing components. The HyperFlow workflow programming approach and its advanced capabilities are presented. The HyperFlow engine is also briefly described. Four example workflow applications from various domains, including flood threat assessment, are studied as a demonstration of the HyperFlow programming approach and a comparison with related solutions.

A message passing program composition methodology, called Ensemble, applied for Parix is presented. Ensemble overcomes the implementation problems and complexities in developing applications in message passing environments. Parallel applications are virtually specified by Process Communication Graphs (PCGs) annotated with communication information for Parix processes. Annotated PCGs are generated from application scripts by supporting tools. Reusable Parix executable components are defined from which all processes are created. A universal Parix program loader interprets the annotated PCGs creating the application processes from the reusable components and establishing their communication dependencies. Ensemble is applied to compose variations of Parix applications using the same reusable components. The methodology has been applied for PVM.

In multiprocessors, performance improvement is typically achieved by exploring parallelism with fixed granularities, such as instruction-level, task-level, or data-level parallelism. We introduce a new reconfiguration mechanism that facilitates variations in these granularities in order to optimize resource utilization in addition to performance improvements. Our reconfigurable multiprocessor QuadroCore combines the advantages of reconfigurability and parallel processing. In this article, a unified hardware-software approach for the design of our QuadroCore is presented. This design flow is enabled via compiler-driven reconfiguration which matches application-specific characteristics to a fixed set of architectural variations. A special reconfiguration mechanism has been developed that alters the architecture within a single clock cycle. The QuadroCore has been implemented on Xilinx XC2V6000 for functional validation and on UMC’s 90nm standard cell technology for performance estimation. A diverse set of applications have been mapped onto the reconfigurable multiprocessor to meet orthogonal performance characteristics in terms of time and power. Speedup measurements show a 2--11 times performance increase in comparison to a single processor. Additionally, the reconfiguration scheme has been applied to save power in data-parallel applications. Gate-level simulations have been performed to measure the power-performance trade-offs for two computationally complex applications. The power reports confirm that introducing this scheme of reconfiguration results in power savings in the range of 15--24%.

Data partitioning algorithms aiming to minimize the execution time and the energy of computations in self-adaptable data-parallel applications on modern extreme-scale multicore platforms must address two critical challenges. First, they must take into account the new complexities inherent in these platforms such as severe resource contention and non-uniform memory access. Second, they must have low practical runtime and memory costs. The sequential data partitioning algorithms addressing the first challenge have a theoretical time complexity of O( $m * m * p * p$ ) where $m$ is the number of points in the discrete speed/energy function and $p$ is the number of available processors. They, however, exhibit high practical runtime cost and excessive memory footprint, therefore, rendering them impracticable for employment in self-adaptable applications executing on extreme-scale multicore platforms. We present, in this paper, the parallel data partitioning algorithms that address both the challenges. They take as input the functional models of performance and energy consumption against problem size and output workload distributions, which are globally optimal solutions. They have a low time complexity of O( $m * m * p$ ) thereby providing a linear speedup of O( $p$ ) and low memory complexity of O( $n$ ) where $n$ is the workload size expressed as a multiple of granularity. They employ dynamic programming approach, which also facilitates the easier integration of performance and energy models of communications. We experimentally study the practical cost of application of our algorithms in two data-parallel applications, matrix multiplication and fast Fourier transform, on a cluster in Grid’5000 platform. We demonstrate that their practical runtime and memory costs are low making them ideal for employment in self-adaptable applications. We also show that the parallel algorithms exhibit tremendous speedups over the sequential algorithms. Finally, using theoretical analysis for a forecast exascale platform, we demonstrate that the parallel algorithms have negligible execution times compared to the matrix multiplication application executing on the platform.

Increased platform heterogeneity and varying resource availability in distributed systems motivate the design of i>resource-aware applications, which ensure a desired performance level by continuously adapting their behavior to changing resource characteristics. In this paper, we describe an application-independent adaptation framework that simplifies the design of resource-aware applications. This framework eliminates the need for adaptation decisions to be explicitly programmed into the application by relying on two novel components: (1) a i>tunability interface, which exposes adaptation choices in the form of alternate application configurations while encapsulating core application functionality; and (2) a i>virtual execution environment, which emulates application execution under diverse resource availability enabling off-line collection of information about resulting behavior. Together, these components permit automatic run-time decisions on i>when to adapt by continuously monitoring resource conditions and application progress, and i>how to adapt by dynamically choosing an application configuration most appropriate for the prescribed user preference. We evaluate the framework using an interactive distributed image visualization application and a parallel image processing application. The framework permits automatic adaptation to changes in execution environment characteristics such as available network bandwidth or data arrival pattern by choosing a different application configuration that satisfies user preferences of output quality and timeliness.

Speeding up the communications on a cluster using MPI by means of Software Defined Networks

This article introduces a new workflow specification model for parallel applications. Many parallel systems allow only the following basic flow control: (1) a coordinator task distributes the data among the worker tasks, (2) the worker tasks process the data in parallel and (3) the coordinator task gather all the results. In some systems the application execution flow is based on a dependence relationship among tasks, representing a directed acyclic graph (DAG). Even with this model, it is not possible to execute some important parallel applications and, therefore, there is a need for a new specification model with more sophisticated flow controls that allow, for example, conditional branching and iterations at the level of task management. The purpose of this article is to present a proposal for a new parallel application specification model that provides new types of control structures and allows the implementation of a broader range of applications.