The use of interpreted languages for implementing parallel algorithms on distributed systems

The 25th HIPS workshop, a full-day meeting on May 18th at the IEEE IPDPS 2020 conference in New Orleans (now virtual), focuses on high-level programming of multiprocessors, compute clusters, and massively parallel machines. Like previous workshops in the series, which was established in 1996, this event serves as a forum for research in the areas of parallel applications, language design, compilers, runtime systems, and programming tools. It provides a timely forum for scientists and engineers to present the latest ideas and findings in these rapidly changing fields. In our call for papers, we especially encouraged innovative approaches in the areas of emerging programming models for large-scale parallel systems and many-core architectures.

Effective programming of parallel architectures has always been a challenge, and it is especially complicated with their modern diversity. The task of automatic parallelization of program code was formulated from the moment of the appearance of the first parallel computers made in Russia (for example, PS2000). To date, programming languages and technologies have been developed that simplify the work of a programmer (T-System, MC#, Erlang, Go, OpenCL), but do not make parallelization automatic. The current situation requires the development of effective programming tools for parallel computing systems. Such tools should support the development of parallel programs for systems with shared and distributed memory. The paper deals with the problem of automatic parallelization of affine programs for such systems. Methods for calculating space-time mappings that optimize the locality of the program are discussed. The implementation of developed methods is done in Haskell within the source-to-source translator performing automatic parallelization. A comparison of the performance of parallel variants of lu, atax, syr2k programs obtained using the developed tool and the modern Pluto tool is made. The experiments were performed on two x86_64 machines connected by the InfiniBand network. OpenMP and MPI were used as parallelization technologies. The performance of the resulting parallel program indicates the practical applicability of the developed tool for affine programs parallelization.

Objective: Due to the recent development of multiple parallel programming tools with varying features, it is difficult to choose the best tool according to the needs of the user. Methods: This problem is addressed by making a comparative analysis study of different features like license type, source code availability, targeted platforms and languages supportedby these diverse tools. There are different parallel programming languages that support the present multi-core architecture like Message Passing Interface (MPI) and Open Multi-Processing (OpenMP). These are widely used to provide different performance characteristics of parallelism in different test cases. The new architecture and the complexity strengthens the need to monitor and analyze the performance of the various OpenMP kernels and constructs on multi-core processors. Findings: There are many papers that have been published in the past but non of them focuses on a comparative study among the performance analysis tools (PATs) that we mostly opt for. This paper intends to analyze the parallel computing ability of OpenMP and MPI, besides helping the user to understand which tool suites his task the best. Improvement: This study shows that MPI offers the best performance characteristics in the field of shared memory programming whereas OpenMP is a better choice because of the global style of the resulting program. It also provides a roadmap to select the best tool when designing a parallel programming system.

Programmers of sequential and parallel computers share the objective of developing fast, reliable programs as efficiently as possible. Already, considerable effort has been made to develop tools and techniques for sequential computers. Several approaches have been devised to meet the parallel programming challenge. Sequential languages have been adapted for parallel computers; new parallel languages designed; and tools developed which aid the programmer in detecting parallel code. A parallel programming assistant can aid the user in several ways: it can carry out tasks that are simple but mundane to the user, it can make comprehension easier for the user by summarizing program information; and it can cooperate with the user in doing a complex task, thus combining its own knowledge with that of the user's.

Understanding synchronization is important for a parallel programming tool that uses dependence analysis as the basis for advising programmers on the correctness of parallel constructs. This paper discusses static analysis methods that can be applied to parallel programs with event variable synchronization. The objective is to be able to predict potential data races in a parallel program. The focus is on how dependencies and synchronization statements inside loops can be used to analyze complete programs with parallel loop and parallel case style parallelism.

The ParaScope project is building an integrated collection of tools to help scientific programmers develop correct and efficient parallel programs. The centerpiece of this collection is the ParaScope Editor, an intelligent interactive editor for parallel FORTRAN programs. The ParaScope Editor displays data dependencies, which correspond to potential data races among the iterations of a parallel loop, to assist the user in determining the correctness of a proposed parallelization. In addition, it uses dependencies to support a variety of program transformations selectable by the programmer. The eventual goal for the ParaScope Editor is to support arbitrary editing changes by performing full incremental data dependence analysis in response to program changes. In addition, it will understand and recognize when synchronization correctly prevents race conditions. The ParaScope Editor is a new kind of program construction tool; one that not only manages text, but also presents the user with insights into the semantic structure of the program being constructed.

Mainly designed for simulating the fluid motion, Smooth Particle Hydrodynamics (SPH) SPH has been widely applied to many fields, such as aerospace aircraft design and ship design. First, this paper presents an implementation of serial SPH method algorithm using the reflection method as the boundary condition. This paper describes entire process of SPH and gives detailed descriptions of some important procedures. Second, this paper realizes the corresponding parallel algorithm of SPH, which includes neighbor particle searching and building, physical attributes computation and other main SPH method procedures. Third, details of mapping different parts of the SPH serial algorithm to parallel algorithm on GPU with CUDA C as parallel programming language are also shown in this paper. Taking advantage of powerful computation ability of GPU, the experiment approximately achieves a 8X speedup that shows the high efficiency of GPU-based programming. The parallel algorithm not only achieves a high performance, but also promises a reliability result with a small error rate as the experiment result shows.

FuzzyCLIPS is a rule-based language designed especially for developing fuzzy expert systems. A FuzzyCLIPS programmer has no need to write an algorithm to solve a problem. Instead, he only needs to list the rules for dealing with various conditions. However, due to the characteristics of rule-based languages, the execution of an expert system is always more time-consuming than any one of the conventional algorithmic languages. To cope with this problem, we propose to execute a FuzzyCLIPS application in parallel on the emerging heterogeneous cluster system. Furthermore, to maximize the speedup of the parallel execution and to minimize the burden of programmers, we have implemented built-in selfscheduling schemes in the FuzzyCLIPS interpreter for better load balancing. Programmers only need to use our proposed directives to specify where the parallelisms are, no explicit and complicated send and receive routines have to be invoked in their parallel programs. According to the specified directives, the master process will automatically assign the tasks and transmit the required data to slave processes by calling MPI routines. Experimental results show that the built-in load balancing schemes can improve the system performance significantly.

On Grainless Footprint Semantics for Shared-memory Programs

A mixed method combining formal and auto-tuning approaches and aimed at maximizing efficiency of parallel programs (in terms of execution time) is proposed. The formal approach is based on algorithmic algebra and the usage of tools for automated design and synthesis of programs based on high-level algorithm specifications (schemes). Parallel software auto-tuning is the method of adjusting some structural parameters of a program to a target hardware platform to speed-up computation as much as possible. Previously, we have developed a framework intended to automate the generation of an auto-tuner from a program source code. However, auto-tuning for complex and nontrivial parallel systems is usually time-consuming due to empirical evaluation of huge amount of parameter values combinations of an initial parallel program in a target environment. In this paper, we extend our approach with statistical modeling and neural network algorithms that allow to reduce significantly the space of possible parameter combinations. The improvement consists in automatic training of a neural network model on results of “traditional” tuning cycles and the subsequent replacement of some auto-tuner calls with an evaluation from the statistical model. The method allows, particularly, transferring knowledge about the influence of parameters on program performance between “similar” (in terms of hardware architecture) computing environments for the same applications. The idea is to reuse a model trained on data from a similar environment. The use of the method is illustrated by an example of tuning a parallel sorting program which combines several sorting methods.

Due to the culture of sequential programming, the lack of tools and the inherent difficulties of parallel programming, programmers find hard to design and evaluate parallel real-time programs. As a result, a major problem found in the development of parallel real-time systems is the difficulty to predict the performance of the application, and the need to reduce rapid prototypes of the systems behind schedule. This work is part of a bigger project financed by FINEP, with the title Arquitetura Paralela Usando DSPs (Parallel Architecture Using DSPs), whose aim is to use a parallel system for processing and visualising tomography images from the solar atmosphere in real time. In this way, it can be possible to have the 3D reconstruction of the solar corona from the angular tomography, as well as the spectral lines. This paper presents a Visual Environment which should be able to provide continuity in the development of projects using the most common development methods, traditional or object-oriented, by offering support during the phases of implementation, debugging and testing. The environment has a visual interface and the main aim is to facilitate the development process and debugging of source code of applications developed for parallel kernels, particularly for Virtuoso (Virtual Single Processor Programming System – Virtuoso). The environment is composed by four tools: a Parallel Programs Generator, a Worst Case Execution Time Analyser, a Scheduling Analyser, and a Parallel Debugger.

Recursive function theory studies functions that can be evaluated by an algorithm based on some computing model such as a Turing machine or computer program. These functions are called partial recursive (p.r.) and are defined on subsets of N, where N = {0, 1,... }, with values in N. (A good introduction to the theory may be found in [2].) This paper explains the Parameter Theorem, illustrates some unusual computer programs, and explains their connection to recursive function theory. Basing our computing model on a modern high level language, we can construct programs whose existence is known through more formal means. Our functions will operate on strings rather than numbers, however. There is no loss in generality since strings can be viewed as members of N, and vice-versa. From a practical aspect, this approach is somewhat more desirable since conventional programming languages often allow making use of long strings, but place more restrictive limits on the size of integers. We first pose a problem: Does there exist a program (written in some conventional programming language) which halts if and only if its input is identical to the program's text (i.e., source code)? For the remainder of this note X is the set of ASCII characters 'a,' 'b, +,' space, line feed, and so forth, needed to write programs in most programming languages. We denote by X * the set of all strings (finite sequences) over X. Let P1 be the set of all one variable partial recursive functions on X *. A function I belongs to P1 if and only if there is a subset of S of X* and some algorithm which computes 't by associating with each element of S a well-defined element of X *. Similarly, 't belongs to P2 if and only if there is a set S c X * X X *, and an algorithm for 't which associates with each element of S a well-defined element of X *. A partial recursive function I in P1 (or in P2) is recursive if the algorithm is defined on all of X * (or X * X X *). This is equivalent to saying that the algorithm which evaluates I always halts. Our computing model will be based on Adat; its high-level string-handling properties simplify later examples. For readability, however, we will express algorithms in a more familiar Pascal-like style. Readers familiar with either programming language should find forthcoming algorithms sufficiently clear. A formal presentation in Ada is given in the Appendix. A string of ASCII characters, such as in FIGURE 1, represents a function in our language which takes strings to strings. Of course, strings of ASCII characters such as The quick brown fox and %*!! represent nothing at all in our programming language. Let A1 be the set of all strings in X * representing functions in our language of one input STRING which outputs a STRING. Similarly let A2 be the set of all strings in

The multi-focus image fusion with adaptable windows (MF-AW) algorithm for multiple images improves the results of the linear combination of images with variable windows (CLI-VV—from its Spanish acronym) algorithm, using a unique decision map and applying parallel programming. Other algorithms use the same window size throughout the image to produce a decision map; furthermore, a different decision map is produced for each pair of images. MF-AW determines the largest possible window size delimited by the edges of the decision map, which are improved using an iterative process. The execution time is improved using integral images, binary search, and parallel programming; as a result, the fused image is obtained in tenths of a second. Quantitative and qualitative measures indicate that the results obtained with this algorithm outperform the state of the art in terms of both accuracy and execution time.

A comparative study of five parallel programming languages

All scalable parallel computers feature a memory hierarchy, in which some locations are “closer” to a particular processor than others. The hardware in a particular system may support a shared memory or message passing programming model, but these factors effect only the relative costs of local and remote accesses, not the system’s fundamental Non-Uniform Memory Access (NUMA) characteristics. Yet while the efficient management of memory hierarchies is fundamental to high performance in scientific computing, existing parallel languages and tools provide only limited support for this management task. Recognizing this deficiency, we propose abstractions and programming tools that can facilitate the explicit management of memory hierarchies by the programmer, and hence the efficient programming of scalable parallel computers. The abstractions comprise local arrays, global (distributed) arrays, and disk resident arrays located on secondary storage. The tools comprise the Global Arrays library, which supports the transfer of data between local and global arrays, and the Disk Resident Arrays (DRA) library, for transferring data between global and disk resident arrays. We describe the shared memory NUMA model implemented in the tools, discuss extensions for wide area computing environments, and review major applications of the tools, which currently total over one million lines of code.