Foundations of practical parallel programming languages

We describe the main implementation solutions for the MAYaK family of parallel programming languages, as determined by their distinctive features compared to sequential programming languages. These solutions have been implemented in the MAYaK parallel programming system for the ES 1766 multiprocessor computing system.

This chapter will be totally different from previous chapters as so far there is no really grand new parallel programming language to be used already. What people have are only the parallel facilities parasitically affiliated to the existing languages. The discussions before mainly focus on the issues of compilation of sequential programming languages, and in this chapter and the following chapter we will discuss the issues on the compilation of parallel languages. Is it important? The answer is definitely yes. Because in the new century, no doubt, parallel computers will dominate the whole market of computers. Then developing programs for this kind of computers will be the must if some one wants to be the programmer of the new century. Of course he/she also needs to know how his/her programs are compiled by the corresponding compilers. It is not exaggerating that parallel compilation will become the main theme in compiler field.

Mainly designed for simulating the fluid motion, Smooth Particle Hydrodynamics (SPH) SPH has been widely applied to many fields, such as aerospace aircraft design and ship design. First, this paper presents an implementation of serial SPH method algorithm using the reflection method as the boundary condition. This paper describes entire process of SPH and gives detailed descriptions of some important procedures. Second, this paper realizes the corresponding parallel algorithm of SPH, which includes neighbor particle searching and building, physical attributes computation and other main SPH method procedures. Third, details of mapping different parts of the SPH serial algorithm to parallel algorithm on GPU with CUDA C as parallel programming language are also shown in this paper. Taking advantage of powerful computation ability of GPU, the experiment approximately achieves a 8X speedup that shows the high efficiency of GPU-based programming. The parallel algorithm not only achieves a high performance, but also promises a reliability result with a small error rate as the experiment result shows.

We present an Adaptive Mesh Refinement benchmark for evaluating programmability and performance of modern parallel programming languages. Benchmarks employed today by language developing teams, originally designed for performance evaluation of computer architectures, do not fully capture the complexity of state-of-the-art computational software systems running on today's parallel machines or to be run on the emerging ones from the multi-cores to the peta-scale High Productivity Computer Systems. This benchmark, extracted from a real application framework, presents challenges for a programming language in both expressiveness and performance. It consists of an infrastructure for finite difference calculations on block-structured adaptive meshes and a solver for elliptic Partial Differential Equations built on this infrastructure. Adaptive Mesh Refinement algorithms are challenging to implement due to the irregularity introduced by local mesh refinement. We describe those challenges posed by this benchmark through two reference implementations (C++ /Fortran/MPI and Titanium) and in the context of three programming models.

Research in programming environments for supercomputers covers several broad areas, including the design of very high level specification tools, the development of new programming languages, and supercompilers that transform code written in a sequential programming language into equivalent concurrent code that can be efficiently executed on a target machine. This chapter is concerned with the last of these, which is also known as automatic restructuring. It examines techniques and systems which were mainly developed for the purpose of transforming sequential numerical programs for large-scale scientific and engineering applications into concurrent programs. The chapter discusses programming environments to support the task of automatic restructuring. It outlines the typical phases of program development and describes the basic scheme for an interactive restructuring system. The chapter provides most important requirements for a suitable programming environment for concurrent computers. The chapter considers some requirements that should be met by advanced software environments supporting program development for concurrent computers.

This article discusses a serial and a parallel version of a heuristic algorithm for a solution to the traveling salesman problem, implementation of these algorithms, and their performance in terms of finding the most favorable solution, and an average solution. The algorithms implemented in C sequential programming language, and in Occam concurrent programming language were run on different computing platforms to study their impact on the comparative value of the solutions each platform provided. The programs were tested on reference data from TSPLIB data files, a collection of sample instances of traveling salesman problem. Experimental results, using asymmetric cost matrix, indicate good performance compared to the optimal solutions provided in the TSPLIB reference library.

I will discuss the design of Ct, an API for nested data parallel programming in C++. Ct uses meta-programming and functional language ideas to essentially embed a pure functional programming language in impure and unsafe languages, like C++. I will discuss the evolution of the design into functional programming ideas, how this was received in the corporate world, and how we plan to proliferate the technology in the next year.Ct is a deterministic parallel programming model integrating the nested data parallelism ideas of Blelloch and bulk synchronous processing ideas of Valiant. That is, data races are not possible in Ct. Moreover, performance in Ct is relatively predictable. At its inception, Ct was conceived as a simple library implementation behind C++ template magic. However, performance issues quickly forced us to consider some form of compilation. Using template programming was highly undesirable for this purpose as it would have been difficult and overly specific to C++ idiosyncrasies. Moreover, once compilation for performance was considered, we began to consider a language semantics that would enable powerful optimizations like calculational fusion, synchronization barrier elimination, and so on. The end result of this deliberation is an API that exposes a value-oriented, purely functional vector processing language. Additional benefits of this approach are numerous, including the important ability to co-exist within legacy threading programming models (because of the data isolation inherent in the model). We will show how the model applies to a wide range of important (at least by cycle count) applications. Ct targets both shipping multi-core architectures from Intel as well as future announced architectures.The corporate reception to this approach has (pleasantly) surprised us. In the desktop and high-performance computing space, where C, C++, Java, and Fortran are the only programming models people talk about, we have made serious inroads into advocating advanced programming language technologies. The desperate need for productive, scalable, and safe programming languages for multi-core architectures has provided an opening for functional, type-safe languages. We will discuss the struggles of multi-core manufacturers (i.e. Intel) and their software vendors that have created this opening.For Intel, Ct heralds its first serious effort to champion a technology that borrows functional programming technologies from the research community. Though it is a compromise that accommodates the pure in the impure and safe in the unsafe, this is an important opportunity to demonstrate the power of functional programming to the unconverted. We plan to share the technology selectively with partners and collaborators, and will have a fully functional and parallelizing implementation by year's end. At CUFP, we will be prepared to discuss our long term plans in detail.

Process interference due to shared variables is a serious problem in parallel programs written in insecure programming languages. The only effective remedy has been known since the 1970s: a parallel programming language must be designed to permit detection of process interference during compilation. This sound principle has seldom been adopted and enforced consistently for block-structured parallel languages. This paper discusses syntactic control of interference in SuperPascal, a block-structured programming language for parallel scientific computing. SuperPascal omits the insecure concepts of Pascal, and adds parallel statements and synchronous communication channels. Restrictions on the use of variables permit a single-pass compiler to check that parallel processes are disjoint, even if they use procedures with global variables.

Current programming languages and programming models make it easy to create software and hardware systems that fulfill an intended function but also leave such systems open to unintended function and vulnerabilities. Software engineering and code hygiene may make systems incrementally safer, but do not produce the wholesale change necessary for secure systems from the outset. Yet there exists an approach with impressive results: We cite recent examples showing that formal methods, coupled with formally informed digital design, have produced objectively more robust code even beyond the properties directly proven. Though discovery of zero-day vulnerabilities is almost always a surprise and powerful tools like semantic fuzzers can cover a larger search space of vulnerabilities than a developer can conceive of, formal models seem to produce robustness of a higher qualitative order than traditionally developed digital systems. Because the claim is necessarily a qualitative one, we illustrate similar results with an idealized programming language in the form of Boolean networks where we have control of parameters related to stability and adaptability. We argue that verifiability with formal methods is an instance of broader design constraints that promote robustness. We draw analogies to real-world programming models and languages that can be mathematically reasoned about in contrast to ones that are essentially undecidable.

Accelerated clusters, which are cluster systems equipped with accelerators, are one of the most common systems in parallel computing. In order to exploit the performance of such systems, it is important to reduce communication latency between accelerator memories. In addition, there is also a need for a programming language that facilitates the maintenance of high performance by such systems. The goal of the present article is to evaluate XcalableACC (XACC), a parallel programming language, with tightly coupled accelerators/InfiniBand (TCAs/IB) hybrid communication on an accelerated cluster. TCA/IB hybrid communication is a combination of low-latency communication with TCA and high bandwidth with IB. The XACC language, which is a directive-based language for accelerated clusters, enables programmers to use TCA/IB hybrid communication with ease. In order to evaluate the performance of XACC with TCA/IB hybrid communication, we implemented the lattice quantum chromodynamics (LQCD) mini-application and evaluated the application on our accelerated cluster using up to 64 compute nodes. We also implemented the LQCD mini-application using a combination of CUDA and MPI (CUDA + MPI) and that of OpenACC and MPI (OpenACC + MPI) for comparison with XACC. Performance evaluation revealed that the performance of XACC with TCA/IB hybrid communication is 9% better than that of CUDA + MPI and 18% better than that of OpenACC + MPI. Furthermore, the performance of XACC was found to further increase by 7% by new expansion to XACC. Productivity evaluation revealed that XACC requires much less change from the serial LQCD code to implement the parallel LQCD code than CUDA + MPI and OpenACC + MPI. Moreover, since XACC can perform parallelization while maintaining the sequential code image, XACC is highly readable and shows excellent portability due to its directive-based approach.

Recently, MPI has become widely used in many scientific applications, including different non-computer science fields, for parallelizing their applications. An MPI programming model is used for supporting parallelism in several programming languages, including C, C++, and Fortran. MPI also supports integration with some programming models and has several implementations from different vendors, including open-source and commercial implementations. However, testing parallel programs is a difficult task, especially when using programming models with different behaviours and types of error based on the programming model type. In addition, the increased use of these programming models by non-computer science specialists can cause several errors due to lack of experience in programming, which needs to be considered when using any testing tools. We noticed that dynamic testing techniques have been used for testing the majority of MPI programs. The dynamic testing techniques detect errors by analyzing the source code during runtime, which will cause overheads, and this will affect the program's performance, especially when targeting massive parallel applications generating thousands or millions of threads. In this paper, we enhance ACC_TEST to have the ability to test MPI-based programs and detect runtime errors occurring with different types of MPI communications. We decided to use hybrid-testing techniques by combining both static and dynamic testing techniques to gain the benefit of each and reduce the cost.

Shared memory programming and distributed memory programming, are the most prominent ways of parallelize applications requiring high processing times and large amounts of storage in High Performance Computing (HPC) systems; parallel applications can be represented by Parallel Task Graphs (PTG) using Directed Acyclic Graphs (DAGs). The scheduling of PTGs in HPCS is considered a NP-Complete combinatorial problem that requires large amounts of storage and long processing times. Heuristic methods and sequential programming languages have been proposed to address this problem. In the open access paper: Scheduling in Heterogeneous Distributed Computing Systems Based on Internal Structure of Parallel Tasks Graphs with Meta-Heuristics, the Array Method is presented, this method optimizes the use of Processing Elements (PE) in a HPCS and improves response times in scheduling and mapping resource with the use of the Univariate Marginal Distribution Algorithm (UMDA); Array Method uses the internal characteristics of PTGs to make task scheduling; this method was programmed in the C language in sequential form, analyzed and tested with the use of algorithms for the generation of synthetic workloads and DAGs of real applications. Considering the great benefits of parallel software, this research work presents the Array Method using parallel programming with OpenMP. The results of the experiments show an acceleration in the response times of parallel programming compared to sequential programming when evaluating three metrics: waiting time, makespan and quality of assignments.

A new parallel computation model for logic programs is described. A parallel logic programming language Parallel Prolog has been considered as a high level language. A data-flow model is a theoretical background of our computation model. Owing to this the model is asynchronous and utilizes innate kinds of parallelism for a highly parallel all solution strategy in logic programs. We have designed new basic operators for Parallel Prolog supporting OR and AND-stream parallelism. We have designed an abstract parallel machine with distributed structure memory, which is based on data-flow principles. Basic formats of a data-flow instruction, operand and structure memory cell are introduced. A special operator for head unification, calling the first and other body literals, for completion of the clause body, for processing of the facts, built-in predicates and output of results have been designed. A stream, an asynchronous data structure, is used as a communication channel between body literals. A data structure called a template is used for selection of the goal terms for calling of each body literal. A parallel logic program in the form of the data-flow program graph generated by the compiler is introduced as a simple example of the complete program. The designed model will be implemented by its introduced abstract machine. Owing to larger operator granularity this new model has much lower requirements for a communication overhead in comparison with our previous fine grain version.

The AgentSpeak agent-oriented programming language has recently been extended with various new features, such as speech-act based communication, internal belief additions, and support for reasoning with ontological knowledge, which imply the need for belief revision within an AgentSpeak agent. In this paper, we show how a polynomial-time belief-revision algorithm can be incorporated into the Jason AgentSpeak interpreter by making use of Jason's language constructs and customisation features. This is one of the first attempts to include automatic belief revision within an interpreter for a practical agent programming language.

This book provides an accessible introduction to thermal physics with computational approaches that complement the traditional mathematical treatments of classical thermodynamics and statistical mechanics. It guides readers through visualizations and simulations in the Python programming language, helping them to develop their own technical computing skills (including numerical and symbolic calculations, optimizations, recursive operations, and visualizations). Python is a highly readable and practical programming language, making this book appropriate for students without extensive programming experience. This book may serve as a thermal physics textbook for a semester-long undergraduate thermal physics course or may be used as a tutorial on scientific computing with focused examples from thermal physics. This book will also appeal to engineering students studying intermediate-level thermodynamics as well as computer science students looking to understand how to apply their computer programming skills to science. Key features Major concepts in thermal physics are introduced cohesively through computational and mathematical treatments. Computational examples in Python programming language guide students on how to simulate and visualize thermodynamic principles and processes for themselves.