Balinda C++: run-time support for concurrent object-oriented language

A concurrent object-oriented programming language system for highly parallel data-driven computers and its applications

A number of concurrent computers have been designed, such as [Ametek 1988] and [NCUBE 1986]. Taking advantage of concurrency on these computers will require new languages, operating systems, and environments to support the programming, monitoring and control of applications. The Rosette project at MCC is developing a system architecture for general-purpose concurrent computation based on the Actor model [Agha 1986]. The aim of the Rosette architecture is to support the use of concurrent computers in solving computationally intensive problems whose structure is not statically determined. The architecture will support variable grain size, dynamic resource management, reflection, and heterogeneity. A number of concurrent language proposals have employed computational models that are limited in their applicability because of assumptions such as granularity of concurrency, static process topology, or inherent communications cost. Others, for example, functional programming, cannot be used to model shared, modifiable objects. The Actor model addresses these issues directly. However, the difficulties of using concurrent computers cannot be addressed solely through programming languages. Traditional operating systems treat each process as an independent entity for the purposes of resource management (i.e., for the run-time allocation of processor, storage, and communication resources.) In the context of concurrent applications, effective operation on a concurrent computer requires that communities of processes be managed together, taking into account the ways in which they interact, and providing for the application itself to guide resource management. For example, consider that programming systems often provide only limited support for the control of resource consumption in search problems in the face of combinatorial explosion; further, it is generally recognized that to optimize search performance, it is useful to vary the resources provided depending on how promising intermediate results of a given path are judged to be. The Rosette architecture will allow the programmer to integrate high-level algorithms within the application for dividing resources dynamically among the sub-computations that are spawned. These algorithms will provide an assessment of the progress of the sub-computations, and will be used to make the large number of low-level resource management decisions that are necessary in carrying out the computation. Thus, operating system functionality will be folded into the application. The Rosette architecture has two major components: an interface layer and a system environment . The interface layer provides mechanisms for the monitoring and control of concurrent applications. It supports a uniform view of the system resources and applications through the use of actors and message-passing semantics. The interface layer includes a set of actors that represent the processing, storage and communication resources across the hardware base. The system environment contains actor communities which implement resource management policies , and support for programming concurrent applications. The environment consists of an extensible set of actors that are organized via a class-based object model. These actors provide monitoring, debugging, resource management system simulation, and compilation/transformation facilities. To demonstrate the effectiveness of the Rosette architecture, as well as to drive its development, two classes of applications are being used. The first class is the Rosette system itself: all of the system components necessary for resource management, monitoring, debugging, and program development are being constructed in the Rosette language and will take full advantage of both concurrency and object-oriented programming support. The second category of applications is chosen to be somewhat independent of the Rosette system so as to ensure adequate feed-back concerning applicability to domains other than software engineering. These applications are in the general area of knowledge-based signal interpretation. The Rosette architecture is implemented via a concurrent object-oriented programming language that provides the following features: Inherent concurrency : A declarative approach is needed which allows fine-grained concurrency to arise naturally from the program. If the programmer is forced to express all of the details of the concurrent execution of an application, it becomes difficult to re-map a given program to different concurrent computers which have varying concurrency characteristics. Dynamic creation/modification of objects : This allows the system to be extensible and reconfigurable. Sharing : The language should support mechanisms such as inheritance so that code, structure, and values need not be replicated. Classes : Objects can be organized into classes to permit more effective management of resources, as for example in a very large database. Classes should be first-class objects and hence it should be possible to create classes dynamically. The Rosette language is a high-level language that is translated into a base language that directly expresses concurrent computations in terms of the semantics of the Actor model. The Rosette language extends the base language with data abstraction facilities and a library of classes, and is used to implement the non-built-in actors of the interface layer, components of the system environment, and applications. Using data abstraction extensions to the base language, the Rosette language organizes actors into classes according to common sets of behaviors and local state. Inheritance is used as a means of sharing code and structure that is common among classes. There is considerable evidence, for example [LaLonde 1986] and [Snyder 1987], that a robust class system should support separate hierarchies that represent what is modeled by a class and how some class of objects are actually implemented. Our approach is to use three different kinds of classes to organize the various kinds of actors in a system and information about these actors: abstract classes specify observable requests, responses, and actions; representation classes specify resource management characteristics of implementations; and behavior classes contain the actual implementations. Associated with an abstract class is the set of representations that are available to implement the particular abstraction. Each representation is associated with the behavior class that contains the corresponding implementation. The power of this separation is that each of the three kinds of class may be related to others in a class hierarchy. This means that the relationships between abstract classes are not forced to have the same structure as the relationships among implementations. Sharing via inheritance among behavior classes is related to engineering concerns; relations among abstract classes are dictated by external/observable behavior. As an example, an abstract class might specify the operations available over real-matrices . There are several ways of representing a matrix as a community of actors, and each of these would be specified with appropriate representation and behavior classes. There is a well-defined abstract hierarchy of matrices such as invertible, positive , etc. These abstract kinds of matrices are independent of the various implementations, which may be chosen according to application and hardware resource constraints. The Rosette language embodies a reflective model of the interface layer that is similar to that used in ABCL/R, [Watanabe and Yonezawa 1988]. This model is t

Fine-grained concurrent object-oriented programming (COOP) models can simplify the programming of irregular parallel applications but are often perceived as inefficient. In this paper, we study implementation techniques to obtain efficient parallel execution of fine-grained COOP languages using a medium-sized protein dynamics program. We found that even with high data locality and good sequential code efficiency, an implementation using only thread-oriented optimizations and software multithreading fails to achieve parallel efficiency. The two major sources of overhead are the lack of processor-level data reuse and fine-grained threads for remote object accesses. Two processor-oriented optimizations, processor-level caching and communication grouping, overcome these inefficiencies and achieve performance comparable to a highly-tuned SPMD program.

We present fast and highly scalable parallel computations for a number of important and fundamental matrix problems on distributed memory systems (DMS). These problems include matrix multiplication, matrix chain product, and computing the powers, the inverse, the characteristic polynomial, the determinant, the rank, the Krylov matrix, and an LU- and a QR-factorization of a matrix, and solving linear systems of equations. Our highly scalable parallel computations for these problems are based on a highly scalable implementation of the fastest sequential matrix multiplication algorithm on DMS. We show that compared with the best known parallel time complexities on parallel random access machines (PRAM), the most powerful but unrealistic shared memory model of parallel computing, our parallel matrix computations achieve the same speeds on distributed memory parallel computers (DMPC), and have an extra polylog factor in the time complexities on DMS with hypercubic networks. Furthermore, our parallel matrix computations are fully scalable on DMPC and highly scalable over a wide range of system size on DMS with hypercubic networks. Such fast (in terms of parallel time complexity) and highly scalable (in terms of our definition of scalability) parallel matrix computations were rarely seen before on any distributed memory systems.

Intensive research has been conducted on dynamic job scheduling, which dynamically allocates jobs to computing systems. However, most of the existing work is limited to redistribute independent tasks or at the algorithm design level. There is no runtime system available to support automatic redistribution of a running process in a heterogeneous network environment. In this study, we present the design and implementation of a system that dynamically reschedules running processes over a network of computing resources via automatic decision-making and process migration. The system is implemented on top of MPI-2 and HPCM (high performance computing mobility) middleware. Experimental and analytical results show that the runtime system works well. It makes dynamic rescheduling of running tasks possible and improves system performance considerably. While the implementation is for MPI programs and using HPCM, the design of the system is general and can be extended to other distributed environments as well.

Non-blocking multithreaded architectures have been proposed as an effective means to overlap computation and communication in distributed memory systems. However, in these models, communication latency could only be hidden from computation as long as there are enough ready threads. We describe our I-Structure Software Cache (ISSC) run-time system, which takes advantage of the global data locality in these models without adding any specific hardware support. Our ISSC provides a communication latency reduction technique in non-blocking multithreaded architectures while maintaining the ability of tolerating communication latency. Our simulation results show that our ISSC run-time system dramatically decreases network traffic by caching remote requests and also achieves 70% to 95% improvement on the system performance (speedup).

The Complex neuronal network models require support from simulation environment for efficient network simulations. To compute the models increasing complexity necessitated the efforts to parallelize the NEURON simulation environment. The computational neuroscientists have extended NEURON by dividing the equations for its subnet among multiple processors for increasing the competence of hardware. For spiking neuronal networks inter-processor spikes exchange consume significant portion of overall simulation time on parallel machines. In NEURON Message Passing Interface (MPI) is used for inter processor spikes exchange, MPI_Allgather collective operation is used for spikes exchange generated after each interval across distributed memory systems. However, as the number of processors become larger and larger MPI_Allgather method become bottleneck and needs efficient exchange method to reduce the spike exchange time. This work has improved MPI_Allgather method to Remote Memory Access (RMA) based on MPI-3.0 for NEURON simulation environment, MPI based on RMA provides significant advantages through increased communication concurrency in consequence enhances efficiency of NEURON and scaling the overall run time for the simulation of large network models.1

Concurrent object-oriented programming languages (concurrent OOPLs) are promising approach for building large and complex applications on multicomputers, but their dynamic/asynchronous nature makes efficient implementation difficult--support of dynamic concurrent object creation involves scheduling of multiple objects on each node, and sending of asynchronous messages between objects cause buffering of messages, both of which could result in significant overhead. One possible direction to achieve efficiency is to combine dedicated hardware[3, 1] and software tailored for it[2, 7]. The hardware has crucial support for concurrent OOPLs--low overhead remote communication and efficient multithreading. While they are attractive, the majority of current commercial multicomputers such as AP1000[4] and CM5 have no such support. This note summarizes our implementation scheme[5, 8] on these conventional multicomputers, and shows that concurrent OOPLs can greatly benefit from their superior node-local performance while communication performance can be made comparable to dedicated architectures. Basic operations such as message passing or object creation were implemented very efficiently. Intra-node message passing takes about 15 instructions in addition to the cost of procedure call, unless the receiver object is already executing some other method. Inter-node message passing latency is within 10ps for smM1 messages. An application benchmark showed good speedup (440 times speedup on 512 nodes system) for a large program.

Process-thread hybrid programming paradigm is commonly employed in SMP clusters. XPFortran, a parallel programming language that specifies a set of compiler directives and library routines, can be used to realize process-level parallelism in distributed memory systems. In this paper, we introduce hybrid parallel programming by XPFortran to SMP clusters, in which thread-level parallelism is realized by OpenMP. We present the language support and compiler implementation of OpenMP directives in XPFortran, and show some of our experiences in XPFortran-OpenMP hybrid programming. For nested loops parallelized by process-thread hybrid programming, it’s common sense to use process parallelization for outer loops and thread parallelization for inner ones. However, we have found that in some cases it’s possible to write XPFortran-OpenMP hybrid program in a reverse way, i.e., OpenMP outside, XPFortran inside. Our evaluation results show that this programming style sometimes delivers better performance than the traditional one. We therefore recommend using the hybrid parallelization flexibly.

<p><span>Large-eddy simulation (LES) is an often used technique in the geosciences to simulate turbulent oceanic and atmospheric flows. In LES, the effects of the unresolved turbulence scales on the resolved scales (via the Reynolds stress tensor) have to be parameterized with subgrid models. These subgrid models usually require strong assumptions about the relationship between the resolved flow fields and the Reynolds stress tensor, which are often violated in reality and potentially hamper their accuracy.</span></p><p><span>In this study, using the finite-difference computational fluid dynamics code MicroHH (v2.0) and turbulent channel flow as a test case (friction Reynolds number Re<sub>τ</sub> 590), we incorporated and tested a newly emerging subgrid modelling approach that does not require those assumptions. Instead, it relies on neural networks that are highly non-linear and flexible. Similar to currently used subgrid models, we designed our neural networks such that they can be applied locally in the grid domain: at each grid point the neural networks receive as an input the locally resolved flow fields (u,v,w), rather than the full flow fields. As an output, the neural networks give the Reynolds stress tensor at the considered grid point. This local application integrates well with our simulation code, and is necessary to run our code in parallel within distributed memory systems.</span></p><p><span>To allow our neural networks to learn the relationship between the specified input and output, we created a training dataset that contains ~10.000.000 samples of corresponding inputs and outputs. We derived those samples directly from high-resolution 3D direct numerical simulation (DNS) snapshots of turbulent flow fields. Since the DNS explicitly resolves all the relevant turbulence scales, by downsampling the DNS we were able to derive both the Reynolds stress tensor and the corresponding lower-resolution flow fields typical for LES. In this calculation, we took into account both the discretization and interpolation errors introduced by the finite staggered LES grid. Subsequently, using these samples we optimized the parameters of the neural networks to minimize the difference between the predicted and the ‘true’ output derived from DNS.</span></p><p><span>After that, we tested the performance of our neural networks in two different ways:</span></p><ol><li><span>A priori or offline testing, where we used a withheld part of the training dataset (10%) to test the capability of the neural networks to correctly predict the Reynolds stress tensor for data not used to optimize its parameters. We found that the neural networks were, in general, well able to predict the correct values. </span></li> <li><span>A posteriori or online testing, where we incorporated our neural networks directly into our LES. To keep the total involved computational effort feasible, we strongly enhanced the prediction speed of the neural network by relying on highly optimized matrix-vector libraries. The full successful integration of the neural networks within LES remains challenging though, mainly because the neural networks tend to introduce numerical instability into the LES. We are currently investigating ways to minimize this instability, while maintaining the high accuracy in the a priori test and the high prediction speed.</span></li> </ol>

The design of temporally-predictable real-time software is a major issue in a wide range of computer applications. This paper focuses on the temporal modelling of communication protocols in real-time OSI-based computer networks, and discusses a method which can form the foundation of a tool which gives true temporal representations of such systems. The proposed approach, implemented using the power of a concurrent object-oriented programming language (STRAND), utilises a modelling methodology based on an advanced process/channel model. The paper suggests that the techniques can be applied to a range of problems, illustrated here by an example of an OSI protocol.

A new class of normalized explicit approximate inverse matrix techniques, based on normalized approximate factorization procedures, for solving sparse linear systems resulting from the finite difference discretization of partial differential equations in three space variables are introduced. A new parallel normalized explicit preconditioned conjugate gradient square method in conjunction with normalized approximate inverse matrix techniques for solving efficiently sparse linear systems on distributed memory systems, using Message Passing Interface (MPI) communication library, is also presented along with theoretical estimates on speedups and efficiency. The implementation and performance on a distributed memory MIMD machine, using Message Passing Interface (MPI) is also investigated. Applications on characteristic initial/boundary value problems in three dimensions are discussed and numerical results are given.

The task-based programming model associated with dynamic runtime systems has gained popularity for challenging problems because of workload imbalance, heterogeneous resources, or extreme concurrency. During the last decade, low-rank matrix approximations-where the main idea consists of exploiting data sparsity, typically by compressing off-diagonal tiles up to an application-specific accuracy threshold-have been adopted to address the curse of dimensionality at extreme scale. In this paper, we create a bridge between the runtime and the linear algebra by communicating knowledge of the data sparsity to the runtime. We design and implement this synergistic approach with high user productivity in mind, in the context of the PaRSEC runtime system and the HiCMA numerical library. This requires extending PaRSEC with new features to integrate rank information into the dataflow so that proper decisions can be made at runtime. We focus on the tile low-rank (TLR) Cholesky factorization for solving 3D data-sparse covariance matrix problems arising in environmental applications. In particular, we employ the 3D exponential model of the Mateŕn matrix kernel, which exhibits challenging nonuniform high ranks in off-diagonal tiles. We first provide dynamic data structure management driven by a performance model to reduce extra floating-point operations. Next, we optimize the memory footprint of the application by relying on a dynamic memory allocator, and supported by a rank-aware data distribution to cope with the workload imbalance. Finally, we expose further parallelism using kernel recursive formulations to shorten the critical path. Our resulting high-performance implementation outperforms existing data-sparse TLR Cholesky factorization by up to 7-fold on a large-scale distributed-memory system, while minimizing the memory footprint up to a 44-fold factor. This multidisciplinary work highlights the need to empower runtime systems beyond their original duty of task scheduling for servicing next-generation low-rank matrix algebra libraries.

For an increasing number of data intensive scientific applications, parallel I/O concepts are a major performance issue. Tackling this issue, we provide an outline of an input/output system designed for highly efficient, scalable and conveniently usable parallel I/O on distributed memory systems. The main focus of this paper is the parallel I/O runtime system support provided for software-generated programs produced by parallelizing compilers in the context of High Performance FORTRAN efforts. Specifically, our design is presented in the context of the Vienna Fortran Compilation System.KeywordsApplication ProcessArray ElementProcessor ArrayRuntime SystemMass StorageThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

We consider the parallel-machine flexible-resource scheduling (PMFRS) problem in which a set of jobs must be scheduled over a set of parallel machines, where the processing time of each job is a function of the amount of allocated resource. Resource flexibility provides the capability to dynamically reassign a renewable resource across machines to break processing bottlenecks and improve system performance as measured by schedule makespan. The PMFRS problem has many important applications, including production scheduling of manufacturing cells where a cross-trained work force can be dynamically reallocated among cells. The problem is also NP-hard, motivating the development of effective heuristics that approximately determine the allocation of resource to jobs, the sequence of jobs on each machine, and the associated job start times that minimize system makespan. This paper explores heuristics for the PMFRS problem, and in particular the application of tabu-search methodology to this problem setting. We review an existing heuristic (SBH), define two tabu-search heuristics, and discuss extensive computational experience with the procedures. The computational results indicate that the heuristics are effective in obtaining approximate solutions to the PMFRS problem. In particular, the approach that uses tabu-search methodology in tandem with SBH consistently yields high-quality solutions with modest computational effort.