LOCA 1.0 Library of Continuation Algorithms: Theory and Implementation Manual

Distributed memory parallel machines are increasingly being used by the scientific community to provide computational power for solving problems. Building a distributed memory parallel machine offers a tremendous advantage in terms of cost and scalability over building a shared memory parallel machine. Unfortunately, distributed memory machines are much more difficult to program, mainly due to the lack of a single global address space. Communication between processors in a distributed memory machine is accomplished by message passing. The time cost for acquiring data from another processor's memory is significantly greater than the time cost of acquiring data from local memory. To assist the programmer in using a distributed memory machine, compilers have been developed which alleviate the need for the programmer to manage the lack of a global address space. These compilers will generate semantically correct code containing communication routines to produce a logically shared memory machine on a physically distributed memory machine. However, the runtime performance of the code emitted from some of these compilers can be further improved. This thesis is primarily concerned with advanced compilation techniques which will provide better runtime performance of programs written in imperative languages for regular scientific programs. The optimization of communication generation for distributed memory machines is used to produce this result. We look at two "state of the art" compilation systems and find that improvement in the communication code generated by these compilers is still possible. We have simplified existing descriptions of the various formalisms found in these compilers, while still providing enough formalism to study the problems in depth. We have extended some existing optimization techniques by enhancing the elimination of masks, by extending the notion of relaxing the owner computes rule, by extending the notion of replicating computation, and by extending the notion of replicating data. This thesis has discussed a novel technique to split loops so that they do not contain any guarded statements. Finally, we have suggested avenues for future research, as it is clear from this thesis that more work is required before compilation systems for distributed memory parallel machines will be widely used for general purpose computing.

The efficiency of controlling the simulated moving bed (SMB) has long been a critical issue in the chemical engineering industry. Most existing research relies on finite element methods, which often result in lower control efficiency and are unable to achieve online control. To enhance control over the SMB process, this paper employs the Crank–Nicolson method to develop a discrete dynamical model. This approach allows for the investigation of system stability and convergence, fundamentally addressing the sources of error. During the discretization of partial differential equations (PDEs), two main types of errors arise: intrinsic errors from the method itself and truncation errors due to derivative approximations and the discretization process. Research indicates that for the former, the iterative process remains convergent as long as the time and spatial steps are sufficiently small. Regarding truncation errors, studies have demonstrated that they exhibit second-order behavior relative to time and spatial steps. The theoretical validation shows that the iteration works effectively, and simulations confirm that the finite difference method is stable and performs well with varying SMB system parameters and controller processes. This provides a solid theoretical foundation for practical, real-time online control.

Over the last year, a considerable portion of FEFLO, a general-purpose legacy CFD code operating on unstructured grids, was ported to run on GPUs. Like so many other legacy codes, FEFLO is an adaptive, edge-based nite element code for the solution of compressible and incompressible ow, which was primarily written in Fortran 77 and has previously been ported to vector, shared memory parallel and distributed memory parallel machines. Due to the large size of FEFLO and the likelihood of human error in porting, as well as the desire for continued development within a single codebase, a specialized Python script, based on FParser (Peterson, 2009), was written to perform automated translation from the OpenMP-parallelized loops to GPU kernels implemented in CUDA, along with GPU memory management, while integrating with the existing framework for distributed memory parallelism via MPI. The present paper describes extensions of the script and algorithmic techniques that enable the ecient running on GPUs of the modules that are not straightforward to port. In particular, we consider LU-SGS algorithms, linelet preconditioning and particle-mesh algorithms.

In this paper we study the parallel performance of some nonlinear additive Schwarz preconditioned inexact Newton methods for solving large sparse system of nonlinear equations arising from the discretization of partial differential equations. The main idea of nonlinear preconditioning is to replace an ill-conditioned nonlinear system by an equivalent nonlinear system that has more balanced nonlinearities. In addition to balance the nonlinearities through nonlinear preconditioning, we also need to make sure that the multilayered iterative solver is scalable with respect to the number of processors. We focus on some two-level nonlinear additive Schwarz preconditioners, and show numerically that these two-level methods can reduce the nonlinearities and at the same time maintain the parallel scalability. Parallel numerical results for some high Reynolds number incompressible Navier-Stokes equations will be presented.

During my visit to LLNL during the summer of 2010, I worked on algebraic multilevel solvers for large sparse systems of linear equations arising from discretizations of partial differential equations. The particular solver of interest is based on ILU decomposition. The setup phase for this AMG solve is just the single ILU decomposition, and its corresponding error matrix. Because the ILU uses a minimum degree or similar sparse matrix ordering, most of the fill-in, and hence most of the error, is concentrated in the lower right corner of the factored matrix. All of the major multigrid components - the smoother, the coarse level correction matrices, and the fine-to-coarse and coarse-to-fine rectangular transfer matrices, are defined in terms of various blocks of the ILU factorization. Although such a strategy is not likely to be optimal in terms of convergence properties, it has a relatively low setup cost, and therefore is useful in situations where setup costs for more traditional AMG approaches cannot be amortized over the solution of many linear systems using the same matrix. Such a situation arises in adaptive methods, where often just one linear system is solved at each step of an adaptive feedback loop, or in solving nonlinear equations by approximate Newton methods, where the approximate Jacobian might change substantially from iteration to iteration. In general terms, coarse levels are defined in terms of successively smaller lower right blocks of the matrix, typically decreasing geometrically in order. The most difficult issue was the coarse grid correction matrix. The preconditioner/smoother for a given level is just the corresponding lower right blocks of the ILU factorization. The coarse level matrix itself is just the Schur complement; this matrix is not known exactly using just the ILU decomposition in the setup phase. Thus we approximate this matrix using various combinations of the preconditioning matrix and the error matrix. During my visit, several approximations of this type were implemented and tested. While some improved the convergence rate of the overall method, these gains had to be balanced against the additional costs involved in creating and applying these matrices. By this more stringent criterion, none of the improved approximations could be characterized as an unqualified success.

The implementation and performance of a parallel spatial direct numerical simulation (PSDNS) approach on the Intel iPSC/860 hypercube and IBM SP1 and SP2 parallel computers is documented. Spatially evolving disturbances associated with laminar-to-turbulent transition in boundary-layer flows are computed with the PSDNS code. The feasibility of using the PSDNS to perform transition studies on these computers is examined. The results indicate that PSNDS approach can effectively be parallelized on a distributed memory parallel machine by remapping the distributed data structure during the course of the calculation. The performance of the PSDNS indicates that 32 nodes on the IBM SP1 or SP2 are required to match a comparable computation on the CRAY C-90 single processor; the hypercube was found to be two times slower than the CRAY Y/MP for this application.

Loops are the main source of parallelism in scientific programs. Hence, several techniques were developed to detect parallelism in these loops and transform them into parallel forms. In this dissertation, compile time transformations and efficient parallel execution of loops with various type of dependencies are investigated. First, Doacross loops with uniform dependencies are considered for execution on distributed memory parallel machines (multicomputers). Most known Doacross loop execution techniques can be applied efficiently only to shared memory parallel machines. In this thesis, code reordering technique, improvements to partitioning strategies, and finding a balance between communication and parallelism are presented to reduce the execution time of Doacross loops on multicomputers. As with most parallelizing transformation techniques, only single loopnests are considered in the first part of this dissertation. However, parallelizing each loopnest in a program separately, even if an optimal execution can be obtained for each loopnest, may not result in an efficient execution of all the code in the program because of communication overhead across the loops in a multicomputer environment. Hence, across loop data dependence representation and analysis are introduced in this work to improve the parallel execution of the whole code. Our contribution consists of finding and representing data dependencies whose sources and destinations are subspaces of the iteration space mainly common across the loops. This type of dependence information is used in this thesis to improve global iteration space partitioning, automatic generation of communication statements across loops, and index alignment. The final part of this dissertation presents new parallelizing techniques for loops with irregular and complex dependencies. Various data dependence analysis algorithms can be found in the literature even for loops with complex array indexing. However, the improvement in data dependence testing has not been followed by similar amelioration in restructuring transformations for loops with complex dependencies. Such loops are mostly executed in serial mode. Our parallelizing techniques for these loops consists of identifying regions of the iteration space where all iterations can be executed in parallel. The advantages of all the transformations presented in this dissertation are: (1) they significantly reduce the execution time of loops with various types of dependencies as shown in this work using the MasPar machine (2) they can be implemented at compile, time which makes the task of parallel programming easier.

Array remapping are useful to many applications on distributed memory parallel machines. They are available in High Performance Fortran, a Fortran-based data-parallel language. This paper describes techniques to handle dynamic mappings through simple array copies: array remapping are translated into copies between statically mapped distinct versions of the array. It discusses the language restrictions required to do so. The remapping graph which captures all remapping and liveness information is presented, as well as additional data-flow optimizations that can be performed on this graph, so as to avoid useless remapping at run time. Such useless remapping appear for arrays that are not used after a remapping. Live array copies are also kept to avoid other flow-dependent useless remappings. Finally the code generation and runtime required by our scheme are discussed. These techniques are implemented in our prototype HPF compiler.

Abstract: Processing and analyzing large volumes of data plays an increasingly important role in many domains of scientific research. We are developing a compiler that processes data intensive applications written in a dialect of Java and compiles them for efficient execution on distributed memory parallel machines. In this paper, we focus on the problem of generating correct and efficient communication for data intensive applications. We present static analysis techniques for 1) extracting a global reduction function from a data parallel loop, and 2) determining if a subscript function is monotonic. We also present a runtime technique for reducing the volume of communication during the global reduction phase. We have experimented with two data intensive applications to evaluate the efficacy of our techniques. Our results show that 1) our techniques for extracting global reduction functions and establishing monotonicity of subscript functions can successfully handle these applications, 2) significant reduction in communication volume and execution times is achieved through our runtime analysis technique, 3) runtime communication analysis is critical for achieving speedups on parallel configurations.

Abstract: Processing and analyzing large volumes of data plays an increasingly important role in many domains of scientific research. We are developing a compiler that processes data intensive applications written in a dialect of Java and compiles them for efficient execution on distributed memory parallel machines. In this paper, we focus on the problem of generating correct and efficient communication for data intensive applications. We present static analysis techniques for 1) extracting a global reduction function from a data parallel loop, and 2) determining if a subscript function is monotonic. We also present a runtime technique for reducing the volume of communication during the global reduction phase. We have experimented with two data intensive applications to evaluate the efficacy of our techniques. Our results show that 1) our techniques for extracting global reduction functions and establishing monotonicity of subscript functions can successfully handle these applications, 2) significant reduction in communication volume and execution times is achieved through our runtime analysis technique, 3) runtime communication analysis is critical for achieving speedups on parallel configurations.

This paper focuses on two communication modes, namely Logically Instantaneity (li) and Causal Order (co). These communica- tion modes address two different levels of quality of service in message delivery. li means that it is possible to timestamp communication events with integers in such a way that (1) timestamps increase within each process and (2) the sending and the delivery events associated with each message have the same timestamp. So, there is a logical time frame in which for each message, the send event and the corresponding delivery events occur simultaneously. co means that when a process delivers a message m, its delivery occurs in a context where the receiving process knows all the causal past of m. Actually, Li is a property strictly stronger than co.The paper explores these noteworthy communication modes. Their main interest lies in the fact that they deeply simplify the design of message- passing programs that are intended to run on distributed memory parallel machines or cluster of workstations.KeywordsCausal OrderCluster of WorkstationsCommunication ProtocolDistributed MemoryDistributed SystemsLogical TimeLogical InstantaneityRendezvous

Pollutant dispersion models in the atmosphere can describe by Eulerian or Lagrangian approaches. Lagrangian models belong to the class of Monte Carlo methods. This type of method is very flexible, solving more complex problems, however this computational cost is greater than Eulerian models, as it is well established in the atmospheric pollutant and nuclear engineering communities. A parallel version of the Lagrangian particle model - LAMBDA - is developed using the MPI message passing communication library. Performance tests were executed in a distributed memory parallel machine, a multicomputer based on IA-32 architecture. Portions of the pollutant in the air are considered particles emitted from a pollutant source, evolving under stochastic forcing. This yields independent evolution equations for each particle of the model that can be executed by a different processor in a parallel implementation. Speed-up results show that the parallel implementation is suitable for the used architecture.

Towards the Parallelisation of Pressure Correction Method on Unstructured Grids

Communication Optimizations for Irregular Scientific Computations on Distributed Memory Architectures

A radiative transfer solver that implements the LTSn method was optimized and parallelized using the MPI message passing communication library. Timing and profiling information was obtained for the sequential code in order to identify performance bottlenecks. Performance tests were executed in a distributed memory parallel machine, a multicomputer based on IA-32 architecture. The radiative transfer equation was solved for a cloud test case to evaluate the parallel performance of the LTSn method. The LTSn code includes spatial discretization of the domain and Fourier decomposition of the radiances leading to independent azimuthal modes. This yields an independent radiative transfer equation for each mode that can be executed by a different processor in a parallel implementation. Speed-up results show that the parallel implementation is suitable for the used architecture.