Foundations for Program Development on CBE

Non-cancellative Boolean circuits: A generalization of monotone boolean circuits

We compare the measures of sequential time modelled using Turing machines and of parallel size modelled using Boolean circuits. This is done by constructing oracles which show certain relationships between complexity classes. An oracle B is shown for which Δ2P,B has 2n+o(n) size circuits relative to B. On the other hand, we give a C so that PC does not, for any k, have size nk circuits relative to C and yet NPC ≡ coNPC. These techniques can be combined to yield a D relative to which PD has 2n + o(n) size circuits but RD does not have size nk circuits for any k.

On the Circuit Complexity of Sigmoid Feedforward Neural Networks

We examine the power of nondeterministic finite automata with acceptance of an input word defined by a so called leaf language, i. e., a condition on the sequence of leaves in the automaton's computation tree. We study leaf languages either taken from one of the classes of the Chomsky hierarchy, or taken from a time- or space-bounded complexity class. We contrast the obtained results with those known for leaf languages for Turing machines and Boolean circuits.

For a d-ary Boolean function Φ: {0, 1}d → {0, 1} and an assignment to its variables x = (x1, x2, . . . , xd) we consider the problem of finding those subsets of the variables that are sufficient to determine the function value with a given probability δ. This is motivated by the task of interpreting predictions of binary classifiers described as Boolean circuits, which can be seen as special cases of neural networks. We show that the problem of deciding whether such subsets of relevant variables of limited size k ≤ d exist is complete for the complexity class NPPP and thus, generally, unfeasible to solve. We then introduce a variant, in which it suffices to check whether a subset determines the function value with probability at least δ or at most δ − γ for 0 < γ < δ. This promise of a probability gap reduces the complexity to the class NPBPP. Finally, we show that finding the minimal set of relevant variables cannot be reasonably approximated, i.e. with an approximation factor d1−α for α > 0, by a polynomial time algorithm unless P = NP. This holds even with the promise of a probability gap.

We define the counting classes NC/sup 1/, GapNC/sup 1/ PNC/sup 1/ and C/sub =/NC/sup 1/. We prove that Boolean circuits, algebraic circuits, programs over nondeterministic finite automata, and programs over constant integer matrices yield equivalent definitions of the latter three classes. We investigate closure properties. We observe that NC/sup 1//spl sube/L and that C/sub =/NC/sup 1//spl sube/L. Then we exploit our finite automaton model and extend the padding techniques used to investigate leaf languages. Finally, we draw some consequences from the resulting body of leaf language characterizations of complexity classes, including the unconditional separation of ACC/sup 0/ from MOD-PH as well as that of TC/sup 0/ from the counting hierarchy. Moreover we obtain that dlogtime-uniformity and logspace-uniformity for AC/sup 0/ coincide if and only if the polynomial time hierarchy equals PSPACE.

We study Boolean circuits as a representation of Boolean functions and consider different equivalence, audit, and enumeration problems. For a number of restricted sets of gate types (bases) we obtain efficient algorithms, while for all other gate types we show these problems are at least NP-hard.

While multicore hardware has become ubiquitous, explicitly parallel programming models and compiler techniques for exploiting parallelism on these systems have noticeably lagged behind. Stream programming is one model that has wide applicability in the multimedia, graphics, and signal processing domains. Streaming models execute as a set of independent actors that explicitly communicate data through channels. This paper presents a compiler technique for planning and orchestrating the execution of streaming applications on multicore platforms. An integrated unfolding and partitioning step based on integer linear programming is presented that unfolds data parallel actors as needed and maximally packs actors onto cores. Next, the actors are assigned to pipeline stages in such a way that all communication is maximally overlapped with computation on the cores. To facilitate experimentation, a generalized code generation template for mapping the software pipeline onto the Cell architecture is presented. For a range of streaming applications, a geometric mean speedup of 14.7x is achieved on a 16-core Cell platform compared to a single core.

Service-oriented computing and service-level orchestration through visual or graphical programming have been widely used in software and particularly in Web application development. Their further developments and fusing into Internet of Things, Robot as a Service (RaaS), cloud computing, and big data analysis have created the infrastructure for the information technologies to penetrate into all sectors of industry, service, and society. Figure 1 shows the stack of the related concepts, technologies, and applications. IoT service orchestration and visual programming separates the high-level composition from the low-level device programming, which allows the application builders to focus on the business logic, while the service providers to focus on efficient service development. This presentation will discuss the current service orchestration and visual programming environments for IoT application development. The presentation centralizes the topics around IoT. It starts with the upper layer of IoT, including service-oriented computing, Web software development, and service orchestration for system integration, which forms the foundation of cloud computing and a major data source of big data analysis. On the other hand, IoT standards and protocols are presented, which connect the service orchestration infrastructure tohardware and devices. As IoT applications and implementations, ADS (Autonomous Decentralized Systems), DPWS (Device Profile for Web Services), and Robot as a Service will be discussed. Different orchestration environments that have been widely used in industry and academic world will be presented, including BPEL in Oracle SOA Suite, Microsoft Workflow Foundation, Intel IoT Service Orchestration Layer, Microsoft Robotics Development Studio, MIT App Inventor, CMU Alice visual game programming environment, and ASU VIPLE (Visual IoT/Robotics Programming Environment). As a case study, VIPLE will be presented in more detail. VIPLE is a free software with open source IoT middleware developed at Arizona State University. VIPLE orchestration environment has an open interface to IoT devices. Any IoT device that can be programmed to interpret JSON objects and wrap sensory data into JSON objects can interface with VIPLE and be programmed using VIPLE orchestration environment. A demonstration will be given to show how an IoT application can be developed in minutes to control Intel Edison IoT device. VIPLE software, including orchestration environment, IoT middleware, full document, and lecture slides can be downloaded at: http://venus.eas.asu.edu/WSRepository/VIPLE.

The application of parallel computers to reservoir simulation is studied with the aid of two recently introduced Multiple Instruction, Multiple Data (MIMD) computers. The problems posed by reservoir simulators are shown to be particularly well suited for such parallel computers* Fundamental parallel programming techniques using Fortran are presented for the Hetrogeneous Element Processor (HEP) produced by Denelcor and the iPSC Hypercube produced by Intel. These techniques are then applied to two of the most time consuming tasks in reservoir simulation: forming the matrix coefficients and producing a sparse matrix solution. The problem of parallel formation of sparse matrix coefficients is addressed for the single and multiphase cases using both black oil and compositional fluid models. Parallel sparse matrix solution is addressed by considering D4 ordered Gaussian elimination, and the multigrid, conjugate gradient, and Successive Over-Relaxation (SOR) methods. Three parallel algorithms based on SOR are developed to illustrate the possible diversity of parallel algorithms. The red-black and multicolored SOR methods which are often used on vector machines are shown to be easily adaptable to parallel MIMD machine. Matrix partitioning and a new method of iteration pipelining are also presented as parallel SOR methods. The various SOR algorithms, in parallel form, are mapped onto the HEP to illustrate the speedups currently possible on an MIMD machine. Projections are made on the future importance of these types of machines to reservoir simulation.

Effective programming of parallel architectures has always been a challenge, and it is especially complicated with their modern diversity. The task of automatic parallelization of program code was formulated from the moment of the appearance of the first parallel computers made in Russia (for example, PS2000). To date, programming languages and technologies have been developed that simplify the work of a programmer (T-System, MC#, Erlang, Go, OpenCL), but do not make parallelization automatic. The current situation requires the development of effective programming tools for parallel computing systems. Such tools should support the development of parallel programs for systems with shared and distributed memory. The paper deals with the problem of automatic parallelization of affine programs for such systems. Methods for calculating space-time mappings that optimize the locality of the program are discussed. The implementation of developed methods is done in Haskell within the source-to-source translator performing automatic parallelization. A comparison of the performance of parallel variants of lu, atax, syr2k programs obtained using the developed tool and the modern Pluto tool is made. The experiments were performed on two x86_64 machines connected by the InfiniBand network. OpenMP and MPI were used as parallelization technologies. The performance of the resulting parallel program indicates the practical applicability of the developed tool for affine programs parallelization.

Programmers of sequential and parallel computers share the objective of developing fast, reliable programs as efficiently as possible. Already, considerable effort has been made to develop tools and techniques for sequential computers. Several approaches have been devised to meet the parallel programming challenge. Sequential languages have been adapted for parallel computers; new parallel languages designed; and tools developed which aid the programmer in detecting parallel code. A parallel programming assistant can aid the user in several ways: it can carry out tasks that are simple but mundane to the user, it can make comprehension easier for the user by summarizing program information; and it can cooperate with the user in doing a complex task, thus combining its own knowledge with that of the user's.

With multi-core processors in virtually every segment of the computing market, mainstream programmers need to routinely write parallel software. While progress has been made with automatic parallelization (see Chapter 9), experience with parallel computing has shown that in most cases, programmers will need to create parallel software “by hand.” This transition from serial to parallel software is a huge challenge for the software industry [Hwu08]; perhaps the biggest it has ever faced. While parallel computing is new to mainstream programmers, it has a long his-tory in scientific computing. In the 1980s, the scientific computing community was faced with a transition from vector supercomputers supported by vectorizing compilers to explicitly parallel massively parallel processors and clusters. Computer scientists responded by creating countless parallel programming languages, but these for the most part failed. Only a tiny fraction of programmers in the sciences took upparallel programming; and those that did tended to use low-level message passing libraries [MPI] that emphasized portability and control over elegance. New high-level languageswere not enough to solve the parallel programming prob-lem in the 1980s and 1990s, and there is no reason to believe it will be different this time as we help general purpose programmers adopt parallel programming. It is my hypothesis that the key to solving this problem is to focus on how experi-enced programmers think about parallel programming. Experienced parallel programmers, once they understand how the concurrency in their problem supports parallel execution, can use any sufficiently complete parallel programming language to create parallel software. Therefore, to solve the parallel programming problem for mainstream programmers, we need to capture how expert parallel programmers think and provide that understanding to mainstream programmers. In other words, we do not need new languages or sophisticated tools (though they help); we need to help mainstream programmers learn to “think parallel.” Learning to reason about parallel algorithms is a daunting task. Fortunately, wehave a few decades of experience to draw on. All we need to do is distill this collective wisdom into a form that we can put down in writing and communicate to people new to parallel computing. We believe design patterns are an effective tool to help us accomplish this task.

Recent trends have led hardware manufacturers to place multiple processing cores on a single chip, making parallel programming the intended way of taking advantage of the increased processing power. However, bringing concurrency to average programmers is considered to be one of the major challenges in computer science today. The difficulty lies not only in writing correct parallel programs, but also in achieving the required efficiency and performance. For parallel programs, performance is not only about obtaining low execution times on a fixed number of cores, but also about maintaining efficiency as the number of available cores is increased. Ideally, programmers should have in their toolkit techniques that can be used when designing parallel programs, before any code is available for testing on production hardware, and which are then able to predict scalability once the program is implemented. Existing methods are either unreliable at predicting scalability, such as the case of disjoint-access parallelism, or do not apply to lock-based programs, which currently make up a large part of existing concurrent programs. Furthermore, using some of these techniques is so complicated that it outweighs the time required to implement, debug and test the program on real hardware. In this thesis we study the problem of predicting the scalability of concurrent programs without implementing them. This allows programmers in the design phase of a concurrent algorithm to choose only one or a few promising solutions that will be implemented, debugged and tested on production hardware. We first consider disjoint-access parallelism, an existing property that applies only to a very restricted class of programs. After an extensive practical evaluation spanning across a variety of scenarios, we find it to be ineffective at predicting scalability. For predicting the scalability of more general concurrent algorithms, we propose the obstruction degree, a new scalability metric based on the consistency requirements of algorithms. It applies to programs using locks, invalidation primitives and transactional memory. Our metric allows programmers to compare two given algorithms as well as predict their scalability limit, the maximum number of processors to which they can scale, thus allowing programmers to choose the appropriate size hardware for running their programs. We also examine the composition of relaxed memory transactions in order to combine the ease of programming offered by transactional memory with the increased scalability of transactions that circumvent the traditional transactional model. We present outheritance, a property we show to be both necessary and sufficient for ensuring the correct composition of relaxed transactions, and we show how to calculate the obstruction degree of compositions that use this new property. We use outheritance to build OE-STM, a new software transactional memory algorithm having elastic transactions that correctly compose.

Clusters, composed of symmetric multiprocessor (SMP) machines and heterogeneous machines, have become increasingly popular for high‐performance computing. Message‐passing libraries, such as message‐passing interface (MPI) and parallel virtual machine (PVM), are de facto parallel programming libraries for clusters that usually consist of homogeneous and uni‐processor machines. For SMP machines, MPI is combined with multithreading libraries like POSIX Thread and OpenMP to take advantage of the architecture. In addition to existing parallel programming libraries that are in C/C++ and FORTRAN programming languages, the Java programming language presents itself as another alternative with its object‐oriented framework, platform neutral byte code, and ever‐increasing performance. This paper presents a new parallel programming model and a library, VCluster, which implements this model. VCluster is based on migrating virtual threads instead of processes to support clusters of SMP machines more efficiently. The implementation uses thread migration, which can be used in dynamic load balancing. VCluster was developed in pure Java, utilizing the portability of Java to support clusters of heterogeneous machines. Several applications are developed to illustrate the use of this library and compare the usability and performance of VCluster with other approaches. Copyright © 2007 John Wiley & Sons, Ltd.