9 - Parallel Processors

Horizon is a scalable shared-memory Multiple Instruction stream - Multiple Data stream (MIMD) computer architecture independently under study at the Supercomputing Research Center (SRC) and Tera Computer Company. It is composed of a few hundred identical scalar processors and a comparable number of memories, sparsely embedded in a three-dimensional nearest-neighbor network. Each processor has a horizontal instruction set that can issue up to three floating point operations per cycle without resorting to vector operations. Processors will each be capable of performing several hundred Million Floating Point Operations Per Second (FLOPS) in order to achieve an overall system performance target of 100 Billion (1011) FLOPS.This paper describes the architecture of the processor in the Horizon system. In the fashion of the Denelcor HEP, the processor maintains a variable number of Single Instruction stream - Single Data stream (SISD) processes, which are called instruction streams. Memory latency introduced by the large shared memory is hidden by switching context (instruction stream) each machine cycle. The processor functional units are pipelined to achieve high computational throughput rates; however, pipeline dependencies are hidden from user code. Hardware mechanisms manage the resources to guarantee anonymity and independence of instruction streams.

Mixed-mode parallel processors can quickly switch from executing instructions in single instruction, multiple data stream (SIMD) mode to executing them in multiple instruction, multiple data stream (MIMD) mode. To fully exploit this capability, the circumstances under which each mode should be used must be understood. This paper explores mixed-mode parallelism by presenting a general overview of the trade-offs between SIMD and MIMD modes, and then characterizing in detail three of the trade-offs concerning instruction execution performance. These include the effect of overlapping control unit and processor execution in SIMD mode, performance of conditional statements, and the consequences of switching parallelism modes.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Spatial and temporal data such as multiple data streams often have concept drift problems, which refers to changes of the data distributions over time. Once concept drift occurs, a stationary machine learning predictor will probably be invalid because the testing data have a different data distribution from that of the training data. Recent studies in data streams aim to address this issue by concept drift adaptation techniques. Concept drift adaptation methods update the predictor by time and have been validated to provide accurate real-time prediction for a single data stream. However, handling multiple relevant data streams is still an unsolved challenge in this area, considering many real-world applications generate multiple data streams that are simultaneously evolving. To fill this gap, we here present a predicting network for multiple data streams, named MuNet. MuNet leverages the dependency between data streams to efficiently lower down the computational cost of real-time predictions for multiple data streams. In MuNet, an online-learned Bayesian neural network (BNN) is designed as a connector between streams. The BNN-connector can continuously use the real-time information from only one base stream to correct the stationary predictor for the other streams, which avoids the high cost caused by repeated adaptation.

Extensive research has been done on extracting parallelism from single instruction stream processors. This paper presents our investigation into ways to modify MIMD architectures to allow them to extract the instruction level parallelism achieved by current superscalar and VLIW machines. A new architecture is proposed which utilizes the advantages of a multiple instruction stream design while addressing some of the limitations that have prevented MIMD architectures from performing ILP operation. A new code scheduling mechanism is described to support this new architecture by partitioning instructions across multiple processing elements in order to exploit this level of parallelism.

Algorithm for solving tridiagonal matrix problems in parallel

This paper presents a parallelization workflow in model-based development for a MIMD (Multiple Instruction streams, Multiple Data streams) processor with a vector accelerator. In the workflow, an Integer Linear Programming (ILP) formulation is proposed to deploy tasks extracted from the Simulink model. The proposed formulation can determine the number of processors that a task needs, and whether vectorization should be used to accelerate a task to reduce synchronization overhead and execution time. In the experiments, a MATLAB script was used to simulate the ILP formulation with randomly generated task graphs and evaluate its performance. Furthermore, we applied ILP task mapping to a real-world Model Predictive Control (MPC) application on DR1000C, a type of RISC-V MIMD processor with a vector accelerator. The maximum speedup of 10. 05x is obtained, which is promising for use in real-time embedded systems.

The classification of large amounts of data is a challenging problem that only a small number of classification algorithms can handle. In this paper we propose a Parallel Classification System based on an Ensemble of Mixture of Experts (PCEM). The system uses MIMD (Multiple Instruction and Multiple Data Stream) architecture, using a set of process that communicates via messages. PCEM is implemented using parallel schemes of traditional classifiers, for the mixture of experts, and using a parallel version of a Genetic Algorithm to implement a voting weighted criterion. The PCEM is a novel algorithm since it allows us to classify large amounts of data with low execution times and high performance measures, which makes it an excellent tool for in classification of large amounts of data. A series of tests were performed with well known databases that allowed us to measure how PCEM performs with many datasets and how well it does compared with other systems available.

The classification of large amounts of data is a challenging task in machine learning, which only some few classifiers can handle. In this paper we propose a Parallel Classifier based on Mixture of Experts (PCME)to handle this challenging task. The PCME is a novel algorithm since it allows us to classify large amounts of data with low execution times and performance measures (accuracy, lift, precision and recall) better than other classifiers. In previous work we saw that heterogeneous ensemble models were better but at a explores steep time cost. This work using parallel programming to overcome the time cost problem. We used the MIMD (Multiple Instruction and Multiple Data Stream) architecture, that uses a set of process that communicate via messages. PCME is implemented using parallel schemes of traditional classifiers, for the mixture of experts, and using also a parallel version of a Genetic Algorithm to implement a weighted voting criterion. We performed a series of tests with a set of databases that place the PCME as a very competitive classifier.

A PE (Processing Element) LSI for a DTW (Dynamic Time Warping) linear array processor has been designed. In designing this LSI, major effort has been focused on achieving regular data-flow among adjacent PEs maintaining pipelined operation in the array. A three data channel structure, a triple buffer structure and sophisticated control schemes make it possible for the designed LSI to carry out MIMD (Multiple Instruction and Multiple Data streams) and continuous pipelined DTW processing in sync with regular pattern input. Due to the high speed real time operation, and the versatile function of this PE-LSI, a high performance linear array processor can be constructed using a small number of PEs. The high speed operation of LSI is pursuited to achieve real time processing. Continuous speech recognition with an approximate vocabulary of 1000 words can be achieved using only 20 to 30 of these PE-LSIs.

Clustering multiple data streams has become an active area of research with many practical applications. Most of the early work in this area focused on one-sided clustering, i.e., clustering data streams based on feature correlation. However, recent research has shown that data streams can be grouped based on the distribution of their features, while features can be grouped based on their distribution across data streams. In this paper, an evolutionary clustering algorithm is proposed for multiple data streams using graph regularization non-negative matrix factorization (EC-NMF) in which the geometric structure of both the data and feature manifold is considered. Instead of directly clustering multiple data streams periodically, EC-NMF works in the low-rank approximation subspace and incorporates prior knowledge from historic results with temporal smoothness. Furthermore, we develop an iterative algorithm and provide convergence and correctness proofs from a theoretical standpoint. The effectiveness and efficiency of the algorithm are both demonstrated in experiments on real and synthetic data sets. The results show that the proposed EC-NMF algorithm outperforms existing methods for clustering multiple data streams evolving over time.

Sequential pattern mining is used to find frequent data sequences over time. When sequential patterns are generated, the newly arriving patterns may not be identified as frequent sequential patterns due to the existence of old data and sequences. Progressive sequential pattern mining aims to find the most up-to-date sequential patterns given that obsolete items will be deleted from the sequences. When sequences come with multiple data streams, it is difficult to maintain and update the current sequential patterns. Even worse, when we consider the sequences across multiple streams, previous methods cannot efficiently compute the frequent sequential patterns. In this work, we propose an efficient algorithm PSP-AMS to address this problem. PSP-AMS uses a novel data structure PSP-MS-tree to insert new items, update current items, and delete obsolete items. By maintaining a PSP-MS-tree, PSP-AMS efficiently finds the frequent sequential patterns across multiple streams. The experimental results show that PSP-AMS significantly outperforms previous algorithms for mining of progressive sequential patterns across multiple streams on synthetic data as well as real data.

In this paper, we present PAN, a generic middleware for distributed real-time complex event detection CED which is able to analyze multiple distributed data streams. In PAN, CED applications are defined as workflows and are executed by dedicated workers in a distributed way in a P2P network. In consequence, PAN is scalable in terms of the number of data streams and the complexity of the analyses. Evaluations based on an extended version of the ACM DEBS 2013 Grand Challenge scenario show the effectiveness and efficiency of PAN.

Parallel processing with minicomputers

The parallel implementation aspects of the Viterbi algorithm, which gives the maximum likelihood solution to decoding conventional codes, are studied. The interconnection of the trellis diagram of a binary convolutional code is similar to the shuffle exchange. This similarity is exploited to present a parallel version of the Viterbi algorithm. The proposed algorithm avoids tasks such as appending and exchanging the survivor lists that are considered pragmatic issues to be accommodated in implementing the Viterbi algorithm. This is done by introducing an additional back-tracking procedure which is almost identical to the forward path where branch and path metrics are computed. The parallel Viterbi algorithm is implemented on the connection machine which is a SIMD (single instruction stream and multiple data stream) parallel machine. >

The problem of quickly detecting the occurrence of an unusual event that happens on one of multiple independent data streams is considered. In the considered problem, all data streams at the initial are under normal state and are generated by probability distribution P <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</sub> . At some unknown time, an unusual event happens and the distribution of one data stream is modified to P <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> while the distributions of the rest remain unchange. The observer can only observe one data stream at one time. With his sequential observations, the observer wants to design an online stopping rule and a data stream switching rule to minimize the detection delay, namely the time difference between the occurrence of the unusual event and the time of raising an alarm, while keeping the false alarm rate under control. We model the problem under non-Bayesian quickest detection framework, and propose a detection procedure based on the CUSUM statistic. We show that this proposed detection procedure is asymptotically optimal.