Data Parallel Programming Model for Many-Core Architectures

An Approach to Correctness of Data Parallel Algorithms

Many high performance applications run well below the peak arithmetic performance of the underlying machine, with inefficiencies often attributed to poor memory system behavior. In the context of scientific computing we examine three emerging processors designed to address the well-known gap between processor and memory performance through the exploitation of data parallelism. The VIRAM architecture uses novel PIM technology to combine embedded DRAM with a vector co-processor for exploiting its large bandwidth potential. The DIVA architecture incorporates a collection of PIM chips as smart-memory coprocessors to a conventional microprocessor, and relies on superword-level parallelism to make effective use of the available memory bandwidth. The Imagine architecture provides a stream-aware memory hierarchy to support the tremendous processing potential of SIMD controlled VLIW clusters. First we develop a scalable synthetic probe that allows us to parametize key performance attributes of VIRAM, DIVA and Imagine while capturing the performance crossover points of these architectures. Next we present results for scientific kernels with different sets of computational characteristics and memory access patterns. Our experiments allow us to evaluate the strategies employed to exploit data parallelism, isolate the set of application characteristics best suited to each architecture and show a promising direction towards interfacing leading-edge processor technology with high-end scientific computations.

The computer industry is at a major inflection point in its hardware roadmap due to the end of a decades-long trend of exponentially increasing clock frequencies. Instead, future computer systems are expected to be built using homogeneous and heterogeneous many-core processors with 10’s to 100’s of cores per chip, and complex hardware designs to address the challenges of concurrency, energy efficiency and resiliency. Unlike previous generations of hardware evolution, this shift towards many-core computing will have a profound impact on software. These software challenges are further compounded by the need to enable parallelism in workloads and application domains that traditionally did not have to worry about multiprocessor parallelism in the past. A recent trend in mainstream desktop systems is the use of graphics processor units (GPUs) to obtain order-of-magnitude performance improvements relative to general-purpose CPUs. Unfortunately, hybrid programming models that support multithreaded execution on CPUs in parallel with CUDA execution on GPUs prove to be too complex for use by mainstream programmers and domain experts, especially when targeting platforms with multiple CPU cores and multiple GPU devices. In this paper, we extend past work on Intel’s Concurrent Collections (CnC) programming model to address the hybrid programming challenge using a model called CnC-CUDA. CnC is a declarative and implicitly parallel coordination language that supports flexible combinations of task and data parallelism while retaining determinism. CnC computations are built using steps that are related by data and control dependence edges, which are represented by a CnC graph. The CnC-CUDA extensions in this paper include the definition of multithreaded steps for execution on GPUs, and automatic generation of data and control flow between CPU steps and GPU steps. Experimental results show that this approach can yield significant performance benefits with both GPU execution and hybrid CPU/GPU execution.

Multimedia and DSP applications have been widely present in embedded devices. Due to their intrinsic nature, such application domains are benefited from Data Level Parallelism (DLP) exploitation, which is mostly employed in current embedded platforms by using vectorization techniques extending the underlying ISA. However, such strategy relies on specific library which affects software productivity and compiler support, such as ARM auto-vectorization approach, which breaks binary compatibility. This work proposes a transparent Dynamic SIMD Assembler (DSA) that is capable of detecting vectorizable code regions at runtime without requiring specific library or compilers. As a case study, we coupled the DSA to a 128-bit wide ARM NEON Engine. Results show that the proposed approach shows performance improvements of 31% over the original execution (without DLP exploitation). In addition, Dynamic SIMD Assembler, besides keeping binary compatibility, outperforms ARM auto-vectorization technique in 6%.

CEP is a widely used technique for the reliability and recognition of arbitrarily complex patterns in enormous data streams with great performance in real time. Real-time detection of crucial events and rapid response to them are the key goals of sophisticated event processing. The performance of event processing systems can be improved by parallelizing CEP evaluation procedures. Utilizing CEP in parallel while deploying a multi-core or distributed environment is one of the most popular and widely recognized tackles to accomplish the goal. This paper demonstrates the ability to use an unusual parallelization strategy to effectively process complicated events over streams of data. This method depends on a dual-tier hybrid paradigm that combines several parallelism levels. Thread-level or task-level parallelism (TLP) and Data-level parallelism (DLP) were combined in this research. Many threads or instruction sequences from a comparable application can run concurrently under the TLP paradigm. In the DLP paradigm, instruc-tions from a single stream operate on several data streams at the same time. In our suggested model, there are four major stages: data mining, pre-processing, load shedding, and optimization. The first phase is online data mining, following which the data is materialized into a publicly available solution that combines a CEP engine with a library. Next, data pre-processing encompasses the efficient adaptation of the content or format of raw data from many, perhaps diverse sources. Finally, parallelization approaches have been created to reduce CEP processing time. By providing this two-type parallelism, our proposed solution combines the benefits of DLP and TLP while addressing their constraints. The JAVA tool will be used to assess the suggested technique. The performance of the suggested technique is compared to that of other current ways for determining the efficacy and efficiency of the proposed algorithm.

Abstract. Thanks to the nature of the graphics processing, the newly released products offer highly parallel processing units with high-memory bandwidth and computational power of more than teraflops per second. The modern GPUs are not only powerful graphic engines but also they are high level parallel programmable processors with very fast computing capabilities and high-memory bandwidth speed compared to central processing units (CPU). Data-parallel computations can be shortly described as mapping data elements to parallel processing threads. The rapid development of GPUs programmability and capabilities attracted the attentions of researchers dealing with complex problems which need high level calculations. This interest has revealed the concepts of “General Purpose Computation on Graphics Processing Units (GPGPU)” and “stream processing”. The graphic processors are powerful hardware which is really cheap and affordable. So the graphic processors became an alternative to computer processors. The graphic chips which were standard application hardware have been transformed into modern, powerful and programmable processors to meet the overall needs. Especially in recent years, the phenomenon of the usage of graphics processing units in general purpose computation has led the researchers and developers to this point. The biggest problem is that the graphics processing units use different programming models unlike current programming methods. Therefore, an efficient GPU programming requires re-coding of the current program algorithm by considering the limitations and the structure of the graphics hardware. Currently, multi-core processors can not be programmed by using traditional programming methods. Event procedure programming method can not be used for programming the multi-core processors. GPUs are especially effective in finding solution for repetition of the computing steps for many data elements when high accuracy is needed. Thus, it provides the computing process more quickly and accurately. Compared to the GPUs, CPUs which perform just one computing in a time according to the flow control are slower in performance. This structure can be evaluated for various applications of computer technology. In this study covers how general purpose parallel programming and computational power of the GPUs can be used in photogrammetric applications especially direct georeferencing. The direct georeferencing algorithm is coded by using GPGPU method and CUDA (Compute Unified Device Architecture) programming language. Results provided by this method were compared with the traditional CPU programming. In the other application the projective rectification is coded by using GPGPU method and CUDA programming language. Sample images of various sizes, as compared to the results of the program were evaluated. GPGPU method can be used especially in repetition of same computations on highly dense data, thus finding the solution quickly.

This paper describes the sourcing, processing, and application of parallel text data for Icelandic and Polish for the purpose of bilingual lexicon induction (BLI), demonstrating how a parallel corpus can be compiled for a low-to-medium resource language pair that has no available parallel data, by pivoting through a common language. We show the usefulness of the corpus by training and evaluating a machine translation (MT) model on the data. Iceland's linguistic landscape is evolving, with an increasing need for multilingual support due to the growing immigrant population. Polish, in particular, stands out as the language of the largest single minority in Iceland, underscoring the importance of this project.

The performance of many digital systems today is limited by the interconnection bandwidth between chips. Although the processing performance of a single chip has increased dramatically since the inception of the integrated circuit technology, the communication bandwidth between chips has not enjoyed as much benefit. Most CMOS chips, when communicating off-chip, drive unterminated lines with full-swing CMOS drivers. Such full-swing CMOS interconnect ring-up the line, and hence has a bandwidth that is limited by the length of the line rather than the performance of the semiconductor technology. Thus, as VLSI technology scales, the pin bandwidth does not improve with the technology, but rather remains limited by board and cable geometry, making off-chip bandwidth an even more critical bottleneck. In order to increase the I/O Bandwidth, some efficient high speed signaling standard must be used which considers the line termination, signal integrity, power dissipation, noise immunity etc In this work, a transmitter has been developed for high speed offchip communication. It consists of low speed input buffer, serializer which converts parallel input data into serial data and a current mode driver which converts the voltage mode input signals into current over the transmission line. Output of 32 low speed input buffers is fed to two serializer, each serializer converting 16 bit parallel data into serial data stream. Output of two serializers is fed to LVDS current mode driver. The serial link technique used in this work is the time division multiplex (TDM) and point-to-point technique. It means that the low-speed parallel signals are transferred to the high-speed serial signal at the transmitter end and the high-speed serial signal is transferred to the low-speed parallel signals at the receiver end. Serial link is the design of choice in any application where the cost of the communication channel is high and duplicating the links in large numbers is uneconomical.

This paper analyzes general parallel programming models and identifies the general parallel programming models stack used for high-performance numerical simulation. Then, on the basis of this analysis, domain-specific parallel programming models are presented and their main constitution, including data structures, computational patterns, component models, and programming frameworks, discussed. The inherent relationship among these elements is also analyzed. The J Adaptive Structured Mesh application INfrastructure (JASMIN) framework is used to verify and validate the possibility and effectiveness of these models. Domain-specific parallel programming models will significantly improve the development efficiency of parallel application software.

Aufgrund fallender Preise zur Speicherung von Daten kann man derzeit eine explosionsartige Zunahme in der Menge der verfügbaren Daten beobachten. Diese Entwicklung gibt Unternehmen und wissenschaftliche Institutionen die Möglichkeit empirische Daten in ungekannter Größenordnung zu analysieren. Für viele Firmen ist die Analyse der gesammelten Daten aus ihrem operationalen Geschäft längst zu einem zentralen strategischen Aspekt geworden. Im Gegensatz zu der seit längerem schon betriebenen Business Intelligence, bestehen diese Analysen nicht mehr nur aus traditionellen relationalen Anfragen. In zunehmendem Anteil kommen komplexe Algorithmen aus den Bereichen Data Mining und Maschinelles Lernen hinzu, um versteckte Muster in den Daten zu erkennen, oder Vorhersagemodelle zu trainieren. Mit zunehmender Datenmenge und Komplexität der Analysen wird jedoch eine neue Generation von Systemen benötigt, die diese Kombination aus Anfragekomplexität und Datenvolumen gewachsen sind. Relationale Datenbanken waren lange Zeit das Zugpferd der Datenanalyse im großen Stil. Grund dafür war zum großen Teil ihre deklarativen Anfragesprache, welche es ermöglichte die logischen und physischen Aspekte der Datenspeicherung und Verarbeitung zu trennen, und Anfragen automatisch zu optimieren. Das starres Datenmodell und ihre beschränkte Menge von möglichen Operationen schränken jedoch die Anwendbarkeit von relationalen Datenbanken für viele der neueren analytischen Probleme stark ein. Diese Erkenntnis hat die Entwicklung einer neuen Generation von Systemen und Architekturen eingeläutet, die sich durch sehr generische Abstraktionen für parallelisierbare analytische Programme auszeichnen; MapReduce kann hier beispielhaft genannt werden, als der zweifelsohne prominenteste Vertreter dieser Systeme. Zwar vereinfachte und erschloss diese neue Generation von Systemen die Datenanalyse in diversen neuen Anwendungsfeldern, sie ist jedoch nicht in der Lage komplexe Anwendungen aus den Bereichen Data Mining und Maschinelles Lernen effizient abzubilden, ohne sich dabei extrem auf spezifische Anwendungen zu spezialisieren. Verglichen mit den relationalen Datenbanken haben MapReduce und vergleichbare Systeme außerdem die deklarative Abstraktion aufgegeben und zwingen den Anwender dazu systemnahe Programme zu schreiben und diese manuell zu optimieren. In dieser Dissertation werden verschiedene Techniken vorgestellt, die es ermöglichen etliche der zentralen Eigenschaften von relationalen Datenbanken im Kontext dieser neuen Generation von daten-parallelen Analysesystemen zu realisieren. Mithilfe dieser Techniken ist es möglich ein Analysesystem zu beschreiben, dessen Programme gleichzeitig sowohl generische und ausdrucksstark, als auch prägnant und deklarativ sind. Im einzelnen stellen wir folgende Techniken vor: Erstens, eine Programmierabstraktion die generisch ist und mit komplexen Datenmodellen umgehen kann, aber gleichzeitig viele der deklarativen Eigenschaften der relationalen Algebra erhält. Programme, die gegen dies Abstraktion entwickelt werden können ähnlich optimiert werden wie relationale Anfragen. Zweitens stellen wir eine Abstraktion für iterative daten-parallele Algorithmen vor. Die Abstraktion unterstützt inkrementelle (delta-basierte) Berechnungen und geht mit zustandsbehafteteten Berechnungen transparent um. Wir beschreiben wie man einen relationalen Anfrageoptimierer erweitern kann so dass dieser iterative Anfragen effektiv optimiert. Wir zeigen dabei dass der Optimierer dadurch in die Lage versetzt wird automatisch Ausführungspläne zu erzeugen, die wohlbekannten, manuell erstellten Programmen entsprechen. Die Abstraktion subsumiert dadurch spezialisierte Systeme (wie Pregel) und bietet vergleichbare Performanz. Drittens stellen wir Methoden vor, um die Programmierabstraktion in eine funktionale Sprachen einzubetten. Diese Integration ermögliche es prägnante Programme zu schreiben und einfach wiederzuverwendenden Komponenten und Bibliotheken, sowie Domänenspezifische Sprachen, zu erstellen. Wir legen dar wie man die Übersetzung und Optimierung des daten-parallelen Programms mit dem Sprachübersetzer der funktionalen Sprache so integriert, dass maximales Optimierungspotenzial besteht.

GPUs offer a powerful bulk synchronous programming model for exploiting data parallelism; however, branch divergence amongst executing warps can lead to serious performance degradation due to execution serialization. We propose a novel profile guided approach to optimize branch divergence while transforming a serial program to a data-parallel program for GPUs. Our approach is based on the observation that branches inside some data parallel loops although divergent, exhibit repetitive regular patterns of outcomes. By exploiting such patterns, loop iterations can be aligned so that the corresponding iterations traverse the same branch path. These aligned iterations when executed as a warp in a GPU, become convergent. We propose a new metric based on the repetitive pattern characteristics that indicates whether a data-parallel loop is worth restructuring. When tested our approach on the well-known Rodinia benchmark, we found that it is possible to achieve upto 48% performance improvement by loop restructuring suggested by the patterns and our metrics.

We present an accurate online scalability prediction model for data-parallel programs on NUMA many-core systems. Memory contention is considered to be the major limiting factor of program scalability as data parallelism limits the amount of synchronization or data dependencies between parallel work units. Reflecting the architecture of NUMA systems, contention is modeled at the last-level caches of the compute nodes and the memory nodes using a two-level queuing model to estimate the mean service time of the individual memory nodes. Scalability predictions for individual or co-located parallel applications are based solely on data obtained during a short sampling period at runtime; this allows the presented model to be employed in a variety of scenarios. The proposed model has been implemented into an open-source OpenCL and the GNU OpenMP runtime and evaluated on a 64-core AMD system. For a wide variety of parallel workloads and configurations, the evaluations show that the model is able to predict the scalability of data-parallel kernels with high accuracy.

In the situation of monitoring an experiment, it is often necessary to control several independently running tasks from one graphical user interface (GUI). Such a GUI must be able to execute commands in the tasks even if they are busy, i.e., getting data, analyzing data, or waiting for data. Moreover, the tasks, being controlled by data streams (i.e., event data samples or slow control data), must be able to send data asynchronously to the GUI for visualization. A multitasking package (C++ class library) that meets these demands has been developed at the Gesellschaft fur Schwerionenforschung (GSI), Darmstadt, Germany, in the framework of a new analysis system, Go4, which is based on the ROOT system [CERN, R. Brun et al]. The package provides a thread manager, a task handler, and asynchronous intertask communication between threads through sockets. Hence, objects can be sent at any time from a task to the GUI or vice versa. At the GUI side, an incoming object is accepted by a thread and processed. In a task, an incoming command is queued by the accepting thread and executed in the execution thread. Utilizing the package one can implement nonblocking GUIs to control one or several tasks processing data in parallel and updating graphical elements in the GUI. The package could also be useful in building data dispatchers or in slow control applications. All components have been tested with Go4 analysis tasks and a very preliminary GUI.

We present a fast collision detection method for deformable surfaces with parallel spatial hashing on GPU architecture. The efficient update and access of the uniform grid are exploited to accelerate the performance in our method. To deal with the inflexible memory system, which makes the building of stream data a challenging task on GPU, we propose to subdivide the whole workload into irregular segments and design an efficient evaluation algorithm, which employs parallel scan and stream compaction, to build the stream data in parallel. The load balancing is a key aspect that needs to be considered in the SIMD parallelism. We break the heavy and irregular collision computation down into lightweight part and heavyweight part, ensuring the later perfectly run in load balancing manner with each concurrent thread processes just a single collision. In practice, our approach can perform collision detection in tens of milliseconds on a PC with NVIDIAGTX 260 graphics card on benchmarks composed of millions of triangles. The results highlight our speedups over prior CPU-based and GPU-based algorithms.

Research on programming distributed memory multiprocessors has resulted in a well‐understood programming model, namely data‐parallel programming. However, data‐parallel programming in a multithreaded environment is far less understood. For example, if multiple threads within the same process belong to different data‐parallel computations, then the architecture, compiler, or run‐time system must ensure that relative indexing and collective operations are handled properly and efficiently. We introduce a run‐time‐based solution for data‐parallel programming in a distributed memory environment that handles the problems of relative indexing and collective communications among thread groups. As a result, the data‐parallel programming model can now be executed in a multithreaded environment, such as a system using threads to support both task and data parallelism.