Optimizing the MapReduce Framework for CPU-MIC Heterogeneous Cluster

We have setup a new course on the large scale data processing using clusters. It introduces the concepts and design of distributed systems. Many newly developed ideas such as Google file system and MapReduce programming framework for processing large scale data sets are introduced. Students will gain practical experience with distributed programming technologies via several small labs and one large multi-week final project. Labs and projects will be completed using Hadoop, an open-source implementation of Google's distributed file system and MapReduce programming model. We have taught this class named Mass Data Processing Technology on Large Scale Clusters for two years. This paper will describe the design, perform of the course as well as the experiences and lessons learned.

We present a novel 4096 complex-point, fully systolic VLSI FFT architecture based on the combination of three consecutive radix-4 stages resulting in a 64-point FFT engine. The outcome of cascading these 64-point FFT engines is an improved architecture that efficiently processes large input data sets in real time. Using 64-point FFT engines reduces the buffering and the latency to one third of a fully unfolded radix-4 architecture, while the radix-4 schema simplifies the calculations within each engine. The proposed 4096 complex point architecture has been implemented on a FPGA achieving a post-route clock frequency of 200 MHz resulting in a sustained throughput of 4096 point/20.48 μs. It has also been implemented on a high performance 0.13 μm, 1P8M CMOS process achieving a worst-case (0.9 V, 125 C) post-route clock frequency of 604.5 MHz and a sustained throughput of 4096 point/3.89 μs while consuming 4.4 W. The architecture is extended to accomplish FFT computations of 16K, 64K and 256K complex points with 352, 256 and 188 MHz operating frequencies respectively.

MapReduce is a distributed programming framework to process large scale data set by employing clusters in scale-out ways. However, scaling-up the single node is better than scale-out solution because of less communication overhead. As Intel MIC has a higher performance than ordinary CPU, we propose an efficient MapReduce framework for Intel MIC cluster. Our framework provides several new features, such as fault tolerant mechanism for MIC management, efficient buffer management in MIC memory, and asynchronous task transfer between CPU and MIC. It could manage a large scale MIC cluster and exploit applications in MapReduce like ways. The experimental results show that our system is up to 1.35x and 6.8x faster than Hadoop on ordinary CPU cluster.KeywordsMapReduceHadoopMany integrated coreBig data

Clustering is useful for mining the underlying structure of a dataset in order to support decision making since target or high-risk groups can be identified. However, for high dimensional datasets, the result of traditional clustering methods can be meaningless as clusters may only be depicted with respect to a small part of features. Taking customer datasets as an example, certain customers may correlate with their salary and education, and the others may correlate with their job and house location. If one uses all the features of a customer for clustering, these local-correlated clusters may not be revealed. In addition, processing high dimensions and large datasets is a challenging problem in decision making. Searching all the combinations of every feature with every record to extract local-correlated clusters is infeasible, which is in exponential scale in terms of data dimensionality and cardinality. In this paper, we propose a scalable 2-Leveled Approximated Hyper-Image-based Clustering framework, referred as 2L-HIC-A, for mining local-correlated clusters, where each level clustering process requires only one scan of the original dataset. Moreover, the data-processing time of 2L-HIC-A can be independent of the input data size. In 2L-HIC-A, various well-developed image processing techniques can be exploited for mining clusters. In stead of proposing a new clustering algorithm, our framework can accommodate other clustering methods for mining local-corrected clusters, and to shed new light on the existing clustering techniques.

3D rendering is a kind of application which is not only data intensive but also computation intensive and we can render different frames in parallel. For a 3D rendering system, the I/O performance and the ability to handle the failures of both hardware and software are very important. Traditional method uses OpenPBS to schedule the rendering tasks and puts the data on a NFS data server. But NFS data server can very easily be the bottleneck of the system and OpenPBS can't reschedule the failing tasks, so new method is required to solve those problems. MapReduce is a programming model serving for processing large scale data sets in a parallel manner and hadoop is an open source implementation of the MapReduce programming model. In this paper we propose a method which uses hadoop to do the 3D rendering work in parallel. The proposed method is based on hadoop-0.20.2, evaluation on our method and the traditional method shows that for a scene with 40 frames, our method can reduce the execution time more significantly than traditional method as the number of rendering nodes increases, and our method can also handle failures of both the rendering node and rendering program.

It is our great pleasure to welcome you to the 2013 ACM International Workshop on Energy Efficient High Performance Parallel and Distributed Computing - EEHPDC'1). This workshop aims to provide a timely forum for presenting novel ideas and latest findings as well as to discuss research directions and challenges related to Energy Efficient for High Performance Computing Parallel and Distributed Computing. EEHPDC'13 gives researchers and practitioners a unique opportunity to share their perspectives with others interested in the various aspects of the design, implementation, evaluation, and the use of parallel and distributed systems for high-end computing with the focus on energy efficiency. The program comprises a keynote speech by Laurent Lefevre on the impact of application and service knowledge for energy efficiency in HPC, a technical session with technical papers on energy efficient communications, cloud computing systems, task scheduling, and multi-core and heterogeneous architectures, and an industry session. We hope that these proceedings will serve as a valuable reference for energy efficiency researchers and developers.

3 Introduction 4 Background 4 Applications 5 Systems-Level Investigations 13 Selected References and Important Web Sites 27

Power efficiency - performance relative to power - is one of the most important concerns when designing RADAR processing systems. This paper analyzes power and performance trade-offs for a typical Space Time Adaptive Processing (STAP) application. We study STAP implementations for CUDA and OpenMP on two architectures, Intel Haswell Core I7-4770TE and NVIDIA Kayla with a GK208 GPU. We analyze the power and performance of STAP's computationally intensive kernels across the two hardware testbeds. We discuss an efficient parallel implementation for the Haswell CPU architecture. We also show the impact and trade-offs of GPU optimization techniques. The GPU architecture is able to process large size data sets without increase in power requirement. The use of shared memory has a significant impact on the power requirement for the GPU. Finally, we show that a balance between the use of shared memory and main memory access leads to an improved performance in a typical STAP application.

Direct measurement of the thickness of mountain glaciers is difficult over large areas, yet knowledge of the thickness is essential for calculating their volumes and future evolution. We develop a new method for estimating the ice thickness along glacier flow lines, using the “perfect‐plasticity” rheological assumption that relates the thickness and surface slope to a yield stress. Previous studies have used this assumption with the shallow‐ice approximation to estimate the ice thickness, but the standard approach neglects the effect of side drag on glacier stress balance. Our method addresses this shortcoming and extends the standard method by accounting for the side drag via the glacier width. Besides the assumed yield stress, the inputs for our method are the outline and surface topography of the glacier; surface velocity and mass balance data are unnecessary. We validated the extended method on five glaciers in northwest China where thickness data are available from radio echo soundings, finding that it can reproduce measured thicknesses with a mean absolute error of 11.8% (like the standard method). Moreover, for long glacier tongues confined to flow between parallel valley sides, this method is found to give more accurate thickness estimates than does the standard method, with a mean absolute error of as low as 5.3%. Sensitivity analysis shows that the estimated ice thickness depends strongly on yield stress and surface slope and less strongly on glacier width. Because this method is physically more realistic than the standard method and its inputs are easily derivable from remote‐sensing observations, it has the potential to be used for processing large glacier data sets.

With a lot of medical data coming from various sources, steer decisions can be made from the insights expand through big data by using various Machine Learning Algorithms. Traditionally, physicians use their knowledge while making treatment assessments, but in the last few years there has been a shift towards evidence-based medicine. This involves systematical appraisal of clinical data and making treatment decisions based on the finest accessible information. In this paper, we apply the predictive analysis technique in Hadoop/Map Reduce background to predict and classify the type of diabetes. And also focuses on how MapReduce is used, how map and reduce evaluations are adapted, implemented in various situations such as in medical field to produce medical reports by processing large medical data sets. This system provides capable way to care and cure the patients at low cost with improved outcomes like affordability and accessibility.

While resource management and task scheduling are identified challenges of Grid computing, current Grid scheduling systems mainly focus on CPU and network availability. Recent performance improvement of CPU and computer network has made memory usage a significant factor of overall performance. In this study, we consider memory availability as a performance factor and introduce memory conscious task partition and scheduling. Three task partition policies are discussed. They are CPU-based, memory-based, and CPU-memory combined partition. We first investigate the three task partition policies on dedicated resources and verify the effectiveness of the CPU-memory combined partition algorithm in finding an optimal solution. We then extend the task partition policies in non-dedicated environments with the consideration of resource sharing. Analytical and experimental results show that the CPU-memory combined scheduling approach outperforms either the CPU-based or memory-based scheduling approach considerably for memory-intensive applications in Grid environments.

The massive growth of cloud computing leads to huge amounts of energy consumption and release of carbon footprints as data centers are housed by a large number of servers. Consequently, the cloud service providers are looking for eco-friendly solutions to reduce energy consumption and carbon emissions. As a result, task scheduling has drawn attention, in which efficient resource utilization and minimum energy consumption take into great consideration. This is an exigent issue, especially for the heterogeneous environment. In this work, we put forward an energy-efficient task scheduling algorithm (ETSA) to address the demerits associated with task consolidation and scheduling. The proposed algorithm ETSA takes into account the completion time and total utilization of a task on the resources, and follows a normalization procedure to make a scheduling decision. We evaluate the proposed algorithm ETSA to measure energy efficiency and makespan in the heterogeneous environment. The experimental results are compared with recent algorithms, namely random, round robin, dynamic cloud list scheduling, energy-aware task consolidation, energy-conscious task consolidation and MaxUtil. The proposed algorithm ETSA provides an elegant trade-off between energy efficiency and makespan than the existing algorithms.

Biclustering refers to simultaneous clustering of objects and their features. Use of biclustering is gaining momentum in areas such as text mining, gene expression analysis and collaborative filtering. Due to requirements for high performance in large scale data processing applications such as Collaborative filtering in E-commerce systems and large scale genome-wide gene expression analysis in microarray experiments, a high performance prallel/distributed solution for biclustering problem is highly desirable. Recently, Ahmad et al [1] showed that Bipartite Spectral Partitioning, which is a popular technique for biclustering, can be reformulated as a graph drawing problem where objective is to minimize Hall's energy of the bipartite graph representation of the input data. They showed that optimal solution to this problem is achieved when nodes are placed at the barycenter of their neighbors. In this paper, we provide a parallel algorithm for biclustering based on this formulation. We show that parallel energy minimization using barycenter heuristic is embarrassingly parallel. The challenge is to design a bi-cluster identification algorithm which is scalable as well as accurate. We show that our parallel implementation is not just extremely scalable, it is comparable in accuracy as well with serial implementation. We have evaluated proposed parallel biclustering algorithm with large synthetic data sets on upto 256 processors. Experimental evaluation shows large superlinear speedups, scalability and high level of accuracy.

Big data is one of the important development direction of modern information technology, realizing the sharing and analysis of large data will bring immeasurable economic value, but also has a tremendous role in promoting the social. In the age of big data, unified the data representation, large data processing, query, analysis and visualization are the key problem to be solved urgently. In order to provide a standardized framework construction of the large data service platform, this paper designed a large data service oriented architecture user experience. Secondly, in the aspect of data model, in order to achieve high data service for non structured data, the design of the non structured data model based on subject behavior. In large data service model, algebraic model large data services and their composition was established by using process algebra. In large data service applications, detailed retrieval, process analysis and visualization services, and by improving the retrieval accuracy and efficiency of the service in two aspects of measures to achieve the high data service optimization.

As the attention given to big data grows, cluster computing systems for distributed processing of large data sets become the mainstream and critical requirement in high performance distributed system research. One of the most successful systems is Hadoop, which uses MapReduce as a programming/execution model and takes disks as intermedia to process huge volumes of data. Spark, as an in-memory computing engine, can solve the iterative and interactive problems more efficiently. However, currently it is a consensus that they are not the final solutions to big data due to a MapReduce-like programming model, synchronous execution model and the constraint that only supports batch processing, and so on. A new solution, especially, a fundamental evolution is needed to bring big data solutions into a new era. In this paper, we introduce a new cluster computing system called HAMR which supports both batch and streaming processing. To achieve better performance, HAMR integrates high performance computing approaches, i.e. dataflow fundamental into a big data solution. With more specifications, HAMR is fully designed based on in-memory computing to reduce the unnecessary disk access overhead; task scheduling and memory management are in fine-grain manner to explore more parallelism; asynchronous execution improves efficiency of computation resource usage, and also makes workload balance across the whole cluster better. The experimental results show that HAMR can outperform Hadoop MapReduce and Spark by up to 19x and 7x respectively, in the same cluster environment. Furthermore, HAMR can handle scaling data size well beyond the capabilities of Spark.