Adaptively Accelerating Map-Reduce/Spark with GPUs: A Case Study

With the advancement in technology, a huge amount of data containing useful information, called Big Data, is generated on a daily basis. For processing such tremendous volume of data, there is a need of Big Data frameworks such as Hadoop MapReduce, Apache Spark etc. Among these, Apache Spark performs up to 100 times faster than conventional frameworks like Hadoop Mapreduce. For the effective analysis and interpretation of this data, scalable Machine Learning methods are required to overcome the space and time bottlenecks. Partitional clustering algorithms are widely adopted by researchers for clustering large datasets due to their low computational requirements. Thus, we focus on the design of partitional clustering algorithm and its implementation on Apache Spark. In this paper, we propose a partitional based clustering algorithm called Scalable Random Sampling with Iterative Optimization Fuzzy c-Means algorithm (SRSIO-FCM) which is implemented on Apache Spark to handle the challenges associated with Big Data Clustering. Experimentation is performed on several big datasets to show the effectiveness of SRSIO-FCM in comparison with a proposed scalable version of the Literal Fuzzy c-Means (LFCM) called SLFCM implemented on Apache Spark. The comparative results are reported in terms of value of F-measure, ARI, Objective function, Run-time and Scalability. The reported results show the great potential of SRSIO-FCM for Big Data clustering.

Size of the data used by enterprises has been growing at exponential rates since last few years; handling such huge data from various sources is a challenge for Businesses. In addition, Big Data becomes one of the major areas of research for Cloud Service providers due to a large amount of data produced every day, and the inefficiency of traditional algorithms and technologies to handle these large amounts of data. In order to resolve the aforementioned problems and to meet the increasing demand for high-speed and data-intensive computing, several solutions have been developed by researches and developers. Among these solutions, there are Cloud Computing tools such as Hadoop MapReduce and Apache Spark, which work on the principles of parallel computing. This chapter focuses on how big data processing challenges can be handled by using Cloud Computing frameworks and the importance of using Cloud Computing by businesses

The Resilient Distributed Dataset (RDD) cache is an important way to improve the efficiency of application on Spark. There are some specific characteristics of iterative application. We design the classified and active caching strategy for iterative application. It can reduce the time delay of RDD creation of data set for iterative application and avoid creating or recomputing it again during the iterative application. At the same time, the RDD of parameter set should be renewed and cached after every iteration to reduce the time overhead of the subsequent iteration. The prototype is implemented based on Spark and one iterative application is used to test and compared with Spark. The results show that the classified and active caching strategy can reduce the 12%-45% time overhead of iterative application.

Apache Spark is a data processing engine for large data sets. Apache Spark is much faster (up to 100 times faster in memory) than Apache Hadoop MapReduce. In cluster mode, Spark applications run as independent processes coordinated by the SparkContext object in the driver program, which is the main program. The SparkContext may connect to several types of cluster managers to allocate resources to Spark applications. The supported cluster managers include the Standalone cluster manager, Mesos and YARN. Apache Spark is designed to access data from varied data sources including the HDFS, Apache HBase and NoSQL databases such as Apache Cassandra and MongoDB. In this chapter we shall use the same CDH Docker image that we used for several of the Apache Hadoop frameworks including Apache Hive and Apache HBase. We shall run an Apache Spark Master in cluster mode using the YARN cluster manager in a Docker container.

The field of data analytics is currently going through a renaissance as a result of ever-increasing dataset sizes, the value of the models that can be trained from those datasets, and a surge in flexible, distributed programming models. In particular, the Apache Hadoop and Spark programming systems, as well as their supporting projects (e.g. HDFS, SparkSQL), have greatly simplified the analysis and transformation of datasets whose size exceeds the capacity of a single machine. While these programming models facilitate the use of distributed systems to analyze large datasets, they have been plagued by performance issues. The I/O performance bottlenecks of Hadoop are partially responsible for the creation of Spark. Performance bottlenecks in Spark due to the JVM object model, garbage collection, interpreted/managed execution, and other abstraction layers are responsible for the creation of additional optimization layers, such as Project Tungsten. Indeed, the Project Tungsten issue tracker states that the "majority of Spark workloads are not bottlenecked by I/O or network, but rather CPU and memory".

Big data is the biggest challenges as we need huge processing power system and good algorithms to make an decision. We need Hadoop environment with pig hive, machine learning and hadoopecosystem components. The data comes from industries. Many devices around us and sensor, and from social media sites. According to McKinsey There will be a shortage of 15000000 big data professionals by the end of 2020. There are lots of technologies to solve the problem of big data Storage and processing. Such technologies are Apache Hadoop, Apache Spark, Apache Kafka, and many more. Here we analyse the processing speed for the 4GB data on cloudx lab with Hadoop mapreduce with varing mappers and reducers and with pig script and Hive querries and spark environment along with machine learning technology and from the results we can say that machine learning with Hadoop will enhance the processing performance along with with spark, and also we can say that spark is better than Hadoop mapreduce pig and hive, spark with hive and machine learning will be the best performance enhanced compared with pig and hive, Hadoop mapreduce jar.

Different multimodel ensemble methods are used to forecast precipitations in China, 1998, and their forecast skills are compared with those of individual models. Datasets were obtained from monthly simulations of eight models during the period of January 1979 to December 1998 from the “Climate of the 20th Century Experiment” (20C3M) for the Fourth IPCC Assessment Report. Climate Research Unit (CRU) data were chosen for the observation analysis field. Root mean square (RMS) error and correlation coefficients (R) are used to measure the forecast skills. In addition, superensemble forecasts based on different input data and weights are analyzed. Results show that for original data, superensemble forecasting based on multiple linear regression (MLR) performs best. However, for bias-corrected data, the superensemble based on singular value decomposition (SVD) produces a lower RMS error and a higher R than in the MLR superensemble. It is an interesting result that the SVD superensemble based on bias-corrected data performs better than the MLR superensemble, but that the SVD superensemble based on original data is inferior to the corresponding MLR superensemble. In addition, weights calculated by different data formats are shown to affect the forecast skills of the superensembles. In comparison with the MLR superensemble, a slightly significant effect is present in the SVD superensemble. However, both the SVD and MLR superensembles based on different weight formats outperform the ensemble mean of bias-corrected data.

ion called Resilient Distributed Datasets (RDDs) [88]. RDDs is proposed to let programmers perform in-memory computations on large clusters in a fault-tolerant manner. This abstraction is motivated by two types of applications: iterative algorithms and interactive data mining tools that current computing frameworks handle inefficiently. As the cost of implementing machine learning algorithms as low-level MapReduce jobs on varying data and machine cluster sizes can be prohibitive, SystemML [89] is proposed to express machine learning algorithms in a higher-level language which are Scala. http://www.scala-lang.org. 56 compiled and executed in a MapReduce environment. MLbase [90] is a scalable machine learning system for both end-users and machine learning researchers. Two main functions are provided by MLbase: it enables machine learning researchers to implement scalable machine learning methods without requiring deep system knowledge; it provides a novel optimizer to select and dynamically adapt the choice of learning algorithms. Similar to SystemML, MLbase applies simple declarative way to specify machine learning tasks.

Apache Spark is an increasingly popular distributed computation framework based on in-memory computations, which enables iterative or interactive applications to run faster. In Spark, memory management is the key to performance enhancement to avoid memory bloat problems. Compared with previous static memory manager, in Spark 1.6 and later versions, unified memory manager is implemented as the default memory management model, targeting to achieve optimal memory utilization by borrowing between storage and execution memory. However, the storage memory borrowed from execution memory may frequently be evicted when memory pressure arises. It is because of frequently re- computation during cache evicted and frequent garbage collection during shuffle in iterative applications. This situation will produce runtime overhead caused by garbage collection including cache eviction and cache re-computation. We propose a memory constraint strategy for unified memory manager in Spark to reduce runtime overhead caused by garbage collection by reducing the cache eviction size. We implement the strategy in Spark 1.6.1 using SparkPageRank, WordCount and GroupByTest to compare three different memory managers. Experimental results reveal that compared unified memory manager, memory constraint strategy can achieve better performance improvement with lower job runtime and garbage collection time when the dataset sizes or the iterations are increasing.

Cloud computing and big data: Technologies and applications

Parallel computing has become fundamental in processing the massive data volumes generated in modern science and industry. This paper presents a comprehensive survey and practical review of parallel computing in data processing, examining key frameworks (MPI, Open MP, CUDA, Hadoop MapReduce, Apache Spark, etc.), implementations, and real- world case studies. We discuss the architectures underlying shared-memory and distributed-memory systems and illustrate how parallelism on CPUs and GPUs is exploited for high- performance computing (HPC) and big data analytics. We review theoretical foundations (including Amdahl’s law for speedup limits) and compare programming models in terms of design, performance, and fault tolerance. A two-column IEEE-style for- mat is used to present critical insights from scholarly literature, with diagrams illustrating parallel architectures and workflows, and tabular summaries of tools and results. Case studies—from scientific HPC simulations to large-scale data analytics—highlight practical parallelism on multicore CPUs, GPUs, and computing clusters. We conclude with a discussion on the convergence of HPC and big data paradigms, the challenges of heterogeneous computing, and emerging trends. This survey serves as a reference for PhD researchers and practitioners seeking to understand frameworks and real-world practices in parallel computing for data processing.

One of the main challenges in Big data is the processing speed and scalability. Solid State Drive (SSD) helps for faster processing than HDD. Here along with SSD, Spark is also accompanied with hadoop framework for more scalability and fast processing. Apache Spark is a general-purpose engine for large-scale data processing on any cluster. It is a framework which can afford more than 8000 nodes in a cluster Spark allows for code reuse across batch, interactive, and streaming applications. Spark is much faster than MapReduce. It was generally coded from Java; Spark supports not only Java, but also Python and Scala, which is a newer language that contains some attractive properties for manipulating data. Spark runs up to 100 times faster than Hadoop MapReduce in memory and 10 times faster on disk. This paper tries to integrate spark with Hadoop ecosystem along the SSD. It increases the processing speed.

Context. Considered question correct interpretation information flow in distributed information systems. The object of study methods are promotion “big data” on cluster system.Objective. Is the study promising areas and technology for the analysis of structures data in distributed information systems.Method. The big data tendency prospects as well as timeliness of the problem are studied in this paper. The principles of work with them are addressed. Big data processing technologies are provided. The analysis of each one is performed. An example of “MapReduce” paradigm application, uploading of big volumes of data, processing and analyzing of unstructured information and its distribution into the clustered database is provided. The article summarizes the concept of “big data”. Examples of methods for working with arrays of unstructured data. Dedicated scientific guidance for analyzing big data. The principles of unstructured data in distributed information systems. Driven work platform “Hadoop MapReduce” and “Apache Spark”. Analyzed their properties and given the differences. An analysis of comparative performance against both platforms – the performance of the number of iterations. Consider ways to create RDD: parallelization transmitted collection program and a link to an external file system in “Hadoop”. There is an example rozparalelenoyi system RDD. Proposed work lone class for basic database operations: database connection, create a table, a table, get in line id, returning all elements of the database, update, delete and create the line.Results. The analysis Models Spark and Hadoop MapReduce for phased construction distributed information system. built up SparkConf object, containing information about applique and is the final version of the experiment.Conclusions. Conducted experiment confirmed efficiency the proposed method, are capable process horizontal data arrays, that parallelization by defective presentation of information. These promising areas of analyze structure data for the purpose of forecast results and create algorithms advanced correlation, contributing new understanding activity distributed information systems further research can consist in wide use information systems, that would provide a full range technological process adaptation information flows in clusters.

In this paper, iterative learning identification algorithms are proposed to estimate the time-varying parameters in multi-input-single-output (MISO) Wiener nonlinear time-varying systems. The regression model of the Wiener system is built by using the polynomial expansion of the nonlinear inverse function. Then, two iterative learning algorithms, including iterative learning gradient identification and iterative learning least squares identification, are presented to estimate the time-varying parameters of the regression model. The convergence performance of the iterative learning identification algorithms is analyzed, and numerical simulations are provided to verify the effectiveness of the proposed algorithms.

ABSTRACT: Coronary artery heart Disease (CAD) is caused by atherosclerosis in coronary arteries and results in cardiac arrest and heart attack. For diagnosis of CAD, angiography is used which is a costly time consuming and highly technical invasive method. Researchers are therefore, prompted for alternative methods such as machine learning algorithms that could use non-invasive clinical data for the heart Disease diagnosis and assessing its severity. In this study, we present a novel hybrid method for CAD diagnosis; including risk factor identification using correlation based feature subset (CFS) selection with particle swam optimization (PSO) search method and K-Means clustering algorithms. Supervised learning algorithms such as multi-layer perceptron (MLP), multinomial logistic regression (MLR), fuzzy unordered rule induction algorithm (FURIA) and C4.5 are then used to model CAD cases. We tested this approach on clinical data consisting of 26 features and 335 instances collected at the Department of Cardiology, Indira Gandhi Medical College, and Shimla, India. MLR achieves highest prediction accuracy of 88.4 %.We tested this approach on benchmarked Cleveland heart Disease data as well. In this case also, MLR outperforms other techniques. Proposed hybridized model improves the accuracy of classification algorithms from 8.3 % to 11.4 % for the Cleveland data. The proposed method is, therefore, a promising tool for identification of CAD improved prediction accuracy. KEY WORDS: Coronary artery heart Disease, fuzzy unordered rule induction algorithm, multinomial logistic regression