Improving spark application throughput via memory aware task co-location

Fuzzy clustering is an invaluable data mining technique that allows each data point to belong to more than one cluster with some degree of membership. It is widely employed in exploratory data mining to discover overlapping communities in social networks, find structure in spectral data, and capture user interests in recommendation systems. Nowadays, the variety and volume of data are increasing at a tremendous rate. Data is power; the massive data, along with an effective technique, can unravel valuable information. The existing fuzzy clustering algorithms do not perform well on massive heterogeneous datasets. Processing an enormous amount of data is beyond the capacity of a single processor. The need of the hour is to develop fuzzy clustering techniques that can work on a distributed framework for Big Data processing and can handle heterogeneous data. In this research, we evaluate the performance of the recently proposed algorithm for the Fuzzy clustering of mixed-mode data FCMD-MD (D’Urso and Massari in Inf Sci 505:513–534, 2019) with different real-world datasets. We develop a distributed FCMD-MD, a fuzzy clustering algorithm for mixed-mode data in Apache SPARK. The experimental results show that the algorithm is scalable, performs well in a distributed environment, and clusters enormous heterogeneous data with high accuracy. We also compared the performance of distributed FCMD-MD and the distributed k-medoid algorithm.

This research examines what impact the Big Data Processing Framework (BDPF) has on Artificial Intelligence (AI) applications within Corporate Marketing Communication (CMC), and thereby the research question stated is: What is the potential impact of the BDPF on AI applications within the CMC tactical and managerial functions? To fulfill the purpose of this research, a qualitative research strategy was applied, including semi-structured interviews with experts within the different fields of examination: management, AI technology and CMC. The findings were analyzed through performing a thematic analysis, where coding was conducted in two steps. AI has many useful applications within CMC, which currently mainly are of the basic form of AI, so-called rule-based systems. However, the more complicated communication systems are used in some areas. Based on these findings, the impact of the BDPF on AI applications is assessed by examining different characteristics of the processing frameworks. The BDPF initially imposes both an administrative and compliance burden on organizations within this industry, and is particularly severe when machine learning is used. These burdens foremost stem from the general restriction of processing personal data and the data erasure requirement. However, in the long term, these burdens instead contribute to a positive impact on machine learning. The timeframe until enforcement contributes to a somewhat negative impact in the short term, which is also true for the uncertainty around interpretations of the BDPF requirements. Yet, the BDPF provides flexibility in how to become compliant, which is favorable for AI applications. Finally, BDPF compliance can increase company value, and thereby incentivize investments into AI models of higher transparency. The impact of the BDPF is quite insignificant for the basic forms of AI applications, which are currently most common within CMC. However, for the more complicated applications that are used, the BDPF is found to have a more severe negative impact in the short term, while it instead has a positive impact in the long term.

Apache Spark is a leading open-source data processing engine used for batch processing, machine learning, stream processing, and large-scale SQL (structured query language).It has been designed to make big data processing quicker and easier.Since its inception, Spark has gained huge popularity as a big data processing framework and is extensively used by different industries and businesses that are dealing with large volumes of data.This paper will exhibit actionable solutions to maximize our chances of reducing computation time by optimizing Spark jobs.The strategy lays out different run stages, wherein each run stage builds upon the previous and improves the computation time by making new enhancements and recommendations.

SUMMARYIt is well known that data centers are consuming a large amount of energy that incurs significant financial and environmental costs. Recently, there has been an increasing interest in utilizing green energy for data centers, where green energy sources include solar and wind. This paper studies the crucial problem of maximizing the utilization of green energy through scheduling complex jobs in data centers in order to reduce the use of traditional brown energy. However, it is highly challenging for data centers to make use of green energy. First, the availability of typical green energy is variable to dynamic changes of natural environments, for example, weather. Second, although predictions can be made for the future availability of green energy, it is inevitable that such predictions have errors. Third, jobs are associated with strict deadlines, and it is required that jobs are completed before their deadlines. Finally, because the reliability in a data center relies upon temperature, the awareness of temperature should be taken into account while maximizing the green energy. In this paper, we consider online scheduling of jobs whose arrivals to the data center system dynamically. In addition, we explicitly take the power consumption of switches into account when scheduling jobs onto computing nodes. Two solar energy‐aware algorithms called SEEDMin and SEEDMax have been proposed. Then, we extend SEED to RSEED with the awareness of reliability. To evaluate the effectiveness of the proposed algorithms, comprehensive simulations have been conducted, and the proposed algorithms are compared with other state‐of‐art algorithms. Experimental results demonstrate that both SEEDMin and SEEDMax can significantly increase the utilization of solar energy without violating job deadlines and overall energy budget. The amount of solar energy utilized by SEEDMin and SEEDMax is 33.4%and35.3% larger than that of two traditional scheduling algorithms, MinMin and MinMax, respectively. Also, it can be seen that RSEED greatly improves the reliability by decreasing the temperature. Copyright © 2013 John Wiley & Sons, Ltd.

For MapReduce jobs in data center, network traffic is generated in shuffling phase, causing east-west communication bottleneck. Aiming at this problem, an optimization scheme is proposed to aggregate relevant network traffic flows into local areas. Firstly, the characteristics of pre-scheduling of MapReduce jobs are extracted, the communication activity degree of jobs is defined and the computing jobs are divided into two types: active or inactive communication job. Then the Bayesian classification with active learning is used as the prediction model, and this model after training by sample data can determine job type. The active communication jobs are deployed in the same rack to improve network bandwidth utilization. The experiment results of small-scale data center show that the proposed communication optimization scheme has a significant effect on shuffling intensive jobs, reaching 4.2%-5.6%. In the case of larger amount of input data, this scheme has better robustness and can effectively reduce east-west communication delay in data center.

Now a day's data is growing very rapidly, where processing and analyzing data to get useful information is the main task. There are many big data processing tools and framework such as Hadoop, Hive, Cassandra etc. Spark is one of the fastest big data processing framework in cluster computation.Basic Idea is to analyze the performance of java virtual machine (JVM) [1], by characterizing java virtual machine using SparkBench benchmark on Apache Spark™ [2]. Java virtual machine is a core execution platform for spark application. When we run the spark application on java virtual machine, its behavior is affected, which needs to be monitored to analyze the JVM performance. Here we are considering Machine Learning workloads like K-Means, Matrix Factorization and Logistic Regression. Main goal here is to analyze the machine learning workloads end to end across the cluster, with respect to following parameters such as garbage collection, memory such as heap usage, CPU process time. Characterization of JVM is done with spark cluster setup and HDFS is used as storage with distributed Hadoop cluster setup.

The scale of large processing tasks being run within the scientific Grid and Cloud environments have introduced a need for stability guarantees from geographically spanning resources, to ensure that failures are detected and handled preemptively. Performance inefficiencies within stacked service environments are a challenge to detect, where failures stem from a multitude of causes, often requiring expert intervention. Online reporting and classification of performance fluctuations can aid experts, central services, and users to target service areas where optimizations can be introduced. This paper describes an approach for modeling performance states for production tasks running within the ALICE Grid. We first provide an overview for the ALICE data and software workflow, focusing on the production job computational profile. Data center event state is then developed, based on data center job, computing, storage, and user behavior. With across site analysis, we then train groups to classify service domain states. Our approach is able to detect periods of service instability and the affected service domains. This can guide users, central, and data center experts to take action in advance of service failure effects.

In order to reduce power and energy costs, giant cloud providers now mix online and batch jobs on the same cluster. Although the co-allocation of such jobs improves machine utilization, it challenges the data center scheduler and workload assignment in terms of quality of service, fault tolerance, and failure recovery, especially for latency critical online services. In this paper, we explore various characteristics of co-allocated online services and batch jobs from a production cluster containing 1.3k servers in Alibaba Cloud. From the trace data, we find the following: 1) For batch jobs with multiple tasks and instances, 50.8% failed tasks wait and halted after a very long time interval when their first and the only one instance fails. This wastes much time and resources as the remaining instances are running for an impossible successful termination. 2) For online services jobs, they are clustered in 25 categories according to their requested CPU, memory, and disk resources. Such clustering can help the co-allocation of online services jobs with batch jobs. 3) Servers are clustered into seven groups by CPU utilization, memory utilization, and their correlations. Machines with a strong correlation between CPU and memory utilization provides an opportunity for job co-allocation and resource utilization estimation. 4) The MTBF (mean time between failures) of instances are in the interval [400, 800] seconds while the average completion time of the 99th percentile is 1003 seconds. We also compare the cumulative distribution functions of jobs and servers and explain the differences and opportunities for workload assignment between them. Our findings and insights presented in this paper can help the community and data center operators better understand the workload characteristics, improve resource utilization, and failure recovery design.

In the big data processing systems, the amount of data is increasing. At the same time, the real-time requirement of data processing and analysis is higher and higher. Therefore, it is required that the big data processing and analysis systems have better performance. Job scheduling plays an important role in improving the overall system performance in big data processing frameworks. However, job scheduling is a difficult NP-hard problem. There are many factors that need to be considered for job scheduling. For example, jobs have dependencies among stages, therefore we should not allocate resources to tasks that are not ready. Sometimes, there are constraints between jobs. These are a challenge to the scheduling performance of big data processing and analysis systems. In this paper, we try to solve the problem by translating it into Boolean Satisfiability Problem (SAT) which is an exact method. SAT-based scheduling algorithm is not a new approach, but in the past it mainly used to solve the static scheduling problems. For dynamic scheduling system, it requires all problems to be solved within a limited time, which is a challenge for SAT encoding. In this paper, we refer to the previous SAT solution to the Job Shop Scheduling Problem, and adjust the algorithm to meet the requirements of the big data processing system. At the same time, we optimized the coding approach and reduced the number of clauses. Thus, the efficiency of the problem solved is improved to meet the performance requirements. The experimental results show that the number of clauses is reduced by more than 30%, and the processing time of the SAT solver to get the solution can be reduced by more than 50%. To demonstrate its effectiveness, we have also implemented our new job scheduler in Apache Hadoop YARN, and validated its effectiveness

Recent years have witnessed a surge of new generation applications involving big data. The de facto framework for big data processing, MapReduce, has been increasingly embraced by both academic and industrial users. Data locality seeks to co-locate computation with data, which effectively reduces remote data access and improves MapReduce’s performance in physical machine clusters. State-of-the-art public clouds heavily rely on virtualization to enable resource sharing and scaling for massive users, however. In this article, through real-world experiments, we show strong evidence that the conventional notion of data locality is unfortunately not always beneficial for MapReduce in a virtualized environment. The observations suggest that the measure of node-local must be extended to distinguish physical and virtual entities. We develop vLocality, a comprehensive and practical solution for data locality in virtualized environments. It incorporates a novel storage architecture that efficiently mitigates the shared disk contention, and an enhanced task scheduling algorithm that prioritizes co-located VMs. We have implemented a prototype of vLocality based on Hadoop 1.2.1, and have validated its effectiveness on a typical virtualized cloud platform consisting of 22 nodes. Our experimental results demonstrate that vLocality can improve the job finish time to around a quarter of that for typical Hadoop benchmark applications.

Streams of data are produced today at an unprecedented scale. Efficient and stable processing of these streams requires a careful interplay between the parameters of the streaming application and of the underlying stream processing framework. Today, finding these parameters happens by trial and error on the complex, deployed framework. This paper shows that high-level models can help to determine these parameters by predicting and comparing the performance of streaming applications running on stream processing frameworks with different configurations. To demonstrate this approach, this paper considers Spark Streaming, a widely used framework to leverage data streams on the fly and provide real-time stream processing. Technically, we develop a configurable and executable model to simulate both the streaming applications and the underlying Spark stream processing framework. Furthermore, we model the deployment of Spark Streaming on Apache YARN, which is a popular open-source distributed software framework for big data processing. We show that the developed model provides a satisfactory accuracy for predicting performance by means of empirical validation.

Extensive usage of Internet based applications in day to day life has led to generation of huge amounts of data every minute. Apart from humans, data is generated by machines like sensors, satellite, CCTV etc. This huge collection of heterogeneous data is often referred as Big Data which can be processed to draw useful insights. Apache Hadoop has emerged has widely used open source software framework for Big Data Processing and it is a cluster of cooperative computers enabling distributed parallel processing. Hadoop Distributed File System is used to store data blocks replicated and spanned across different nodes. HDFS uses an AES based cryptographic techniques at block level which is transparent and end to end in nature. However cryptography provides security from unauthorized access to the data blocks, but a legitimate user can still harm the data. One such example was execution of malicious map reduce jar files by legitimate user which can harm the data in the HDFS. We developed a mechanism where every map reduce jar will be tested by our sandbox security to ensure the jar is not malicious and suspicious jar files are not allowed to process the data in the HDFS. This feature is not present in the existing Apache Hadoop framework and our work is made available in github for consideration and inclusion in the future versions of Apache Hadoop.

Improving Hadoop MapReduce performance on heterogeneous single board computer clusters

Modern data centers typically contain thousands of servers, providing various computing and storage services for users. The strategy to provide reliable and high-performance online services is to over-allocate resources for online services, which results in a waste of cluster resources. Therefore, cloud vendors tend to co-locate online services and offline batch jobs into the same cluster to improve resource utilization. However, the co-location leads to contention on shared resources and causes mutual performance interference, which may degrade the QoS (Quality of Service) of online services. We present a performance interference model based on linear regression to predict the performance interference. Furthermore, the model can perceive the status of servers in real-time for more refined and accurate prediction. Then, we design an interference-aware workload scheduling strategy that can schedule batch jobs to the server while introducing minimal interference. The evaluation demonstrates that our scheduling strategy can at best increase the throughput of batch jobs by 48.95% and 27.09% compared with round-robin scheduling and random scheduling while guaranteeing the QoS of online services. The paper aims to provide some elements of answering the following general question: How do we design cloud systems and data centers scheduling strategies that can withstand the human pressure of global-scale use and still provide robust and secure services to end-users?KeywordsCloud computingWorkload co-locationPerformance interferencePerformance predictionWorkload schedulingData centers

Decreasing the soaring energy cost is imperative in large data centers. Meanwhile, limited computational resources need to be fairly allocated among different organizations. Latency is another major concern for resource management. Nevertheless, energy cost, resource allocation fairness, and latency are important but often contradicting metrics on scheduling data center workloads. Moreover, with the ever-increasing power density, data center operation must be judiciously optimized to prevent server overheating. In this paper, we explore the benefit of electricity price variations across time and locations. We study the problem of scheduling batch jobs to multiple geographically-distributed data centers. We propose a provably-efficient online scheduling algorithm - GreFar - which optimizes the energy cost and fairness among different organizations subject to queueing delay constraints, while satisfying the maximum server inlet temperature constraints. GreFar does not require any statistical information of workload arrivals or electricity prices. We prove that it can minimize the cost arbitrarily close to that of the optimal offline algorithm with future information. Moreover, we compare the performance of GreFar with ones of a similar algorithm, referred to as T-unaware, that is not able to consider the server inlet temperature in the scheduling process. We prove that GreFar is able to save up to 16 percent of energy-fairness cost with respect to T-unaware.