Optimize Large-Scale Data Processing Via Spark Tuning

This research examines what impact the Big Data Processing Framework (BDPF) has on Artificial Intelligence (AI) applications within Corporate Marketing Communication (CMC), and thereby the research question stated is: What is the potential impact of the BDPF on AI applications within the CMC tactical and managerial functions? To fulfill the purpose of this research, a qualitative research strategy was applied, including semi-structured interviews with experts within the different fields of examination: management, AI technology and CMC. The findings were analyzed through performing a thematic analysis, where coding was conducted in two steps. AI has many useful applications within CMC, which currently mainly are of the basic form of AI, so-called rule-based systems. However, the more complicated communication systems are used in some areas. Based on these findings, the impact of the BDPF on AI applications is assessed by examining different characteristics of the processing frameworks. The BDPF initially imposes both an administrative and compliance burden on organizations within this industry, and is particularly severe when machine learning is used. These burdens foremost stem from the general restriction of processing personal data and the data erasure requirement. However, in the long term, these burdens instead contribute to a positive impact on machine learning. The timeframe until enforcement contributes to a somewhat negative impact in the short term, which is also true for the uncertainty around interpretations of the BDPF requirements. Yet, the BDPF provides flexibility in how to become compliant, which is favorable for AI applications. Finally, BDPF compliance can increase company value, and thereby incentivize investments into AI models of higher transparency. The impact of the BDPF is quite insignificant for the basic forms of AI applications, which are currently most common within CMC. However, for the more complicated applications that are used, the BDPF is found to have a more severe negative impact in the short term, while it instead has a positive impact in the long term.

In the big data processing systems, the amount of data is increasing. At the same time, the real-time requirement of data processing and analysis is higher and higher. Therefore, it is required that the big data processing and analysis systems have better performance. Job scheduling plays an important role in improving the overall system performance in big data processing frameworks. However, job scheduling is a difficult NP-hard problem. There are many factors that need to be considered for job scheduling. For example, jobs have dependencies among stages, therefore we should not allocate resources to tasks that are not ready. Sometimes, there are constraints between jobs. These are a challenge to the scheduling performance of big data processing and analysis systems. In this paper, we try to solve the problem by translating it into Boolean Satisfiability Problem (SAT) which is an exact method. SAT-based scheduling algorithm is not a new approach, but in the past it mainly used to solve the static scheduling problems. For dynamic scheduling system, it requires all problems to be solved within a limited time, which is a challenge for SAT encoding. In this paper, we refer to the previous SAT solution to the Job Shop Scheduling Problem, and adjust the algorithm to meet the requirements of the big data processing system. At the same time, we optimized the coding approach and reduced the number of clauses. Thus, the efficiency of the problem solved is improved to meet the performance requirements. The experimental results show that the number of clauses is reduced by more than 30%, and the processing time of the SAT solver to get the solution can be reduced by more than 50%. To demonstrate its effectiveness, we have also implemented our new job scheduler in Apache Hadoop YARN, and validated its effectiveness

All applications in current trends need to use Machine Learning and Big Data in this huge data world.Machine learning is a type of artificial intelligence that allows software applications to become more accurate at predicting outcomes without being explicitly programmed to do so.Machine learning use historical data as input to predict new output values.Small Scale and even large developers cannot maintain Machine Learning Algorithms and other big data processing techniques all by themselves.To harvest, process and analyze the data we need a data processing engine.Our project is to provide them with a machine learning engine which runs on.a specified port on their servers which collects the data sets in real time.This which can be queried using a simple Query Language and analyze the database on the query.The purpose of our project is to create a Data Processing Engine for Big Data Analytics. Machine learning and Big Data plays a major role in the current huge data world. Our project is to maintain a dedicated machine learning and data processing engine, so that even small developers without the knowledge of machine learning and big data analysis can produce expected results based on Machine Learning to make their application smooth. All applications in current trends need to use Machine Learning and Big Data in this huge data world. Machine learning is a subset of artificial intelligence that allows software applications to become more accurate at predicting outcomes without being explicitly programmed to do so. Machine learning uses historical data as input to predict new output values. Small Scale and even large developers cannot maintain Machine Learning Algorithms and other big data processing techniques all by themselves. To harvest, process and analyze the data we need a data processing engine. This project is to provide them a data processing engine which runs on a specified port on their servers which collects the data sets in real time. This can be queried using a simple Query Language and analyze the data and produce respected results using ML based on the query

The complexity of the Internet and the volume of network traffic have dramatically increased in the last few years, making it more challenging to design scalable Network Traffic Monitoring and Analysis (NTMA) systems. Critical NTMA applications such as the detection of network attacks and anomalies require fast mechanisms for on-line analysis of thousands of events per second, as well as efficient techniques for off-line analysis of massive historical data. The high-dimensionality of network data provided by current network monitoring systems opens the door to the massive application of machine learning approaches to improve the detection and classification of network attacks and anomalies, but this higher dimensionality comes with an extra data processing overhead. In this paper we present Big-DAMA, a big data analytics framework (BDAF) for NTMA applications. Big-DAMA is a flexible BDAF, capable to analyze and store big amounts of both structured and unstructured heterogeneous data sources, with both stream and batch processing capabilities. Big-DAMA uses off-the-shelf big data storage and processing engines to offer both stream data processing and batch processing capabilities, decomposing separate engines for stream, batch and query, following a Data Stream Warehouse (DSW) paradigm. Big-DAMA implements several algorithms for anomaly detection and network security using supervised and unsupervised machine learning (ML) models, using off-the-shelf ML libraries. We apply Big-DAMA to the detection of different types of network attacks and anomalies, benchmarking multiple supervised ML models. Evaluations are conducted on top of real network measurements collected at the WIDE backbone network, using the well-known MAWILab dataset for attacks labeling. Big-DAMA can speed up computations by a factor of 10 when compared to a standard Apache Spark cluster, and can be easily deployed in cloud environments, using hardware virtualization technology.

Fuzzy clustering is an invaluable data mining technique that allows each data point to belong to more than one cluster with some degree of membership. It is widely employed in exploratory data mining to discover overlapping communities in social networks, find structure in spectral data, and capture user interests in recommendation systems. Nowadays, the variety and volume of data are increasing at a tremendous rate. Data is power; the massive data, along with an effective technique, can unravel valuable information. The existing fuzzy clustering algorithms do not perform well on massive heterogeneous datasets. Processing an enormous amount of data is beyond the capacity of a single processor. The need of the hour is to develop fuzzy clustering techniques that can work on a distributed framework for Big Data processing and can handle heterogeneous data. In this research, we evaluate the performance of the recently proposed algorithm for the Fuzzy clustering of mixed-mode data FCMD-MD (D’Urso and Massari in Inf Sci 505:513–534, 2019) with different real-world datasets. We develop a distributed FCMD-MD, a fuzzy clustering algorithm for mixed-mode data in Apache SPARK. The experimental results show that the algorithm is scalable, performs well in a distributed environment, and clusters enormous heterogeneous data with high accuracy. We also compared the performance of distributed FCMD-MD and the distributed k-medoid algorithm.

With the development of various technologies such as high-speed Internet and SNS dissemination, there have been many fields that require processing of big data generated in real time. Accordingly, real-time streaming data processing technology has been developed, and representative platforms include Apache Storm, Apache Spark, and Hadoop. These processing technologies provide scalability to configure distributed systems using multiple servers because they vary in performance, such as throughput and processing speed, depending on the server environment, but the more the number of servers, the more difficult it is to manage. To solve this problem, a problem can be solved by using a docker, a kind of virtualization system that provides ease of expansion. However, there is a place to maintain a native environment without using Docker due to the problem that performance may be reduced, which is a disadvantage of all virtualization systems. In this paper, we build Apache Storm and Apache Spark, which are real-time data processing systems in Docker and Native environments and conduct performance measurements through experiments processing JSON-format data to verify how much performance decreases in Docker environments.

In the 4-th Industrial Revolution era, various intelligent solutions and services have been emerging recently. To provide high quality service in those intelligent applications, the big data should be collected without any loss and comprehensively analyzed. Especially, when using machine and deep learning techniques, the big data processing delays should be minimized in order to guarantee the freshness of models. In this paper, we evaluate the performance of Apache Spark which is one of the most popular big data processing and analysis frameworks. Beyond the performance analysis of Spark in distributed cluster environment, we evaluate the performance of TensorFlowOnSpark which is the promising distributed deep learning framework designed to handle big data efficiently. From the experimental results, we can conclude that Spark on YARN is a solid underlying framework that guarantees the performance and scalability of distributed machine and deep learning by efficiently processing its data and algorithms in a parallel and distributed manner.

Streams of data are produced today at an unprecedented scale. Efficient and stable processing of these streams requires a careful interplay between the parameters of the streaming application and of the underlying stream processing framework. Today, finding these parameters happens by trial and error on the complex, deployed framework. This paper shows that high-level models can help to determine these parameters by predicting and comparing the performance of streaming applications running on stream processing frameworks with different configurations. To demonstrate this approach, this paper considers Spark Streaming, a widely used framework to leverage data streams on the fly and provide real-time stream processing. Technically, we develop a configurable and executable model to simulate both the streaming applications and the underlying Spark stream processing framework. Furthermore, we model the deployment of Spark Streaming on Apache YARN, which is a popular open-source distributed software framework for big data processing. We show that the developed model provides a satisfactory accuracy for predicting performance by means of empirical validation.

Cloud computing and big data: Technologies and applications

Now a day's data is growing very rapidly, where processing and analyzing data to get useful information is the main task. There are many big data processing tools and framework such as Hadoop, Hive, Cassandra etc. Spark is one of the fastest big data processing framework in cluster computation.Basic Idea is to analyze the performance of java virtual machine (JVM) [1], by characterizing java virtual machine using SparkBench benchmark on Apache Spark™ [2]. Java virtual machine is a core execution platform for spark application. When we run the spark application on java virtual machine, its behavior is affected, which needs to be monitored to analyze the JVM performance. Here we are considering Machine Learning workloads like K-Means, Matrix Factorization and Logistic Regression. Main goal here is to analyze the machine learning workloads end to end across the cluster, with respect to following parameters such as garbage collection, memory such as heap usage, CPU process time. Characterization of JVM is done with spark cluster setup and HDFS is used as storage with distributed Hadoop cluster setup.

In-Mapper combiner based MapReduce algorithm for processing of big climate data

Due to the rising demand for large-scale data processing, there is a growing interest in both batch processing, where large volumes of data are processed offline, and stream processing, where large quantities of streaming data are processed online. The dichotomy between these vastly different computing paradigms has led to the development of substantially different methodologies and systems. As there is an increasing number of applications requiring stream and batch processing, there is a need to bridge this gap and offer support for both paradigms. We introduce a new direction for the unification of stream and batch processing, which, contrary to other proposed approaches, uses a stream processing platform as its foundation and supports batch processing on top. Our proof-of-concept implementation of such a middleware layer, called Cyclone, offers the widely popular MapReduce programming model and translates MapReduce jobs for execution on the underlying streaming platform. Cyclone not only achieves a tight integration of batch and stream processing, our evaluation further shows significant performance gains, in particular for sequential and iterative jobs, which naturally arise in many applications.

This article delves into the communication quality monitoring system driven by big data under the C/S (client/server) architecture, aiming to provide a comprehensive, real-time, and efficient security monitoring solution for factory production and manufacturing environments. The system integrates LoRa wireless communication technology, big data processing and analysis technology, and innovative dynamic expansion algorithms to achieve real-time monitoring and rapid response to various safety hazards in the production environment. The core of the system lies in the construction of LoRa wireless communication network, which adopts an innovative umbrella shaped network structure to ensure stable and fast transmission of sensor monitoring data to the server. By using the STM32 development board with LoRa module, dynamic expansion of hardware integration modules has been achieved, enhancing the flexibility and scalability of the system. At the level of data transmission and processing, introducing Kafka distributed message queue as a data cache effectively alleviates the pressure of processing massive real-time data. Combining the Spark Streaming streaming data processing framework, a distributed data processing model was constructed to achieve real-time consumption and efficient parallel processing of messages in Kafka queues. At the same time, the designed dynamic extension algorithm model can automatically persist the data of new monitoring points, ensuring the system's rapid adaptation to data changes. In terms of data storage and visualization, a data persistence layer architecture for a big data platform has been constructed. Real time data, historical data, and log data are stored separately in Redis, MySQL, and HDFS systems through a data streaming model, improving data storage efficiency and system performance. Based on Tomcat network server and SSM architecture, a B/S structured web server visualization platform has been developed, which supports users to remotely query the security status of the production environment through computer or mobile browser, and receive alarm notifications in case of abnormalities. The system integration test results show that the communication quality monitoring system performs excellently in wireless communication quality, data processing speed, data storage efficiency, and data visualization, and can meet the complex monitoring needs of factory production and manufacturing environments, providing solid technical support for enterprise safety production. However, there is still room for improvement in real-time data collection loss rate, alarm function completeness, and mobile access experience. Future research will focus on optimizing data transmission software models, adding multi-channel alarm functions, and developing mobile apps to enhance user experience.

Hardware accelerators are playing an increasingly important role in the field of computer science and engineering, especially in accelerating deep learning and large-scale data processing. However, the traditional hardware accelerator design often relies on manual adjustment and experience accumulation, which is inefficient. Therefore, the use of machine learning technology to optimize hardware accelerator design has become a new trend. In this paper, a hardware accelerator optimization design method based on machine learning is proposed, including hardware design, algorithm optimization, system-level optimization, and deep learning model compression. Through the research and verification of the actual case, this research proves the effectiveness and feasibility of the hardware accelerator optimization design method based on machine learning. Compared with the traditional design method, this method can find better design scheme in shorter time and has higher performance prediction accuracy. Therefore, this method can significantly improve the design efficiency and performance of hardware accelerators. The results of this study are of great significance for promoting the development of hardware accelerator technology. It can promote scientific research and engineering applications in related fields to make greater progress, and promote the development of artificial intelligence, big data processing and other fields.

Data analytic applications built upon big data processing frameworks such as Apache Spark are an important class of applications. Many of these applications are not latency-sensitive and thus can run as batch jobs in data centers. By running multiple applications on a computing host, task co-location can significantly improve the server utilization and system throughput. However, effective task co-location is a non-trivial task, as it requires an understanding of the computing resource requirement of the co-running applications, in order to determine what tasks, and how many of them, can be co-located. State-of-the-art co-location schemes either require the user to supply the resource demands which are often far beyond what is needed; or use a one-size-fits-all function to estimate the requirement, which, unfortunately, is unlikely to capture the diverse behaviors of applications.