Big Data & Analytics, Data Automation Recent Trends: Smart Memory Offloading for Enhanced Data Processing Throughput

With the development of various technologies such as high-speed Internet and SNS dissemination, there have been many fields that require processing of big data generated in real time. Accordingly, real-time streaming data processing technology has been developed, and representative platforms include Apache Storm, Apache Spark, and Hadoop. These processing technologies provide scalability to configure distributed systems using multiple servers because they vary in performance, such as throughput and processing speed, depending on the server environment, but the more the number of servers, the more difficult it is to manage. To solve this problem, a problem can be solved by using a docker, a kind of virtualization system that provides ease of expansion. However, there is a place to maintain a native environment without using Docker due to the problem that performance may be reduced, which is a disadvantage of all virtualization systems. In this paper, we build Apache Storm and Apache Spark, which are real-time data processing systems in Docker and Native environments and conduct performance measurements through experiments processing JSON-format data to verify how much performance decreases in Docker environments.

With the onset of the big data age, data is growing exponentially, and the issue of how to optimize large-scale data processing is especially significant.Large-scale global optimization (LSGO) is a research topic with great interest in academia and industry.Spark is a popular cloud computing framework that can cluster large-scale data, and it can effectively support the functions of iterative calculation through resilient distributed datasets (RDD).In this paper, we propose a hybrid mechanism of particle swarm optimization (PSO) and differential evolution (DE) algorithms based on Spark (SparkPSODE).The SparkPSODE algorithm is a parallel algorithm, in which the RDD and island models are employed.The island model is used to divide the global population into several subpopulations, which are applied to reduce the computational time by corresponding to RDD's partitions.To preserve population diversity and avoid premature convergence, the evolutionary strategy of DE is integrated into SparkPSODE.Finally, SparkPSODE is conducted on a set of benchmark problems on LSGO and show that, in comparison with several algorithms, the proposed SparkPSODE algorithm obtains better optimization performance through experimental results.

The field of data analytics is currently going through a renaissance as a result of ever-increasing dataset sizes, the value of the models that can be trained from those datasets, and a surge in flexible, distributed programming models. In particular, the Apache Hadoop and Spark programming systems, as well as their supporting projects (e.g. HDFS, SparkSQL), have greatly simplified the analysis and transformation of datasets whose size exceeds the capacity of a single machine. While these programming models facilitate the use of distributed systems to analyze large datasets, they have been plagued by performance issues. The I/O performance bottlenecks of Hadoop are partially responsible for the creation of Spark. Performance bottlenecks in Spark due to the JVM object model, garbage collection, interpreted/managed execution, and other abstraction layers are responsible for the creation of additional optimization layers, such as Project Tungsten. Indeed, the Project Tungsten issue tracker states that the "majority of Spark workloads are not bottlenecked by I/O or network, but rather CPU and memory".

Apache Spark is an increasingly popular fast big data analytics engine, which focuses on a large-scale data processing. Currently, Spark's memory management mainly oriented to single application, and do not directly support the typical scenarios of multiple data processing applications. In this paper, we propose an extension of Apache Spark, called Shared Memory Spark (SMSpark). SMSpark introduces shared memory based on RDD data sharing between applications. We conducted experiment in a cluster using two typical applications. Experimental results show that compared to Spark, SMSpark gains better performance.

The in-memory data processing framework, Apache Spark, has been stealing the limelight for low-latency interactive applications, iterative and batch computations. Our early experience study [17] has shown that Apache Spark can be enhanced to leverage advanced features (e.g., RDMA) on highperformance networks (e.g., InfiniBand and RoCE) to improve the performance of shuffle phase. With the fast evolving of the Apache Spark ecosystem, the Spark architecture has been changing a lot. This motivates us to investigate whether the earlier RDMA design can be adapted and further enhanced for the new Apache Spark architecture. We also aim to improve the performance for various Spark workloads (e.g., Batch, Graph, SQL). In this paper, we present a detailed design for high-performance RDMA-based Apache Spark on high-performance networks. We conduct systematic performance evaluations on three modern clusters (Chameleon, SDSC Comet, and an in-house cluster) with cutting-edge InfiniBand technologies, such as latest IB EDR (100 Gbps) network, recently introduced Single Root I/O Virtualization (SR-IOV) technology for IB, etc. The evaluation results show that compared to the default Spark running with IP over InfiniBand (IPoIB), our proposed design can achieve up to 79% performance improvement for Spark RDD operation benchmarks (e.g., GroupBy, SortBy), up to 38% performance improvement for batch workloads (e.g., Sort and TeraSort in Intel HiBench), up to 46% performance improvement for graph processing workloads (e.g., PageRank), up to 32% performance improvement for SQL queries (e.g., Aggregation, Join) on varied scales (up to 1,536 cores) of bare-metal IB clusters. Performance evaluations on SR-IOV enabled IB clusters also show 37% improvement achieved by our RDMA-based design. Our RDMA-based Spark design is implemented as a pluggable module and it does not change any Spark APIs, which means that it can be combined with other existing enhanced designs for Apache Spark and Hadoop proposed in the community. To show this, we further evaluate the performance of a combined version of ‘RDMA-Spark+RDMA-HDFS’ and the numbers show that the combination can achieve the best performance with up to 82% improvement for Intel HiBench Sort and TeraSort on SDSC Comet cluster.

Large-scale data processing techniques, currently known as Big-Data, are used to manage the huge amount of data that are generated by sequencers. Although these techniques have significant advantages, few biological applications have adopted them. In the Bioinformatic scientific area, Multiple Sequence Alignment (MSA) tools are widely applied for evolution and phylogenetic analysis, homology and domain structure prediction. Highly-rated MSA tools, such as MAFFT, ProbCons and T-Coffee (TC), use the probabilistic consistency as a prior step to the progressive alignment stage in order to improve the final accuracy. In this paper, a novel approach named PPCAS (Probabilistic Pairwise model for Consistency-based multiple alignment in Apache Spark) is presented. PPCAS is based on the MapReduce processing paradigm in order to enable large datasets to be processed with the aim of improving the performance and scalability of the original algorithm.

Big Data paradigm has placed pressure on well-established relational databases during the last decade. Both academia and industry have proposed several alternative database schemes in order to model the captured data more efficiently. Among these approaches, graph databases seem the most promising candidate to supplement relational schemes. Within this study, a comparison is performed among Neo4j, one of the leading graph databases, and Apache Spark, a unified engine for distributed large-scale data processing environment, in terms of processing limits. The results reveal that Neo4j is limited to the physical RAM memory of the processing environment. Yet, until this limit is reached, the processing engine of Neo4j outperforms the Apache Spark engine.

Apache Spark is a leading open-source data processing engine used for batch processing, machine learning, stream processing, and large-scale SQL (structured query language).It has been designed to make big data processing quicker and easier.Since its inception, Spark has gained huge popularity as a big data processing framework and is extensively used by different industries and businesses that are dealing with large volumes of data.This paper will exhibit actionable solutions to maximize our chances of reducing computation time by optimizing Spark jobs.The strategy lays out different run stages, wherein each run stage builds upon the previous and improves the computation time by making new enhancements and recommendations.

Today, big and small organizations alike collect huge amounts of data, and they do so with one goal in mind: extract "value" through sophisticated exploratory analysis, and use it as the basis to make decisions as varied as personalized treatment and ad targeting. Unfortunately, existing data analytics tools are slow in answering queries, as they typically require to sift through huge amounts of data stored on disk, and are even less suitable for complex computations, such as machine learning algorithms. These limitations leave the potential of extracting value of big data unfulfilled.

Big data processing clusters are suffering from a long job completion time due to the inefficient utilization of the RDMA capability. Our production measurement results in a large-scale cluster with hundreds of server nodes to process large-scale jobs have shown that the existing deployment of RDMA technique results in a long-tail job completion time, with some jobs even taking up more than twice the average time to complete. In this paper, we present the design and implementation of Turbo, an efficient communication framework for the large-scale data processing cluster to achieve high performance and scalability. The core of Turbo's approach is to leverage a dynamic block-level flowlet transmission mechanism and a non-blocking communication middleware to improve the network throughput and enhance system's scalability. Furthermore, Turbo ensures high system reliability by utilizing an external shuffle service as well as TCP serving as a backup. We integrate Turbo into Apache Spark and evaluate Turbo in a small-scale testbed and a large-scale cluster consisting of hundreds of server nodes. The small-scale testbed evaluation results show that Turbo improves the network throughput by 15.1% while maintaining high system reliability. The large-scale production results have shown Turbo can reduce the job completion time by 23.9% and increase the job completion rate by 2.03× over the existing RDMA solutions.

Fuzzy clustering is an invaluable data mining technique that allows each data point to belong to more than one cluster with some degree of membership. It is widely employed in exploratory data mining to discover overlapping communities in social networks, find structure in spectral data, and capture user interests in recommendation systems. Nowadays, the variety and volume of data are increasing at a tremendous rate. Data is power; the massive data, along with an effective technique, can unravel valuable information. The existing fuzzy clustering algorithms do not perform well on massive heterogeneous datasets. Processing an enormous amount of data is beyond the capacity of a single processor. The need of the hour is to develop fuzzy clustering techniques that can work on a distributed framework for Big Data processing and can handle heterogeneous data. In this research, we evaluate the performance of the recently proposed algorithm for the Fuzzy clustering of mixed-mode data FCMD-MD (D’Urso and Massari in Inf Sci 505:513–534, 2019) with different real-world datasets. We develop a distributed FCMD-MD, a fuzzy clustering algorithm for mixed-mode data in Apache SPARK. The experimental results show that the algorithm is scalable, performs well in a distributed environment, and clusters enormous heterogeneous data with high accuracy. We also compared the performance of distributed FCMD-MD and the distributed k-medoid algorithm.

One of the main challenges in Big data is the processing speed and scalability. Solid State Drive (SSD) helps for faster processing than HDD. Here along with SSD, Spark is also accompanied with hadoop framework for more scalability and fast processing. Apache Spark is a general-purpose engine for large-scale data processing on any cluster. It is a framework which can afford more than 8000 nodes in a cluster Spark allows for code reuse across batch, interactive, and streaming applications. Spark is much faster than MapReduce. It was generally coded from Java; Spark supports not only Java, but also Python and Scala, which is a newer language that contains some attractive properties for manipulating data. Spark runs up to 100 times faster than Hadoop MapReduce in memory and 10 times faster on disk. This paper tries to integrate spark with Hadoop ecosystem along the SSD. It increases the processing speed.

Cloud computing paradigm builds on the foundations of distributed computing, grid computing, virtualisation, service orientation, etc. Cloud storage, which is one of the most attractive cloud services, offers numerous benefits from both the technology and functionality perspectives such as increased availability, flexibility, and functionality. Therefore, Cloud-based platforms have become fundamental to collaborative and data management systems. However, the new Cloud environment creates major collaborative and data management challenges such as efficient and reliable remote storage, online data query processing, data integrity, outsourcing computation, and secure virtualisation, among others. Such issues need to be carefully studied and solved to enable the wide deployment and adoption of Cloud computing in businesses, industry, etc. The purpose of this special issue is to publish recent advances in Cloud-based collaborative systems and Cloud storage. The special issue comprises five high-quality papers, which are arranged as follows. De Maio et al in the first paper1 “Social Media Marketing through Time-Aware Collaborative Filtering,” define a time-aware collaborative filtering for estimating users' interest along the time in Twitter. Their approach uses text analysis services to semantically annotate tweets' content and to track concepts considering post frequencies along the time. A model-based approach implementing K-Nearest Neighbors is used to estimate user's similarity representing their profile by sampling user's interest with three different techniques: Vectorial Representation, Symbolic Aggregation Approximation, and Median. The authors present experimental results comparing these techniques and performing model training in different time windows. The second paper2 “SERNOTATE: An Automated Approach for Business Service Description Annotation for efficient Service Retrieval and Composition” by Chotipant et al proposes an automated approach, namely, SERNOTATE, for business service description annotation for efficient service retrieval and composition. To that end, the authors employ new semantic-based linking approaches, namely, Extended Case-based Reasoning, Vector-based and Classification-based, that automatically annotate business services to relevant service concepts. The experimental results test and validate the applicability of the proposed approaches to the automatic annotation of business service descriptions to service concepts on a real-world dataset. In the third paper,3 “Sensor Data Management in the Cloud: Data Storage, Data Ingestion, and Data Retrieval,” the authors address an increasing need to capture, store, and analyse the dynamic semistructured data from sensors. A similar growth of semistructured data in the modern Web has led to the creation of NoSQL data stores for scalability, availability, and performance, whereas large-scale data processing frameworks for parallel analysis. The authors study how sensor data management can benefit from MongoDB with Apache Spark technologies, specifically for ingesting high-velocity sensor data and parallel retrieval of high volume data. The performance of MongoDB sharding and no-sharding databases with Apache Spark are evaluated to identify the right software environment for sensor data management. Park et al in the fourth paper,4 “Study on the SDN-IP based Solution of Well-known Bottleneck Problems in Private Sector of National R&E Network for Big Data Transfer,” deal with issues arising in Software Defined Networking (SDN). The key feature of SDN networks is the provisioning of dynamic service architecture. Therefore, it is more appropriate for the big data science computing trends, for handling today's big data science, which requires massive parallel processing and a constant demand for additional capacity and on demand connectivity. The authors bring the experience from KREONET (Korea Research Environment Open Network) and Korea National R&E Network, which adopt SDN as a platform service for big data scientific application. The big data application for KREONET SDN service comprises high-energy physics, astrophysics, biology/genetics, meteorology, artificial satellite data, etc. The last paper5 “Improved Outsourced Private Set Intersection Protocol Based on Polynomial Interpolation” by Yang et al investigates issues in private set intersection (PSI) protocols, which enable two parties to compute the intersection of their inputs without compromising anything about the datasets beyond the intersection. The authors present a variant of delegated private set intersection protocol secure in the semi-honest model under RSA assumption as well as an efficient and secure outsourcing computation algorithm for RSA crypto-system. Based on this algorithm, the authors transform a variant of delegated private set intersection protocol into an improved outsourced one, which enables the clients to perform only simple modular multiplication for computing during the execution of the protocol. Compared with the state of the art, the proposed protocol has a great advantage in efficiency. The editor of this special issue would like to thank all authors for their contributions and the reviewers for their constructive and useful feedback to authors. I would like to thank Prof Geoffrey Fox, the Editor-in-Chief of Concurrency and Computation: Practice and Experience, for the opportunity to edit this special issue and his support.

This article introduces an advanced solution for anonymizing large-scale sensitive data, addressing the limitations of traditional approaches when applied to vast datasets. By leveraging the Spark distributed computing framework, we propose a method that parallelizes the data anonymization process, enhancing efficiency and scalability. Utilizing Spark's resilient distributed datasets (RDD), the approach integrates two primary operations, Map_RDD and ReduceByKey_RDD, to execute the anonymization tasks. Our comprehensive experimental evaluation demonstrates our solution's effectiveness and improved performance in preserving data privacy while balancing data utility and confidentiality. A significant contribution of our study is the development of a wide array of solutions for data owners, particularly notable for a 500 MB dataset at an anonymity level of K=100, where our methodology produces 832 unique solutions. This study also opens avenues for future research in applying different privacy models within the Spark ecosystem, such as l-diversity and t-closeness.

This work was partially funded by the Spanish Ministry of Economy, Industry and Competitiveness under the grant TIN2016-79637-P ”Towards Unification of HPC and Big Data Paradigms”; the Spanish Ministry of Education under the FPU15/00422 Training Program for Academic and Teaching Staff Grant; the Advanced Scientific Computing
\nResearch, Office of Science, U.S. Department of Energy, under Contract DE-AC02-06CH11357; and by DOE with agreement No. DE-DC000122495, program manager Laura Biven.