Comparative Analysis of ETL Tools in Big Data Analytics

The transition to digital society is characterised by the development of new methods and tools for big data processing. New technologies have a substantial impact on the education sector. The article represents the results of applying big data to analyse and transform the learning content of Moscow’s schools. The analysis of the school curriculum comprised the following: (a) identifying one-topic lesson scripts, (b) analysing cross-disciplinary connections between subjects, (c) verifying the compliance of the lesson script digital content to the Federal Educational Standards. The analysed material included 36,644 lesson scripts. The analysis has been conducted using specifically designed digital tools featuring data mining algorithms. The article considers the issue of applying data mining algorithms to analyse school curriculum for the improvement of its quality.

Big Data processing has become crucial in many domains because the amount of the produced data has enormously increased almost everywhere. The effective selection of the right Big Data processing tool is hard due to the high number and large variety of the available state-of-the-art tools. Many research results agree that there is no one best Big Data solution for all needs and requirements. It is therefore essential to be able to navigate more efficiently in the world of Big Data processing tools. In this paper, we present a map of current Big Data processing tools, recommended according to their capabilities and advantageous properties identified in previously published academic benchmarks. This map—as a navigation diagram—is aimed at helping researchers and practitioners to filter a large amount of available Big Data processing tools according to the requirements and properties of their tasks. Additionally, we provide recommendations for future experiments comparing Big Data processing tools, to improve the navigation diagram.

With the development of peta- and exascale size computational systems there is growing interest in running Big Data and Artificial Intelligence (AI) applications on them. Big Data and AI applications are implemented in Java, Scala, Python and other languages that are not widely used in High-Performance Computing (HPC) which is still dominated by C and Fortran. Moreover, they are based on dedicated environments such as Hadoop or Spark which are difficult to integrate with the traditional HPC management systems. We have developed the Parallel Computing in Java (PCJ) library, a tool for scalable high-performance computing and Big Data processing in Java. In this paper, we present the basic functionality of the PCJ library with examples of highly scalable applications running on the large resources. The performance results are presented for different classes of applications including traditional computational intensive (HPC) workloads (e.g. stencil), as well as communication-intensive algorithms such as Fast Fourier Transform (FFT). We present implementation details and performance results for Big Data type processing running on petascale size systems. The examples of large scale AI workloads parallelized using PCJ are presented.

MapReduce is emerging as a prominent tool for big data processing. Data locality is a key feature in MapReduce that is extensively leveraged in data-intensive cloud systems: it avoids network saturation when processing large amounts of data by co-allocating computation and data storage, particularly for the map phase. However, our studies with Hadoop, a widely used MapReduce implementation, demonstrate that the presence of partitioning skew (Partitioning skew refers to the case when a variation in either the intermediate keys’ frequencies or their distributions or both among different data nodes) causes a huge amount of data transfer during the shuffle phase and leads to significant unfairness on the reduce input among different data nodes. As a result, the applications severe performance degradation due to the long data transfer during the shuffle phase along with the computation skew, particularly in reduce phase. In this paper, we develop a novel algorithm named LEEN for locality-aware and fairness-aware key partitioning in MapReduce. LEEN embraces an asynchronous map and reduce scheme. All buffered intermediate keys are partitioned according to their frequencies and the fairness of the expected data distribution after the shuffle phase. We have integrated LEEN into Hadoop. Our experiments demonstrate that LEEN can efficiently achieve higher locality and reduce the amount of shuffled data. More importantly, LEEN guarantees fair distribution of the reduce inputs. As a result, LEEN achieves a performance improvement of up to 45 % on different workloads.

ETL (Extract Transform Load) is the widely used standard process for creating and maintaining a Data Warehouse (DW). ETL is the most resource, cost and time demanding process in DW implementation and maintenance . Now a days, many Graphical User Interfaces (GUI) based solutions are available to facilitate the ETL processes. In spite of the high popularity of GUI based tool, there is still some downside of such approach. This paper focuses on alternative ETL developmental approach taken by hand coding. In some context, it is appropriate to custom develop an ETL code which can be cheaper, faster and maintainable. Some well-known code based open source ETL tool (Pygrametl, Petl, Scriptella, R_etl) developed by the academic world has been studied in this article. Their architecture and implementation details are addressed here. The aim of this paper is to present a comparative evaluation of these code based ETL tools. Not to acclaim that code based ETL is superior to GUI based approach. It depends on the particular requirement, data strategy and infrastructure of any organization to choose the path between Code based and GUI based approach.

Extract, Transform, Load (ETL) processes form the backbone of data manage- ment and consolidation in today’s data-driven enterprises with prevalent large- scale data pipelines. One of the widely used ETL tools is Microsoft SQL Server Integration Services (SSIS), yet its optimization for performance for large-scale data loads remains a challenge. As the volumes of data grow exponentially, inefficient ETL processes create bottlenecks, increased processing time, and ex- haustion of system resources. This work discusses major SSIS optimizations that minimize ETL processing time, allowing for effective and scalable data integration. One of the key areas of optimization is data flow optimization, such as lever- aging the use of the Fast Load mode in OLE DB Destination to perform batch inserts instead of row-by-row. Similarly, Bulk Insert operations can signifi- cantly reduce data movement time. Additionally, buffer size and DefaultBuffer- MaxRows tuning allows SSIS to process data in memory more efficiently, thereby minimizing disk I/O operations. Another major area of focus is source query optimization. With the utiliza- tion of indexed views, partitioned tables, and filtering in the WHERE clause, unnecessary data extraction is avoided, restricting the load on the source sys- tem. NOLOCK hints also minimize database contention in high-concurrency environments. Parallel execution of multiple operations within SSIS can also accelerate execution, with multithreading and batch processing enabling con- current data conversion. Lookup transformations, a common performance bottleneck, can be opti- mized using cache mode, where reference data is pre-loaded instead of querying the database for each row. Furthermore, replacing row-based transformations with set-based operations significantly reduces processing overhead. For incremental data loading, change tracking or CDC (Change Data Cap- ture) enables altered record processing in place of full set loads. This saves time in processing and optimizes utilization of resources. ETL logging and error- handling mechanisms play an important role as well; selective SSIS logging and event-based error-handling mechanisms can prevent performance degradation due to overlogging. Lastly, SSIS package configurations can be tuned by having proper indexing of destination tables, turning off unnecessary constraints during loading, and applying table partitioning to maximize parallel loads of data. By utilizing these SSIS optimizations, organizations can reduce ETL pro- cessing by significant quantities, optimize data pipelines, and overall enhance enterprise-level data integration performance. These approaches make large- scale big-data scale data pipelines have very low latency, thus making SSIS a more efficient and scalable solution for enterprise-level data workflows.

The main purpose of research was to achieve data integration between Database of Topographic Objects (BDOT) scale 1:500 and BDOT scale 1:10000. The presented approach assumes the construction of a complex Extract, Transform, Load (ETL) process for harmonization of built-up areas that supplies the 1:10k dataset with the most accurate geometric data derived from the base map. Full data integration of attributes between BDOT500 and BDOT10k databases was achieved by constructing an ETL process using Talend Open Studio for Data Integration [17]. Geospatial vector data stored in the shapefiles were input to the harmonization process. The automated approach requires minimal user intervention to build the combined output. The data quality is maintained by the appropriate ETL components. For examples that were carried out, process was effective and consistent with predictions. However, for more complex models more attention should be paid to preparing an appropriate remapping table.

Cloud resources are more often used for large scale computing and data processing. However, the usage of the cloud is different than traditional High-Performance Computing (HPC) systems and both algorithms and codes have to be adjusted. This work is often time-consuming and performance is not guaranteed. To address this problem we have developed the PCJ library (Parallel Computing in Java), a novel tool for scalable high-performance computing and big data processing in Java. In this paper, we present a performance evaluation of parallel applications implemented in Java using the PCJ library. The performance evaluation is based on the examples of highly scalable applications that run on the traditional HPC system and Amazon AWS Cloud. For the cloud, we have used Intel x86 and ARM processors running Java codes without changing any line of the program code and without the need for time-consuming recompilation. Presented applications have been parallelized using the PGAS programming model and its realization in the PCJ library. Our results prove that the PCJ library, due to its performance and ability to create simple portable code, has great promise to be successful for the parallelization of various applications and run them on the cloud with a similar performance as for HPC systems.

With the continuous development of diagnosis, prediction and systems management technologies for complex pipeline systems and driven by the context of Industry 4.0, digital twin technology has become a research hotspot in the field of smart manufacturing and intelligent operation and maintenance of pipeline systems. Digital twin can be defined as an adaptive model of complex physical systems. The prospect of digital twin and its impact on society is brought closer to reality with recent advances in multiphysics simulation, artificial intelligence, big data cybernetics, data processing and management tools. With a case study on Sino-Myanmar pipelines, the author uses artificial intelligence models on the basis of digital twin technology to predict the pipeline transmission demand to make sure the smooth operation of pipelines with large topographic fluctuations and geological complexity. The objective is to achieve intelligent operation such as real-time autonomous optimization of pipeline operation and control of safety warning, thus realizing data analysis and intelligent diagnosis, as well as providing a reference for the application of demand prediction of complex pipeline networks and system health management in digital twin technology.

This technical article critically examines the strengths and advantages of the modern data analytics platform in comparison to legacy ETL (Extract, Transform, Load) tools through an in-depth exploration of key features, performance, scalability, ease of use, and integration capabilities. This article will highlight how the advancement in technology has resulted in offering superior capabilities in terms of real-time processing, scalability, agility, advanced analytics, unified platforms, governance, security, deployment flexibility, cost efficiency, and ecosystem integration compared to traditional ETL processes. These advantages make them crucial for organizations aiming to maximize the value of their data assets and foster innovation in their business operations. There are diverse options available for implementing a new data pipeline or upgrading the existing orchestration mechanism. This article tends to do a thorough analysis by carefully evaluating these benefits, weaknesses, and considerations. Researchers and industry specialists can make informed decisions about selecting the appropriate ETL components or considering alternative data integration and processing approaches (such as ELT, streaming data platforms, or modern data analytics tools) that better suit their specific requirements and use cases.

Now a day's data is growing very rapidly, where processing and analyzing data to get useful information is the main task. There are many big data processing tools and framework such as Hadoop, Hive, Cassandra etc. Spark is one of the fastest big data processing framework in cluster computation.Basic Idea is to analyze the performance of java virtual machine (JVM) [1], by characterizing java virtual machine using SparkBench benchmark on Apache Spark™ [2]. Java virtual machine is a core execution platform for spark application. When we run the spark application on java virtual machine, its behavior is affected, which needs to be monitored to analyze the JVM performance. Here we are considering Machine Learning workloads like K-Means, Matrix Factorization and Logistic Regression. Main goal here is to analyze the machine learning workloads end to end across the cluster, with respect to following parameters such as garbage collection, memory such as heap usage, CPU process time. Characterization of JVM is done with spark cluster setup and HDFS is used as storage with distributed Hadoop cluster setup.

Ambient intelligence is an emerging platform that provides advances in sensors and sensor networks, pervasive computing, and artificial intelligence to capture the real time climate data. This result continuously generates several exabytes of unstructured sensor data and so it is often called big climate data. Nowadays, researchers are trying to use big climate data to monitor and predict the climate change and possible diseases. Traditional data processing techniques and tools are not capable of handling such huge amount of climate data. Hence, there is a need to develop advanced big data architecture for processing the real time climate data. The purpose of this paper is to propose a big data based surveillance system that analyzes spatial climate big data and performs continuous monitoring of correlation between climate change and Dengue. Proposed disease surveillance system has been implemented with the help of Apache Hadoop MapReduce and its supporting tools.

The interoperability of geographic data sets is required between geo-information system applications to use geographic data sets effectively. Thus, international standardisation organisation/technical committee 211 standards supporting model-driven architecture were examined to test the ability of producing interoperable geographic data sets. With Turkey case, unified modelling language application schemas were designed for base geographic data themes and encoded to data interchange model based on geography markup language. To test the applicability of the open data models, extract-transform-load (ETL) tools were developed and applied for case applications such as topographic map and web urban atlas. This study gives a methodology and indicates that ETL tools should be created to enable multiple uses of geo-data sets without spending time and labour. However, data model design should be refined and kept as simple as possible because data transformation is laborious to use the models in the applications.

In this study, we delve into the realm of efficient Big Data Engineering and Extract, Transform, Load (ETL) processes within the healthcare sector, leveraging the robust foundation provided by the MIMIC-III Clinical Database. Our investigation entails a comprehensive exploration of various methodologies aimed at enhancing the efficiency of ETL processes, with a primary emphasis on optimizing time and resource utilization. Through meticulous experimentation utilizing a representative dataset, we shed light on the advantages associated with the incorporation of PySpark and Docker containerized applications. Our research illuminates significant advancements in time efficiency, process streamlining, and resource optimization attained through the utilization of PySpark for distributed computing within Big Data Engineering workflows. Additionally, we underscore the strategic integration of Docker containers, delineating their pivotal role in augmenting scalability and reproducibility within the ETL pipeline. This paper encapsulates the pivotal insights gleaned from our experimental journey, accentuating the practical implications and benefits entailed in the adoption of PySpark and Docker. By streamlining Big Data Engineering and ETL processes in the context of clinical big data, our study contributes to the ongoing discourse on optimizing data processing efficiency in healthcare applications. The source code is available on request.

Applications in the World Wide Web aggregate vast amounts of information from different data sources. The aggregation process is often implemented with Extract, Transform and Load (ETL) processes. Usually ETL processes require information for aggregation available in structured formats, e. g. XML or JSON. In many cases the information is provided in natural language text which makes the application of ETL processes impractical. Due to the fact that information is provided in natural language, Information Extraction (IE) systems have been evolved. They make use of Natural Language Processing (NLP) tools to derive meaning from natural language text. State-of-the-art NLP tools apply Machine Learning methods. These NLP tools perform on newspapers with good quality, but they drop accuracy in other domains. However, to improve the quality for IE systems in specific domains often NLP tools are trained on domain specific text which is a time consuming process. This paper introduces an approach using a Continuous Integration pipeline for organizing and monitoring the annotation process on domain specific corpora.