Performance Analysis on Parallel Data Loading based on Concurrency Features

Data warehouse store historical records on which analysis queries are executed. Data warehouse are populated from the transaction database through a process called as Extract Transform and Load (ETL). Extract is to identify the transaction database records to be moved to the data warehouse. Transform is to make the records normalized to suit the data warehouse environment. Load is to actually store the records in the data warehouse. Today business wants to perform real time analysis of data. Availability of updated records in the data warehouse is necessary for real time analysis. This could be achieved through the process of near real time (ETL), i.e., to improve existing ETL process. Improvement to existing ETL could be achieved in many ways such as to increase the frequency of loads, to identify relevant changes that are required for analysis and move only those changes. Only through increasing the frequency of load would not be useful in many cases. Hence we should identify the changes and then move if they are useful for analysis making it smart ETL. In this paper we study three such techniques. First one is to create a replica of dimension table in data warehouse and move the changes to the replica this reduces the query response time. Second is to load the data in parallel so that loading time could be reduced. Third is to identify changes and trigger the loading process. We have used the first approach to create replica and loaded the data in parallel and observed that by loading in parallel not only does the loading time reduces but also the query response time. Then in the first approach is the query response time reduces to a certain limit methods suggest to move the data from replica dimensions to the original dimensions. Here we bring in right time trigger to move instead of just the query response time. We found that when we have a combined approach to query response time and right time trigger the number of moves are reduced.

Real-time analysis of data is the new trend to get useful insights in very less time spend on data preprocessing. Analysis of data requires the movement of data from various heterogeneous/homogenous sources to a common place known as the data warehouse. Data source for data warehouse is the transaction processing systems. Movement of data from the transactional database to the data warehouse is done using the process of extract, transform, and load (ETL). ETL previously was done during of peak hours like a night load or on weekends. The requirement of real-time analysis demands the ETL to be fast and not wait for off-peak hours. This leads to the concept of near real-time ETL, and here techniques are employed to identify the potential changed data at the transaction database and move it to the analysis database with a very minimal delay. This movement of data in real time from multiple sources in an incremental form could lead to anomalies in the data warehouse. This work discusses the various causes of anomalies and solutions to overcome them. Our main contribution is the application of loading data into temporary tables for reducing query execution time in case of overcoming refreshment anomalies.

In recent years, the application and wide adoption of Internet of Things (IoT)-based technologies have increased the proliferation of monitoring systems, which has consequently exponentially increased the amounts of heterogeneous data generated. Processing and analysing the massive amount of data produced is cumbersome and gradually moving from classical ‘batch’ processing—extract, transform, load (ETL) technique to real-time processing. For instance, in environmental monitoring and management domain, time-series data and historical dataset are crucial for prediction models. However, the environmental monitoring domain still utilises legacy systems, which complicates the real-time analysis of the essential data, integration with big data platforms and reliance on batch processing. Herein, as a solution, a distributed stream processing middleware framework for real-time analysis of heterogeneous environmental monitoring and management data is presented and tested on a cluster using open source technologies in a big data environment. The system ingests datasets from legacy systems and sensor data from heterogeneous automated weather systems irrespective of the data types to Apache Kafka topics using Kafka Connect APIs for processing by the Kafka streaming processing engine. The stream processing engine executes the predictive numerical models and algorithms represented in event processing (EP) languages for real-time analysis of the data streams. To prove the feasibility of the proposed framework, we implemented the system using a case study scenario of drought prediction and forecasting based on the Effective Drought Index (EDI) model. Firstly, we transform the predictive model into a form that could be executed by the streaming engine for real-time computing. Secondly, the model is applied to the ingested data streams and datasets to predict drought through persistent querying of the infinite streams to detect anomalies. As a conclusion of this study, a performance evaluation of the distributed stream processing middleware infrastructure is calculated to determine the real-time effectiveness of the framework.

ObjectiveTo access Electronic Health Record (EHR) data, hospitals have implemented Clinical Data Warehouses (CDWs) using Extract Transform and Load (ETL) processes. While ETL performances are typically evaluated individually, our study examines the cumulative impact of ETLs on data availability. MethodsUsing a real multi-hospital CDW as a case study, we modeled EHR data processing from the software sources to the CDW's data store. We simulated a scenario where researchers aimed to reconstruct breast cancer care trajectories using EHR data. We calculated the size and characteristics of the data store population, and compared them to the original population. ResultsEHR data are recorded in various software depending on data category, hospital, and year, each requiring specific series of ETLs for integration in the CDW. Despite acceptable transfer rates for each ETL (range 73 %-100 %), cumulative losses led to study populations in the data store being up to 90 % smaller than anticipated when researchers required data exhaustivity for patients. Population size decreased steeply with the more data categories required. No difference was found in population characteristics between the data store and the original cohorts. Discussion & ConclusionResearchers should scrutinize data availability in CDWs as missing data could result from outsourced care, incomplete input, or underperforming ETLs. Integrating more data sources in CDWs increases the number of data routes, necessitating time for ETL implementation and maintenance, and increases data loss risks. Though commonly perceived as a “black box”, data transformation can significantly influence the reliability of populations studied in CDWs. Public interest SummaryTo access data generated during care, researchers build Clinical Data Warehouses (CDWs). CDWs are infrastructures composed of a series of processing steps to extract the data from the data source, transform it according to the needs and load it into a data store. Usually, the performances of these processing steps are evaluated one a time. However, each data point goes through a series of processing steps before being made available for research. In this study, we aim to evaluate the impact of the entire data processing pipeline on the availability of data points in a CDW by simulating a study on breast cancer and evaluating the impact on the size and the characteristics of the final cohort. The cumulative losses of the processing steps resulted in a population 90 % smaller than anticipated. The characteristics of the final population showed no difference to those of the original cohort.

Data migration schemes are critical to balance the load in storage clusters for performance improvement. However, as NAND flash based SSDs are widely deployed in storage systems, extending the lifespan of SSD storage clusters becomes a new challenge for data migration. Prior approaches designed for HDD storage clusters, however, are inefficient due to excessive write amplification during data migration, which significantly decrease the lifespan of SSD storage clusters. To overcome this problem, we propose EDM, an endurance aware data migration scheme with careful data placement and movement to minimize the data migrated, so as to limit the worn-out of SSDs while improving the performance. Based on the observation that performance degradation is dominated by the wear speed of an SSD, which is affected by both the storage utilization and the write intensity, two complementary data migration policies are designed to explore the trade-offs among throughput, response time during migration, and lifetime of SSD storage clusters. Moreover, we design an SSD wear model and quantitatively calculate the amount of data migrated as well as the sources and destinations of the migration, so as to reduce the write amplification caused by migration. Results on a real storage cluster using real-world traces show that EDM performs favorably versus existing HDD based migration techniques, reducing cluster-wide aggregate erase count by up to 40%. In the meantime, it improves the performance by 25% on average compared to the baseline system which achieves almost the same effectiveness of performance improvement as previous migration techniques.

The rise in mergers and acquisitions (M&A) has highlighted the critical need for robust data management and migration tools. Data consolidation and modernization are central to achieving synergies in M&A transactions. SAP Master Data Governance (SAP MDG) emerges as a strategic enabler in streamlining and accelerating the data migration process while ensuring data integrity and quality. This white paper explores SAP MDG's role in addressing data migration challenges during M&A, its architecture, capabilities, and strategic value for data modernization initiatives. The paper provides detailed examples, discusses implementation strategies, and concludes with an evaluation of SAP MDG's long-term benefits for organizations engaging in M&A.

In order to fulfil real time signal processing tasks such as clutter rejection, moving target detection (MTD) and constant false alarm rate (CFAR) control in airborne radar, an airborne radar parallel signal processing system (ARPS2) is proposed with DSP chips as its kernel processing nodes. The DSP chips are used with parallel architecture. Each node has its private input and output memory. It adopts several parallel techniques, such as parallel storage, parallel processing, parallel code loading and parallel data organization to achieve high efficiency. It has a simple structure, excellent flexibility and easiness in developing. ARPS2 is going to be applied to an airborne radar. It can also be applied to perform high-speed real time signal processing algorithms in other kinds of radar.

Metadata is the command center in the process of data warehousing,and which is very helpful to data ETL (Extraction, Transformation and Loading), data storage management,data analysis and data mining.CWM (Common Warehouse Meta-model) is an open industry metadata standard,and which is widely used currently,public meta-models and its rules defined in CWM can properly support data transformation and cleansing.One metadata-driven development approach is proposed to support the design and development of data ETL process.One data transformation module is realized, and which is used to specify how to use metadata in ETL process. Finally, the development process and steps of using CWM to build ETL tool are given.

The increasing volume, variety, and velocity of data necessitate robust data integration and transformation solutions. As enterprises shift towards data-driven decision-making, efficient data processing has become a cornerstone of business success. Talend, a widely adopted open-source and commercial data integration tool, offers comprehensive features for Extract, Transform, Load (ETL) processes, enabling organizations to manage their data efficiently. This paper explores the role of Talend in modern enterprise solutions, emphasizing its functionalities, advantages, challenges, and case studies of successful implementations. The study examines the architecture of Talend, its ability to integrate with cloud-based and on-premises environments, and its impact on data governance and security. Furthermore, it evaluates Talend’s efficiency in handling big data and real-time processing compared to traditional ETL tools. The findings indicate that Talend provides enterprises with a scalable, cost-effective, and flexible solution for data transformation and integration. By leveraging Talend, organizations can ensure high data quality, streamline business operations, and enhance analytics capabilities, thereby gaining a competitive edge in the evolving digital landscape.

Practical Mechanisms for Reducing Processor–Memory Data Movement in Modern Workloads

Big data analytic applications give rise to large-scale extract-transformload (ETL) as a fundamental step to transform new data into a native representation. ETL workloads pose significant performance challenges on conventional architectures, so we propose the design of the unstructured data processor (UDP), a software programmable accelerator that includes multi-way dispatch, variable-size symbol support, flexible-source dispatch (stream buffer and scalar registers), and memory addressing to accelerate ETL kernels both for current and novel future encoding and compression. Specifically, UDP excels at branch-intensive and symbol and pattern-oriented workloads, and can offload them from CPUs. To evaluate UDP, we use a broad set of data processing workloads inspired by ETL, but broad enough to also apply to query execution, stream processing, and intrusion detection/monitoring. A single UDP accelerates these data processing tasks 20-fold (geometric mean, largest increase from 0.4 GB/s to 40 GB/s) and performance per watt by a geomean of 1,900-fold. UDP ASIC implementation in 28nm CMOS shows UDP logic area of 3.82mm2 (8.69mm2 with 1MB local memory), and logic power of 0.149W (0.864W with 1MB local memory); both much smaller than a single core. CCS CONCEPTS • Information systems → Extraction, transformation and loading; • Computer systems organization → Parallel architectures; • Hardware → Application specific processors; • Theory of computation → Pattern matching;

The growth of available information sources in the industry requires new efficient methods for data access by end-users whose ability to analyse data is at the core of making business decisions. Data analysis suffer from end-users' ability to efficiently clean and integrate and query data spread across a multitude of different data sources. Current centralised approaches, where an IT expert translates the requirements of domain experts into Extract-Transform-Load (ETL) processes to integrate the data and to apply predefined analytical reporting tools, are too heavy-weight and inflexible. To support interactive and efficient data exploration directly by domain experts, without IT expert involvement, a more flexible and modular information systems architecture is needed. The Optique platform reduces the time and cost of data access dramatically by automating the process of translating an information requirement into the retrieval of the relevant data. The key to this automated translation is Ontology-Based Data Access (OBDA), where the central idea is to use an ontology, a semantically rich conceptual model of the problem domain, and mappings, that describe the relationship between the ontology and the data sources, to allow users to formulate queries and receive answers in a domain-centric and intelligible form, without any knowledge of the implementational details of the data sources. We present the Optique platform as a next generation OBDA system featuring user-friendly query formulation interface(s), automated query translation over multiple sources using novel ontology and mapping management components, and distributed query optimisation and execution exploiting cloud resources for scale-out. At Statoil Exploration, the Optique platform integrates multiple large and complex data sources from different vendors, and allows G&G experts to execute and present detailed information needs in the subsurface domain directly into existing expert client tools and also into web portals for easy information distribution of central assets across the enterprise.

A growth in data volume, combined with increasing demand for real-time analysis (using the most recent data), has resulted in the emergence of database systems that concurrently support transactions and data analytics. These hybrid transactional and analytical processing (HTAP) database systems can support real-time data analysis without the high costs of synchronizing across separate single-purpose databases. Unfortunately, for many applications that perform a high rate of data updates, state-of-the-art HTAP systems incur significant losses in transactional (up to 74.6%) and/or analytical (up to 49.8%) throughput compared to performing only transactional or only analytical queries in isolation, due to (1) data movement be-tween the CPU and memory, (2) data update propagation from transactional to analytical workloads, and (3) the cost to main-tain a consistent view of data across the system. We propose Polynesia, a hardware-software co-designed system for in-memory HTAP databases that avoids the large throughput losses of traditional HTAP systems. Polynesia (1) di-vides the HTAP system into transactional and analytical pro-cessing islands, (2) implements new custom hardware that un-locks software optimizations to reduce the costs of update prop-agation and consistency, and (3) exploits processing-in-memory for the analytical islands to alleviate data movement overheads. Our evaluation shows that Polynesia outperforms three state-of-the-art HTAP systems, with average transactional/analytical throughput improvements of 1.7×/3.7×, and reduces energy consumption by 48% over the prior lowest-energy HTAP sys-tem.

Wearable body area network is a key component of the modern-day e-healthcare system (e.g., telemedicine), particularly as the number and types of wearable medical monitoring systems increase. The importance of such systems is reinforced in the current COVID-19 pandemic. In addition to the need for a secure collection of medical data, there is also a need to process data in real-time. In this article, we design an improved symmetric homomorphic cryptosystem and a fog-based communication architecture to support delay- or time-sensitive monitoring and other-related applications. Specifically, medical data can be analyzed at the fog servers in a secure manner. This will facilitate decision making, for example, allowing relevant stakeholders to detect and respond to emergency situations, based on real-time data analysis. We present two attack games to demonstrate that our approach is secure (i.e., chosen-plaintext attack resilience under the computational Diffie-Hellman assumption), and evaluate the complexity of its computations. A comparative summary of its performance and three other related approaches suggests that our approach enables privacy-assured medical data aggregation, and the simulation experiments using Microsoft Azure further demonstrate the utility of our scheme.

Big data analysis and optimization and platform components