Combating Dirty Data using Data Virtualization

In the era of big data, enterprises increasingly rely on near-real-time data ingestion pipelines to drive advanced analytics and machine learning models. The complexity and diversity of heterogeneous data sources pose significant challenges to maintaining high data quality and error resilience in these pipelines. This paper investigates strategies to ensure robust data quality and error management within autonomous self-schedulable libraries designed for handling diverse data formats. We explore architectural designs, best practices, and innovative techniques that enable seamless integration and real-time processing of disparate data sources. Key areas of focus include error detection and correction mechanisms, data validation frameworks, and resilient pipeline orchestration. Through comprehensive case studies and experimental evaluations, we demonstrate the efficacy of these strategies in enhancing the reliability and accuracy of data ingestion processes. Our findings provide a roadmap for enterprises seeking to optimize their data pipelines, ensuring they are equipped to handle the complexities of heterogeneous data environments with minimal human intervention.

Nowadays, the IoT ecosystem is evolving rapidly, with multiple heterogeneous sources producing high volumes of data and processes transforming this data into meaningful or “smart” information. These volumes of data, including IoT data, need to be stored in repositories that can host raw, unprocessed, relational and non-relational types of data, such as Data Lakes. Due to the weakness of metadata management, security & access control is one of the main challenges of Big Data storage architectures as Data Lakes can be replaced without oversight of the contents. Recently, the Blockchain technology has been introduced as an effective solution to build trust between different entities, where trust is either nonexistent or unproven, and to address security and privacy concerns. In this paper we introduce DLMetaChain, an extended Data Lake metadata mechanism that consists of data from heterogeneous data sources which interact with IoT data. The extended mechanism mainly focuses on developing an architecture to ensure that the data in the Data Lake is not modified or altered by taking into advantage the capabilities of the Blockchain.

The COviD-19 Ontology for cases and patient information (CODO) provides a model for the collection and analysis of data about the COVID-19 pandemic. The ontology provides a standards-based open-source model that facilitates the integration of data from heterogeneous data sources. The ontology was designed by analysing disparate COVID-19 data sources such as datasets, literature, services, etc. The ontology follows the best practices for vocabularies by re-using concepts from other leading vocabularies and by using the W3C standards RDF, OWL, SWRL, and SPARQL. The ontology already has one independent user and has incorporated real-world data from the government of India.

sRNAFinder is a new gene prediction system for systematic identification of noncoding genes in bacteria. Most noncoding RNAs in prokaryotes belong to a class of genes denoted as small RNAs (sRNAs). In the model organism Escherichia coli, over 70 sRNA genes have been identified, and the existence of many more has been hypothesized. While various sources of information have proven useful for prediction of novel sRNA genes, most computational approaches do not take advantage of the disparate sources of data available for identifying these noncoding RNA genes. We present a general probabilistic method for predicting sRNA genes in bacteria. The method, based on a general Markov model, is implemented in the computational tool sRNAFinder. sRNAFinder incorporates heterogeneous data sources for gene prediction, including primary sequence data, transcript expression data from microarray experiments, and conserved RNA structure information as determined from comparative genomics analysis. We demonstrate that sRNAFinder improves upon current tools for identifying small, noncoding genes in bacteria.

The analysis of modern production environments requires a clear and understandable uniform visualization of the information from all levels of the production. This visualization has to be on the one hand light weight to avoid cognitive overload of the users and on the other hand rich enough to provide information about all production information sources. To solve this issue we propose to use a unified component-based visualization where single items can be exchanged to fulfill the informational needs of the users from heterogeneous data sources.

Towards a content agnostic computable knowledge repository for data quality assessment

Machine-to-Machine (M2M) technology unremittingly motivates any time-place-objects connectivity of the devices in and around the world. Every day, a rapid growth of large M2M networks and digital storage technology, lead to a massive heterogeneous data depository, in which the M2M data are captured and warehoused in the diverse database frameworks as a magnitude of heterogeneous data sources. Hence, the M2M that handles Big Data might perform poorly or not according to the goals of their operator due to massive heterogeneous data sources may face various incompatibilities, such as data quality, processing and computational efficiency, analysis and feature extraction applications. Therefore, to address the aforementioned constraints, this paper presents a Big Data Analytical architecture based on Divide-and-Conquer approach. The designed system architecture exploits divide-and-conquer approach, where big data sets are first transformed into a several data blocks that can be quickly processed, then it classifies and reorganizes these data blocks from the same source. In addition, the data blocks are aggregated in a sequential manner based on a machine ID, and equally partitions the data using filtration and load balancing algorithms. The feasibility and efficiency of the proposed system architecture are implemented on Hadoop single node setup. The results show that the proposed system architecture efficiently extract various features (such as River) from the massive volume of satellite data.

Infoscopes will be the microscopes and telescopes of the information systems of the future. The emergence of information highways and multimedia computing has resulted in exponential growth in the availability of multimedia data. Most information in computers used to be alphanumeric. Increasingly information has been appearing in graphic, image, audio, and video forms. Many approaches are being proposed for storing, retrieving, assimilating, harvesting, and prospecting information from disparate data sources. Infoscopes will allow users to access information independent of the locations and types of data sources and will provide a unified picture of information to a user. Due to their ability to represent information at different levels of abstractions these systems must recover and assimilate information from disparate sources. In this chapter, we discuss requirements of these emerging information systems. We discuss basic architecture and data models for these systems. Finally, we briefly present a few examples of early infoscopes.

• Offers a new vision on automated data harmonization in Dataspace (DS) systems. • Introduces LLM-based methods for scalable DS ingestion of heterogeneous datasources. • Presents a system with Harmonizer, Transformer, Evaluator components for ingestion. • Demonstrated an automated data ingestion prototype using LLM agents. • Validates the system with healthcare use case harmonizing heterogeneous data sources. Dataspaces (DS) enable stakeholders to collaborate on innovative, data-driven services by integrating data across domains. However, the realization and adoption of DS remain challenging due to domain-specific heterogeneity at the system, service, and data levels. While system and service-level heterogeneity can often be addressed through standards, data-level heterogeneity, namely data structures and semantics variations, remains challenging. To effectively ingest data into the DS, two communication endpoints must correctly interpret each other’s data models, therefore, DS ecosystems rely on “harmonization”, the process of generating a unified target data model from heterogeneous sources and transforming incoming data accordingly. Currently, harmonization and transformation are performed manually whenever new data sources are integrated. This is time-consuming, costly, and difficult to scale, posing a critical barrier to the realization and adoption of DS in practice. This study proposes a novel methodology for automated data harmonization during ingestion into DS ecosystems. The approach integrates harmonization, transformation, and human-in-the-loop evaluation within an automated system powered by modern LLM-based AI agents. These agents address data-level heterogeneity and generate harmonized target data models, representing a substantial departure from current manually-handled data harmonization. The system is validated through a healthcare use case, demonstrating its practical feasibility for harmonization during data ingestion into the DS. Overall, this work provides a foundational step toward seamless, efficient, and scalable data integration in DS. By automating data harmonization, it delivers substantial value to industry digital solutions as well as domains where data heterogeneity persists, including IoT or Big Data platforms.

The processing strategy based on measurement metadata is a data stream engine running on Apache Storm, who is able to process measures in real-time. In the data stream context, the data have no an associated limit, they are al-ways arriving. The Attribute-Relation File Format (ARFF) is used by popular software like Weka, allowing offline analysis in the machine learning and data mining area. However, the ARFF file has a finite size. The CincamimisConversor library allows exporting from the data streams organized under a measurement interchange schema to a columnar-data organization in real-time. Here, an extension to the library is introduced for supporting the real-time translating and storing from the heterogeneous data streams to the ARFF file format. This is very useful, because through the library now is possible to collect data from heterogeneous data sources (e.g. Internet-of-Thing-IoT-devices) and export them in real-time for offline analysis in Weka. Even, this could foster a lot of educational applications among IoT, the measurement process with heterogeneous sources, data stream processing strategy, and Weka. A discrete simulation was carried out, obtaining promising results. It is just required at most 0.2387 ms for translating 5000 measures, while the storing operation for them consumed less than 0.2028 ms on a Solid-State disk.

Data quality is paramount in data-driven decision-making processes, especially when dealing with large volumes of data in environments like data warehouses and data lakes. These systems store vast amounts of raw and processed data from multiple sources, making data management and quality assurance complex yet critical. With the growing adoption of Artificial Intelligence (AI), new techniques and tools have emerged that can significantly enhance data quality. This paper discusses how AI can improve the quality of data within both data warehouses and data lakes by automating data cleansing, validation, anomaly detection, and ensuring consistency. It explores the benefits, challenges, and methodologies for integrating AI tools into these systems. Keywords: Data quality, AI in data quality, data warehouse, data lakes, big data, data processing, data cleansing, data profiling.

Researchers need to be able to integrate ever-increasing amounts of data into their institutional databases, regardless of the source, format, or size of the data. It is then necessary to use the increasing diversity of data to derive greater value from data for their organization. The processing of electronic data plays a central role in modern society. Data constitute a fundamental part of operational processes in companies and scientific organizations. In addition, they form the basis for decisions. Bad data quality can negatively affect decisions and have a negative impact on results. The quality of the data is crucial. This includes the new theme of data wrangling, sometimes referred to as data munging or data crunching, to find the dirty data and to transform and clean them. The aim of data wrangling is to prepare a lot of raw data in their original state so that they can be used for further analysis steps. Only then can knowledge be obtained that may bring added value. This paper shows how the data wrangling process works and how it can be used in database systems to clean up data from heterogeneous data sources during their acquisition and integration.

More than 16 million Americans served in World War II. Of these service members, over 400,000 were killed in action during the war. Today, more than 72,000 service members remain unaccounted for from World War II. The United States continues to diligently locate, recover, and identify missing personnel from World War II and other past conflicts to provide the fullest possible accounting. This work importantly provides closure and resolution to numerous US families. To fulfill this mission, massive amounts of information must be integrated from historical records, genealogy records, anthropological data, archeological data, odontology data, and DNA. These disparate data sources are produced and maintained by multiple agencies, with different data governance rules and different internal structuring of service member information. Previously, a manual approach had been undertaken to Extract, Transform, Load (ETL) records from these different data sources, which creates the potential for introduced human error. In addition, a large number of person-hours were required to synthesize this data on a biweekly basis. To address this issue, we implemented (i) a regex decision tree to translate genealogical relationships into DNA type availability and (ii) a machine learning approach for record-linkage between disparate data sources. This application is currently in production and greatly reduces person-hours needed and has a very low error rate for record translation and integration.

In the last years, enterprises increase their effort to collect large amounts of data from many heterogeneous data sources and store it in modern architectures like data lakes. However, this approach faces different drawbacks for finding and understanding data sources. Ontology-Based Data Access (OBDA) originating from the Semantic Web enables a homogeneous access to the data sources by using a mapping, called semantic model, between a data source and a target ontology. However, OBDA requires a detailed ontology, which is usually created by ontology engineers and domain experts resulting along with high effort for designing and maintaining. To overcome these limitations, we develop an approach consisting of a knowledge graph, which features an internal growing ontology and linked data-source specific semantic models. The ontology continuously evolves on-demand based on newly added data sources along with their corresponding semantic models, which are created by domain experts. To ensure the knowledge graph's stability, we develop an intuitive user-oriented assistant and combine it with a semi-supervised evolving strategy that assists the user with the help of external knowledge bases. We evaluate accuracy and usability of our approach by conducting a user study with a heterogeneous group of participants that define semantic models upon pre-defined data sets. The results show that semantic models become more objective and consistent with our provided user assistant and thus lead to a knowledge graph with higher interconnectivity and stability.

As the challenge of our time, Big Data still has many research hassles, especially the variety of data. The high diversity of data sources often results in information silos, a collection of non-integrated data management systems with heterogeneous schemas, query languages, and APIs. Data Lake systems have been proposed as a solution to this problem, by providing a schema-less repository for raw data with a common access interface. However, just dumping all data into a data lake without any metadata management, would only lead to a 'data swamp'. To avoid this, we propose Constance, a Data Lake system with sophisticated metadata management over raw data extracted from heterogeneous data sources. Constance discovers, extracts, and summarizes the structural metadata from the data sources, and annotates data and metadata with semantic information to avoid ambiguities. With embedded query rewriting engines supporting structured data and semi-structured data, Constance provides users a unified interface for query processing and data exploration. During the demo, we will walk through each functional component of Constance. Constance will be applied to two real-life use cases in order to show attendees the importance and usefulness of our generic and extensible data lake system.