Data Quality Problems in TPC-DI Based Data Integration Processes

Purpose: This research examines the utilization of machine learning (ML) in data warehousing systems and the extent to which it will transform business intelligence and analytics. It aims to know how ML improves conventional data warehousing systems to support prediction and forecasting. Methodology: This research uses a literature review together with a case analysis. It discusses the issues that may arise when implementing Machine Learning models with data warehouses, such as issues to do with data quality, scalability, and real-time processing. The work examines integration patterns like in-database ML computations, feature stores, and MLOps. Case studies are discussed to demonstrate the value of the use of integration in different fields. Findings: Combining machine learning with DW systems provides significant advantages in different fields. This synergy boosts analytical aptitudes, allowing the organization to go a notch higher than descriptive analytics in predictive and prescriptive analytics. However, such a decision is not simple as it has implementation matters such as data quality problems, scalability, and real-time processing problems. Integration best practices include in-database machine learning processing, a feature store, and proper MLOps practices. Real-life examples from the healthcare industry, banking and financial services, retail, and manufacturing industries show that this integration brings operational enhancements for the business and positive effects on customers and overall organizational performance. Recommendations: This work offers a useful framework for studying and constructing the integration of ML into the data warehouse, which is a transition from the theoretical perspective to the actual one. It provides practical advice for organizations and stresses the integration strategies related to the business goals, data quality, the choice of architecture, security, and training. This study also envisions future trends such as edge computing, AutoML, and Explainable AI and offers a guide on how to harness this technological complementarity. The generated insights help decision-makers and practitioners understand the possibilities of leveraging ML-data warehouse integration as a strategic asset in the contemporary business environment shifting towards data-driven approaches.

Modern information systems consist of many distributed computer and database systems. The integration of such distributed data into a single data warehouse system is confronted with the well known problem of low data quality. In this paper we present an approach that facilitates a dynamic identification of spurious and error-prone data stored in a large data warehouse. The identification of data quality problems is based on data mining techniques, such as clustering, subspace clustering and classification. Furthermore, we present via a case study the applicability of our approach on real data. The experimental results show that our approach efficiently identifies data quality problems.

The process and functioning of data integration is termed as combining information from several sources to provide users with a coherent perspective. The fundamental idea behind data integration is to open up data and make it simpler for individuals and systems to access, utilize, and process. The process of converting data from one format to another, typically from that of a source system into that required by a destination system, is known as data transformation. Data transformation is a component of the majority of data integration and management processes, including data manipulation and data warehousing. Many organizations carry out data transformation and integration because they have requirements with respect to data usage that is important in every situation. This paper proposes an architecture that reduces manual work and abstracts the decisions to be made in the integration and transformation process. This approach can lower the risk of human error and result in significant financial savings for various organizations. A modular approach is followed to ease these complex tasks and get desired results.

After years accumulation of informationization construction, power enterprises have established huge and complex application systems, which include multi-class data related to business like production, operation and management. Along with the demands of developing new business application, the data integration is urgently needed to grasp overall data view; however the complex data integration environment causes the data quality problem particularly outstanding. Under the background of the power industry, this paper first studies the data quality definition and dimension recognition; then proposes a data integration platform architecture and studies the data quality control methods in data integration; elaborates data quality assessment methods and procedures; in the view of data product, designs a total data quality management (TDQM) process finally. In a typical case of application, the results show the practicability of the proposed method.

Data interoperability is a major issue in data management for data science and big data analytics. Probabilistic data integration (PDI) is a specific kind of data integration where extraction and integration problems such as inconsistency and uncertainty are handled by means of a probabilistic data representation. This allows a data integration process with two phases: (1) a quick partial integration where data quality problems are represented as uncertainty in the resulting integrated data, and (2) using the uncertain data and continuously improving its quality as more evidence is gathered. The main contribution of this paper is an iterative approach for incorporating evidence of users in the probabilistically integrated data. Evidence can be specified as hard or soft rules (i.e., rules that are uncertain themselves).

With the web witnessing an immense shift towards publishing data, integrating data from diverse sources that have heterogeneous security levels become even more challenging. There is a need to maintain security and privacy requirements of each data source participating in a data integration process before and during the integration. In Data Integration Systems (DIS) for critical domains, data needs to be dynamically collected from multiple sources to contain the problem and make faster decisions. Therefore, it is essential to consider Security, Privacy and Trust (SPT) in such a system. However, the literature lacks this comprehensive perspective; therefore, this paper presents a novel framework of SPT requirements that assist software engineers in building a secure DIS.

To solve the problem of data quality in data warehouse, the process of data stream in data warehouse and the factors influencing data quality were analyzed; a method for controlling data quality based on quality grading was proposed and the frame of data quality grading system was established. In each process of ETL, the data quality grading system would execute script to calculate values, such as total, component and standard measures in accordance with the definition of Unit Measure, so as to get representation indexes; and then the quality report would be generated according to the indexes and the quality would be graded. The method is applicable to various data warehouse systems.

The goal of a data warehouse system is to provide a comprehensive overview of the data available in a company, thereby supporting the management decisions. The integration of data coming from heterogeneous sources is one of the key problems in data warehousing. The technical foundations for the integration have been developed in recent years. However, an efficient technical infrastructure is not sufficient to address the following problems. Firstly, the data in the systems involved have different semantics. Secondly, there are different user requirements regarding the quality of data. Existing systems are unable to solve these problems. The present thesis supports the development of data warehouse systems paying special attention to the problems regarding semantics and data quality. The approach is based on the explicit modelling of meta data of data warehouse systems. In particular, the conceptual context, the quality requirements, and the quality characteristics of the individual system components are represented in a formal model. The main contributions of the present thesis are, firstly, an extended meta model of the architecture and processes of a data warehouse system and, secondly, a quality model for the systematic representation of quality requirements and measurements. Furthermore, a classification of quality dimensions and factors is developed that can be used for an extensive quality management in data warehouse systems. The meta data is applied in a model for quality management as well as in a methodology for quality-oriented data integration. The methodology developed in this work uses the meta data by combining different existing approaches to data integration. The results of the present work are validated in various case studies in industrial contexts and in international research projects.

Nowadays, organizations cannot satisfy their information needs from one data source. Moreover, multiple data sources across the organization fuels the need for data integration. Data integration system’s users pose queries in terms of an integrated schema and expect accurate, unambiguous, and complete answers. So the data integration system is not limited to, getting the answers to the queries from the sources, but also it is extended to detect and resolve the data quality problems appeared due to the integration process. The most crucial component in any data integration system is the mappings constructed between the data sources and the integrated schema. In this paper a new mapping approach is proposed to map not only the elements of the integrated schema as done by the existing approaches, but also it maps other elements required in detecting and resolving the duplicates. It provides a means to facilitate future extensibility and changes to both the sources and the integrated schema. The proposed approach provides a linkage between the fundamental components required to provide accurate and unambiguous answers to the users’ queries from the integration system.

To meet customer demands, companies must manage numerous variants and versions of their products. Since product-related data (e.g., requirements’ specifications, geometric models, and source code, or test cases) are usually scattered over a large number of heterogeneous, autonomous information systems, their integration becomes crucial when developing complex products on one hand and aiming at reduced development costs on the other. In general, product data are created in different stages of the product development process. Furthermore, they should be integrated in a complete and consistent way at certain milestones during process development (e.g., prototype construction). Usually, this data integration process is accomplished manually, which is both costly and error prone. Instead semi-automated product data integration is required meeting the data quality requirements of the various stages during product development. In turn, this necessitates a close monitoring of the progress of the data integration process based on proper metrics. Contemporary approaches solely focus on metrics assessing schema integration, while not measuring the quality and progress of data integration. This paper elicits fundamental requirements relevant in this context. Based on them, we develop appropriate metrics for measuring product data quality and apply them in a case study we conducted at an automotive original equipment manufacturer.KeywordsProduct data integrationIntegration qualityIntegration Process

PurposeDespite the increasing role of the data warehouse as a supportive decision-making tool in today's business world, academic research for measuring its effectiveness has been lacking. This paucity of academic interest stimulated us to evaluate data warehousing effectiveness in the organizational context of Jordanian banks.Design/methodology/approachThis paper develops a theoretical model specific to the data warehouse system domain that builds on the DeLone and McLean model. The model is empirically tested by means of structural equation modelling applying the partial least squares approach and using data collected in a survey questionnaire from 127 respondents at Jordanian banks.FindingsEmpirical data analysis supported that data quality, system quality, user satisfaction, individual benefits and organizational benefits have made strong contributions to data warehousing effectiveness in our organizational data context.Practical implicationsThe results provide a better understanding of the data warehouse effectiveness and its importance in enabling the Jordanian banks to be competitive.Originality/valueThis study is indeed one of the first empirical attempts to measure data warehouse system effectiveness and the first of its kind in an emerging country such as Jordan.

The use of interactive applications to support the decision-making process is more common every day. However, a huge amount of data is required in order to make more informed decisions. Fortunately, with the arrival of new technologies there are many data sources available. This requirement of data causes heterogeneity and data quality problems. A set of data quality problems are reduced in the preprocessing stage. However, many data quality issues persist after the preprocessing stage. For this reason, we proposed a methodology to take the data quality problems, to represent them and simultaneously support the analysis process. In addition, an application is developed as a use case of the methodology by analyzing the public transport system in Bogotá. Furthermore, a case study is performed to test the usefulness of the developed application. As a result, the methodology made possible the development of interactive visualizations that constitute an application that is useful to achieve the analysis tasks by including data quality features.

Competitive Intelligence (CI) is the process of managing information emanating from the business environment of an organization in order to support decision making process. CI enables the development of strategies that confer companies a significant competitive advantage. The appropriate decisions are taken when all required data are considered. As the amount of data grows very fast inside and outside of an enterprise, exploiting these mountains of data efficiently became a crucial need. Note that such data are usually multiples, heterogeneous, autonomous and distributed. The main issue is related to the identification and resolution of structural and semantic heterogeneity between data, usually spread in multiple sources. Data integration is a significant solution that addresses these problems. Data Warehouse (DW) is viewed as Data Integration System (DIS) that consolidates several data sources in the same target repository through an Extract-Transform-Load (ETL) process. In a decisional context, DW is a relevant solution to aggregate a huge amount of data and organizing them in order to facilitate their analysis and support decision making. In this paper, we propose an approach for designing Semantic Data Warehouse (SDW) to support CI process. Our proposal takes as inputs decision makers' requirements and a set of Semantic Databases (SDB) sources for building the SDW.

This paper explores the integration of healthcare data into commercial Healthcare Effectiveness Data and Information Set (HEDIS) systems, emphasizing its critical role in improving healthcare quality, transparency, and accountability. HEDIS, maintained by National Committee for Quality Assurance (NCQA), offers a standardized framework to measure health plan performance across various domains such as preventive care, chronic disease management, and mental health. The paper reviews the structure and importance of HEDIS, evaluates commercial systems like Cozeva, Epic, TriZetto, and Waystar, and compares their capabilities in quality reporting, analytics, and interoperability. It delves into data integration challenges—including standardization, interoperability, data quality, and security—and presents technical solutions such as data warehousing, API use, ETL pipelines, and master data management. The study highlights the necessity of strong data governance, secure data exchange, and adherence to regulatory compliance like HIPAA. Additionally, it explores the transformative potential of emerging technologies like AI, machine learning, and predictive analytics in enhancing HEDIS reporting accuracy and efficiency. The paper concludes by presenting a practical roadmap for successful data integration into HEDIS systems, advocating for strategic planning, robust infrastructure, and innovative tools to enable data-driven decision-making and improved healthcare outcomes.

Data quality is a critical issue both in operational databases and in data warehouse systems. Data quality assessment is a strong requirement regarding the ETL subsystem, since bad data may destroy data warehouse credibility. During the last two decades, research and development efforts in the data quality field have produced techniques for data profiling and cleaning, which focus on detecting and correcting bad values in data. Little efforts have been done considering data quality when it relates to the well-formedness of coarse grained data structures resulting from the assembly of linked data records. This paper proposes a metadata model that supports the structural validation of linked data records, from a data quality point of view. The metamodel is built on top of the CWM standard and it supports the specification of data structure quality rules in a high level of abstraction, as well as by means of very specific fine grained business rules.KeywordsData QualityMetadataMetamodelCWM