Range Query Estimation for Dirty Data Management System

Introduction In general, the suitability of spatial (GIS) environmental data sources for analytical use in ecological modeling (Hunsaker et al ., 1993) has not been given as much attention as other aspect of the SDM problem, such as the species data (Chapter 4) and modeling methods (Chapters 6–8). A few studies have explicitly examined issues of spatial resolution and data quality of predictors on SDMs (Aspinall & Pearson, 1996), have shown that including satellite-derived climate variables with land cover as predictors improves SDMs (Suarez-Seoane et al ., 2004), and have illustrated the impact of predictors derived from digital elevation models on SDMs via error propagation (Van Niel & Austin, 2007). Daly (2006) provided guidelines for assessing the quality of interpolated climate maps used in environmental modeling. However, these efforts, while important, do not give comprehensive guidelines for selecting data to represent environmental gradients related to species distributions. Some software systems developed for SDM include global environmental datasets at 0.01- (mostly 0.5-) 1.0 degree resolution (Hijmans et al ., 2001; Stockwell, 2006), but this is the exception rather than the rule. Usually it is the job of the modeler to select and assemble appropriate predictor data, and consider issues of data quality and resolution; the effort involved is non-trivial. Chapter 3 outlined conceptual models of factors driving species distributions, and in this chapter, I will describe the types of environmental data typically used in SDM to represent those factors, including climate, topography, substrate, land cover and vegetation, disturbance maps, remote sensing-based land surface characterizations, measures of landscape pattern, and information about other species (biotic interactions).

BackgroundData quality assessment is important but complex and task dependent. Identifying suitable measurement methods and reference ranges for assessing their results is challenging. Manually inspecting the measurement results and current data driven approaches for learning which results indicate data quality issues have considerable limitations, e.g. to identify task dependent thresholds for measurement results that indicate data quality issues.ObjectivesTo explore the applicability and potential benefits of a data driven approach to learn task dependent knowledge about suitable measurement methods and assessment of their results. Such knowledge could be useful for others to determine whether a local data stock is suitable for a given task.MethodsWe started by creating artificial data with previously defined data quality issues and applied a set of generic measurement methods on this data (e.g. a method to count the number of values in a certain variable or the mean value of the values). We trained decision trees on exported measurement methods’ results and corresponding outcome data (data that indicated the data’s suitability for a use case). For evaluation, we derived rules for potential measurement methods and reference values from the decision trees and compared these regarding their coverage of the true data quality issues artificially created in the dataset. Three researchers independently derived these rules. One with knowledge about present data quality issues and two without.ResultsOur self-trained decision trees were able to indicate rules for 12 of 19 previously defined data quality issues. Learned knowledge about measurement methods and their assessment was complementary to manual interpretation of measurement methods’ results.ConclusionsOur data driven approach derives sensible knowledge for task dependent data quality assessment and complements other current approaches. Based on labeled measurement methods’ results as training data, our approach successfully suggested applicable rules for checking data quality characteristics that determine whether a dataset is suitable for a given task.

Smart manufacturing applications increasingly rely on time-series data from Industrial IoT sensors, yet these data streams often contain data quality issues (DQIs) that affect analysis and disrupt production. While traditional Machine Learning methods are difficult to apply due to the small amount of data available, the knowledge-based approach of Case-Based Reasoning (CBR) offers a way to reuse previously gained experience. We introduce the first end-to-end Case-Based Reasoning (CBR) framework that both detects and remedies DQIs in near real time, even when only a handful of annotated fault instances are available. Our solution encodes expert experience in the four CBR knowledge containers: (i) a vocabulary that represents sensor streams and their context in the DataStream format; (ii) a case base populated with fault-annotated event logs; (iii) tailored similarity measures—including a weighted Dynamic Time Warping variant and structure-aware list mapping—that isolate the signatures of missing-value, missing-sensor, and time-shift errors; and (iv) lightweight adaptation rules that recommend concrete repair actions or, where appropriate, invoke automated imputation and alignment routines. A case study is used to examine and present the suitability of the approach for a specific application domain. Although the case study demonstrates only limited capabilities in identifying Data Quality Issues (DQIs), we aim to support transparent evaluation and future research by publishing (1) a prototype of the Case-Based Reasoning (CBR) system and (2) a publicly accessible, meticulously annotated sensor-log benchmark. Together, these resources provide a reproducible baseline and a modular foundation for advancing similarity metrics, expanding the DQI taxonomy, and enabling knowledge-intensive reasoning in IoT data quality management.

The issue of data quality is as old as data itself. However, the proliferation of diverse, large-scale and often publically available data on the Web has increased the risk of poor data quality and misleading data interpretations. On the other hand, data is now exposed at a much more strategic level e.g. through business intelligence systems, increasing manifold the stakes involved for individuals, corporations as well as government agencies. There, the lack of knowledge about data accuracy, currency or completeness can have erroneous and even catastrophic results. With these changes, traditional approaches to data management in general, and data quality control specifically, are challenged. There is an evident need to incorporate data quality considerations into the whole data cycle, encompassing managerial/governance as well as technical aspects. Data quality experts from research and industry agree that a unified framework for data quality management should bring together organizational, architectural and computational approaches. Accordingly, Sadiq structured this handbook in four parts: Part I is on organizational solutions, i.e. the development of data quality objectives for the organization, and the development of strategies to establish roles, processes, policies, and standards required to manage and ensure data quality. Part II, on architectural solutions, covers the technology landscape required to deploy developed data quality management processes, standards and policies. Part III, on computational solutions, presents effective and efficient tools and techniques related to record linkage, lineage and provenance, data uncertainty, and advanced integrity constraints. Finally, Part IV is devoted to case studies of successful data quality initiatives that highlight the various aspects of data quality in action. The individual chapters present both an overview of the respective topic in terms of historical research and/or practice and state of the art, as well as specific techniques, methodologies and frameworks developed by the individual contributors. Researchers and students of computer science, information systems, or business management as well as data professionals and practitioners will benefit most from this handbook by not only focusing on the various sections relevant to their research area or particular practical work, but by also studying chapters that they may initially consider not to be directly relevant to them, as there they will learn about new perspectives and approaches.

A multidatabase system (MDBS) is a database system which integrates pre-existing databases, called component local database systems (LDBSs), to support global applications accessing data at more than one LDBS. An important research issue in MDBS is query optimization. The query optimization problem in MDBS is quite different from the case of distributed database system (DDBS) since, due to schema heterogeneity and local autonomy of component LDBSs, is not possible to assume that the query optimizer has a complete information on the execution cost and database statistics. In this paper we present a distributed query optimization algorithm that works under very general assumptions for MDBSs with relational global data model. The algorithm is based on the idea of delegating the evaluation of the execution cost of the elementary steps in a query execution plan to the LDBS where the computation is performed. The optimization process is organized as a sequence of steps, in which at each step all LDBSs work in parallel to evaluate the cost of execution plans for partial queries of increasing size, and send their cost estimates to the other LDBS that need them for the next step. The computation is totally distributed, and organized in order to perform no duplicate computation, and to discard as soon as possible the execution plans that may not lead to an optimal solution.KeywordsExecution ModelExecution PlanQuery OptimizationQuery ExecutionExecution CostThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Numerical data quality in IS research and the implications for replication

Harmonizing National Topographic Data Using an Automated Quality Validation Process

As the integration of information systems enables greater accessibility to multiple data sources, the issue of data quality becomes increasingly important. In general, data quality is determined by several factors, or quality parameters. which are often not independent of each other. As a consequence, it is often necessary to represent, and reason with, domain‐specific knowledge about the relationships among quality parameters, if insightful judgments about the overall quality of data are to be made. This article presents a formulation of the data‐quality judgment problem that is amenable to a “knowledge‐based”; approach, where a data consumer can input such domain‐specific knowledge and then rely on an automated system to deduce information about overall data quality. A primary feature of this work is the notion of a data quality calculus. The data quality calculus is a data quality judgment framework based on a “census of needs,”; and provides a framework for deriving an overall data quality value from ...

Data quality issues have been topical for many decades. However, a unified data quality theory has not been proposed yet, since many concepts associated with the term “data quality” are not straightforward enough. The paper proposes a user-oriented data quality theory based on clearly defined concepts. The concepts are defined by using three groups of domain-specific languages (DSLs): (1) the first group uses the concept of a data object to describe the data to be analysed, (2) the second group describes the data quality requirements, and (3) the third group describes the process of data quality evaluation. The proposed idea proved to be simple enough, but at the same time very effective in identifying data defects, despite the different structures of data sets and the complexity of data. Approbation of the approach demonstrated several advantages: (a) a graphical data quality model allows defining of data quality even by non-IT and non-data quality professionals, (b) data quality model is not related to the information system that has accumulated data, i.e., this approach lets users analyse the “third-party” data, and (c) data quality can be described at least at two levels of abstraction - informally, using natural language, or formally, including executable program routines or SQL statements.

The Internet of Things (IoT) and Cyber-Physical Systems (CPS) are the backbones of Industry 4.0, where data quality is crucial for decision support. Data quality in these systems can deteriorate due to sensor failures or uncertain operating environments. Our objective is to summarize and assess the research efforts that address data quality in data-centric CPS/IoT industrial applications. We systematically review the state-of-the-art data quality techniques for CPS and IoT in Industry 4.0 through a systematic literature review (SLR) study. We pose three research questions, define selection and exclusion criteria for primary studies, and extract and synthesize data from these studies to answer our research questions. Our most significant results are (i) the list of data quality issues, their sources, and application domains, (ii) the best practices and metrics for managing data quality, (iii) the software engineering solutions employed to manage data quality, and (iv) the state of the data quality techniques (data repair, cleaning, and monitoring) in the application domains. The results of our SLR can help researchers obtain an overview of existing data quality issues, techniques, metrics, and best practices. We suggest research directions that require attention from the research community for follow-up work.

In the health care environment, a huge volume of data is produced on a daily basis. However, the processes of collecting, storing, sharing, analyzing, and reporting health data usually face with numerous challenges that lead to producing incomplete, inaccurate, and untimely data. As a result, data quality issues have received more attention than before. The purpose of this article is to provide an insight into the data quality definitions, dimensions, and assessment methodologies. In this article, a scoping literature review approach was used to describe and summarize the main concepts related to data quality and data quality assessment methodologies. Search terms were selected to find the relevant articles published between January 1, 2012 and September 31, 2022. The retrieved articles were then reviewed and the results were reported narratively. In total, 23 papers were included in the study. According to the results, data quality dimensions were various and different methodologies were used to assess them. Most studies used quantitative methods to measure data quality dimensions either in paper-based or computer-based medical records. Only two studies investigated respondents' opinions about data quality. In health care, high-quality data not only are important for patient care, but also are vital for improving quality of health care services and better decision making. Therefore, using technical and nontechnical solutions as well as constant assessment and supervision is suggested to improve data quality.

Data quality issues in clinical trials can be caused by a variety of behaviors including fraud, misconduct, intentional or unintentional noncompliance, and significant carelessness. Regardless of how these behaviors are defined, they may compromise the validity of the study results. Reliable study results and quality data are needed to evaluate products for marketing approval and for decisions that are made on the use of medicine. This article focuses on detecting data quality issues, irrespective of origin or motive. Early detection of data quality issues are important so that corrective actions taken can be implemented during the conduct of the trial, recurrence can be prevented, and data quality can be preserved. A survey was distributed to TransCelerate member companies to assess current strategies for detecting and mitigating risks involving fraud and misconduct in clinical trials. A review of literature across many industries from 1985 to 2014 was conducted using multiple platforms. Eighteen TransCelerate member companies anonymously responded to the survey. All of the respondents had one or more existing strategies for fraud and misconduct detection. The literature search identified current practices and methodologies across many industries. TransCelerate recommends the creation of an integrated, multifaceted approach to proactively detect data quality issues. Detection methods should include a strategy tailored to the characteristics of the study. Some sponsors are taking advantage of more advanced methods and integrated processes and systems to proactively detect and address issues, relying on advances in technology to more efficiently review data in real time. Further research is underway to assess statistical data quality detection methodology in clinical trials.

Data quality plays an essential role in decision-making, as the latter may incorporate some risks in different application areas. In the context of industry 4.0, the amount, the versatility, and the speed of information flow for decision-making are important issues. The quality and, in particular, the dependability of data is paramount. This paper investigates the leading data quality characteristics in the industry 4.0 environment with the related issues due to various interactions. It proposes a taxonomy of data sources and flows, from acquisition to the information extraction level for decision-making. The authors highlight the specific issues of error and uncertainty propagation management as significant research challenges for designing intelligent data collection and acquisition systems for industrial manufacturing decision support within the framework of the new generation of industry development. They review data quality characteristics definition and assessment requirements, and suggest classifying these characteristics into four categories. Data quality assessment methods and models with relation to decision-making are also examined. They willfully left aside the investigation of data quality improvement processes for a future more detailed paper.

Despite the proliferation of Web services, the field of service research is still in its infancy, where there has been little foundational work to date. In this chapter, we present the foundational service framework that helps address some fundamental research issues on Web services [44, 74]. The presented framework offers a systematic approach that enables users to achieve an op- timal Service Execution Plan (SEP) by submitting declarative service queries. Figure 3.1 shows the overall architecture of the service query optimization framework. At the bottom, we have the service space for a given application domain that consists of the actual service providers, a.k.a., service instances. The integration layer helps resolve the discrepancy of the service schemas and provides a uniformed view of the service space. It can achieve this by ap- plying the existing data integration techniques [40], which are not the focus of this work. The QoWS manager deals with the collection of quality related information from the service instances. The quality data will be used by the query optimizer to make selection from the competing service providers. The service model captures a set of key features of Web services that lay out a foundation for service query specification, processing, and optimization. The service calculus enables users to use declarative service queries to locate Web services, which is more precise and reliable than the keyword based search. The query processor generates SEPs which can be used by the users to invoke services.

Data quality issues can cause false decisions of clinical decision support systems (CDSSs). Analyzing local data quality has the potential to prevent data quality-related failure of CDSS adoption. To define a shareable set of applicable measurement methods (MMs) for a targeted data quality assessment determining the suitability of local data for our CDSS. We derived task-specific MMs using four approaches: (1) a GUI-based data quality analysis using the open source tool openCQA. (2) Analyzing cases of known false CDSS decisions. (3) Data-driven learning on MM-results. (4) A systematic check to find blind spots in our set of MMs based on the HIDQF data quality framework. We expressed the derived data quality-related knowledge about the CDSS using the 5-tuple-formalization for MMs. We identified some task-specific dataset characteristics that a targeted data quality assessment for our use case should inspect. Altogether, we defined 394 MMs organized in 13 data quality knowledge bases. We have created a set of shareable, applicable MMs that can support targeted data quality assessment for CDSS-based systemic inflammatory response syndrome (SIRS) detection in critically ill, pediatric patients. With the demonstrated approaches for deriving and expressing task-specific MMs, we intend to help promoting targeted data quality assessment as a commonly recognized usual part of research on data-consuming application systems in health care.