Conditional functional dependencies validation for XML data: an approach based on attribute grammar

Many techniques for cleaning dirty data are based on enforcing some set of integrity constraints. Conditional functional dependencies (CFDs) are a combination of traditional Functional dependencies (FDs) and association rules, and are widely used as a constraint formalism for data cleaning. However, the discovery of such CFDs has received limited attention. In this paper, we regard CFDs as an extension of association rules, and present three general methodologies for (approximate) CFD discovery, each using a different way of combining pattern mining for discovering the conditions (the “C” in CFD) with FD discovery. We discuss how existing algorithms fit into these three methodologies, and introduce new techniques to improve the discovery process. We show that the right choice of methodology improves performance over the traditional CFD discovery method CTane. Code related to this paper is available at: https://github.com/j-r77/cfddiscovery, https://codeocean.com/2018/06/20/discovering-conditional-functional-dependencies/code.

Many issues about data quality have been studied in relational data, such as data consistency, data deduplication, data accuracy, data completeness and so on. In this paper, we focus on the discovery of abnormal data in RDF graphs. As the amount of RDF data is increasing, data quality is becoming an important issue for usability of these RDF repositories. Although association rules have been used to find abnormals in RDF graph, existing solutions ignore the latent semantics of connected structures in RDF graphs. In order to detect latent dependencies in RDF graph, firstly, we innovatively define Graph-based Conditional Functional Dependency(GCFD) that can represent the attribute value and semantic structure dependencies of RDF data in a uniform manner. Then, we propose an efficient framework and some novel pruning rules to discover GCFD in large RDF graphs. Extensive experiments on several real-life RDF repositories confirm the superiority of our solution.

Conditional functional dependencies (CFDs) have recently been proposed as a useful integrity constraint to summarize data semantics and identify data inconsistencies. A CFD augments a functional dependency (FD) with a pattern tableau that defines the context (i.e., the subset of tuples) in which the underlying FD holds. While many aspects of CFDs have been studied, including static analysis and detecting and repairing violations, there has not been prior work on generating pattern tableaux, which is critical to realize the full potential of CFDs. This paper is the first to formally characterize a "good" pattern tableau, based on naturally desirable properties of support, confidence and parsimony. We show that the problem of generating an optimal tableau for a given FD is NP-complete but can be approximated in polynomial time via a greedy algorithm. For large data sets, we propose an "on-demand" algorithm providing the same approximation bound, that outperforms the basic greedy algorithm in running time by an order of magnitude. For ordered attributes, we propose the range tableau as a generalization of a pattern tableau, which can achieve even more parsimony. The effectiveness and efficiency of our techniques are experimentally demonstrated on real data.

Several anonymization techniques, such as generalization and bucketization, have been designed for privacy preserving microdata publishing. Recently it has become major importance in research areas and numerous approaches have been proposed to preserve privacy in literature. In order to solve this issue, recent work study the problem of privacy preservation hazard for mined Conditional Functional Dependency (CFD) against (d,l) inference model using Compact frequent pattern growth branch sort algorithm (CFPGBS) methods. Boyce–Codd normal Form (BCNF) method is presented for publishing multiple subtables and it is anonymized through (d, l)- inference model. But the major problem of the work is that initial partitioning is done without considering the importance of the feature, so it contains all quasi identifier features. It becomes more complex to perform initial partitioning if the data samples becomes large,thus may also reduce the privacy results. To overcome this issue in this paper presents a hybrid feature selection method which removes unimportant quasi identifier in the final Boyce–Codd normal Form (BCNF) table. The proposed hybrid feature selection method follows the procedure of hybrid firefly algorithm with harmony search (HFAHS) algorithm to select optimal feature in the table for initial partition of top down approach. For selected features expectation Maximization (EM) based Gaussian mixture distribution (EM-GMM) based top down initial partition is done to group the partition data from the result d,l -inference model along with changed publishing data from BCNF. Experimental results show that the proposed BCNF- d,l -inference model can adeptly anonymize the microdata with less information loss when compared to existing methods against CFD. The effectiveness and privacy results of proposed methods are also compared with existing conventional methods. Our workload experiments confirm that proposed preserves better utility, privacy than d,l -inference model against CFD attack and is more effective in workloads with selection of optimal quasi identifiers.

A variety of data dependencies have been proposed for data cleaning, including conditional functional dependencies, editing rules and so on. Fixing rule is a newly proposed class of data dependencies for data repairing with high accuracy. However, to our knowledge, algorithms for automatically designing fixing rules have not been developed. In this paper, a workflow of generating fixing rules has been proposed and an efficient algorithm for automatically producing fixing rules from constant CFDs has been developed. The algorithm has been proved to have the ability to avoid generating conflict FRs, and experiments show that the algorithm scales well with the size of input and can output a consistent set of FRs with fairly good usability.

Data cleaning is an important problem and data quality rules are the most promising way to face it with a declarative approach. Previous work has focused on specific formalisms, such as functional dependencies (FDs), conditional functional dependencies (CFDs), and matching dependencies (MDs), and those have always been studied in isolation. Moreover, such techniques are usually applied in a pipeline or interleaved. In this work we tackle the problem in a novel, unified framework. First, we let users specify quality rules using denial constraints with ad-hoc predicates. This language subsumes existing formalisms and can express rules involving numerical values, with predicates such as “greater than” and “less than”. More importantly, we exploit the interaction of the heterogeneous constraints by encoding them in a conflict hypergraph. Such holistic view of the conflicts is the starting point for a novel definition of repair context which allows us to compute automatically repairs of better quality w.r.t. previous approaches in the literature. Experimental results on real datasets show that the holistic approach outperforms previous algorithms in terms of quality and efficiency of the repair.

This paper proposes a method on solving the problem of dealing with dirty data in the database. Considering the complexity of the structure of the data, based on the previous methods that work on this problem, our method combines the methods that use regular expression and methods that use conditional functional dependencies, to complete the data quality improvement. This method uses dependencies to improve the repairing speed and the searching time on the data. The repairing based on the regular expression is regular while there exist questions that the repairing efficient is influenced by the amount of data. When dealing with the database from company Standard Solution Group (SSG) which is from the reality world data, we have tried other related methods and inspired by these methods, we propose this method. The experiments on the data from SSG shows that this method is much efficient.

Smart online university party building system based on association rules data mining is designed and shown in the paper. According to the need of data patching, the association rules related to the target attribute will be searched by reasonably reducing the value of support, that is, find the corresponding conditional functional dependencies. For enhancing the data robustness, the neural model is applied into the training stage of the model. Furthermore, the system is implemented through the platform design. The user interface and the experience are jointly improved through the test.

Certain second-order sentences, called conditional implicational dependencies, about relations in a relational database, are denned and studied. These dependencies include some of previously defined dependencies as special cases. Thus, the family of implicational dependencies introduced and studied by R. Fagip can be included in the family of conditional implicational dependencies. The conditional-functional dependencies defined by P. De Bra and J. Paredaens are special cases of new dependencies. A characterization of a conditional implicational dependency implied by a given class in this family, is given. The existence of Armstrong models for the class of conditional implicational dependencies with a single relational constant symbol is shown.

Reconstruction of gene regulatory networks (GRNs), which explicitly represent the causality of developmental or regulatory process, is of utmost interest and has become a challenging computational problem for understanding the complex regulatory mechanisms in cellular systems. However, all existing methods of inferring GRNs from gene expression profiles have their strengths and weaknesses. In particular, many properties of GRNs, such as topology sparseness and non-linear dependence, are generally in regulation mechanism but seldom are taken into account simultaneously in one computational method. In this work, we present a novel method for inferring GRNs from gene expression data considering the non-linear dependence and topological structure of GRNs by employing path consistency algorithm (PCA) based on conditional mutual information (CMI). In this algorithm, the conditional dependence between a pair of genes is represented by the CMI between them. With the general hypothesis of Gaussian distribution underlying gene expression data, CMI between a pair of genes is computed by a concise formula involving the covariance matrices of the related gene expression profiles. The method is validated on the benchmark GRNs from the DREAM challenge and the widely used SOS DNA repair network in Escherichia coli. The cross-validation results confirmed the effectiveness of our method (PCA-CMI), which outperforms significantly other previous methods. Besides its high accuracy, our method is able to distinguish direct (or causal) interactions from indirect associations. All the source data and code are available at: http://csb.shu.edu.cn/subweb/grn.htm. lnchen@sibs.ac.cn; zpliu@sibs.ac.cn Supplementary data are available at Bioinformatics online.

Theme: Data and Linkage QualityObjectives:
 
 To define health data quality from clinical, data science, and health system perspectives
 To describe some of the international best practices related to quality and how they are being applied to Canada’s administrative health data.
 To compare methods for health data quality assessment and improvement in Canada (automated logical checks, chart quality indicators, reabstraction studies, coding manager perspectives)
 To highlight how data linkage can be used to provide new insights into the quality of original data sources
 To highlight current international initiatives for improving coded data quality including results from current ICD-11 field trials
 
 Dr. Keith Denny: Director of Clinical Data Standards and Quality, Canadian Insititute for Health Information (CIHI), Adjunct Research Professor, Carleton University, Ottawa, ON. He provides leadership for CIHI’s information quality initiatives and for the development and application of clinical classifications and terminology standards.
 Maureen Kelly: Manager of Information Quality at CIHI, Ottawa, ON. She leads CIHI’s corporate quality program that is focused on enhancing the quality of CIHI’s data sources and information products and to fostering CIHI’s quality culture.
 Dr. Cathy Eastwood: Scientific Manager, Associate Director of Alberta SPOR Methods & Development Platform, Community Health Sciences, Cumming School of Medicine, University of Calgary, Calgary, AB. She has expertise in clinical data collection, evaluation of local and systemic data quality issues, disease classification coding with ICD-10 and ICD-11.
 Dr. Hude Quan: Professor, Community Health Sciences, Cumming School of Medicine, University of Calgary, Director Alberta SPOR Methods Platform; Co-Chair of Hypertension Canada, Co-Chair of Person to Population Health Collaborative of the Libin Cardiovascular Institute in Calgary, AB. He has expertise in assessing, validating, and linking administrative data sources for conducting data science research including artificial intelligence methods for evaluating and improving data quality.
 Intended Outcomes:“What is quality health data?” The panel of experts will address this common question by discussing how to define high quality health data, and measures being taken to ensure that they are available in Canada. Optimizing the quality of clinical-administrative data, and their use-value, first requires an understanding of the processes used to create the data.
 Subsequently, we can address the limitations in data collection and use these data for diverse applications. Current advances in digital data collection are providing more solutions to improve health data quality at lower cost.
 This panel will describe a number of quality assessment and improvement initiatives aimed at ensuring that health data are fit for a range of secondary uses including data linkage. It will also discuss how the need for the linkage and integration of data sources can influence the views of the data source’s fitness for use.
 CIHI content will include:
 
 Methods for optimizing the value of clinical-administrative data
 CIHI Information Quality Framework
 Reabstraction studies (e.g. physician documentation/coders’ experiences)
 Linkage analytics for data quality
 
 University of Calgary content will include:
 
 Defining/measuring health data quality
 Automated methods for quality assessment and improvement
 ICD-11 features and coding practices
 Electronic health record initiatives

This paper proposes to deduce certain fixes to graphs G based on data quality rules Σ and ground truth Γ ( i.e., validated attribute values and entity matches). We fix errors detected by Σ in G such that the fixes are assured correct as long as Σand Γ are correct. We deduce certain fixes in two paradigms. (a) We interact with users and "incrementally" fix errors online. Whenever users pick a small set V 0 of nodes in G , we fix all errors pertaining to V 0 and accumulate ground truth in the process. (b) Based on accumulated Γ, we repair the entire graph G offline; while this may not correct all errors in G , all fixes are guaranteed certain. We develop techniques for deducing certain fixes. (1) We define data quality rules to support conditional functional dependencies, recursively defined keys and negative rules on graphs, such that we can deduce fixes by combining data repairing and object identification. (2) We show that deducing certain fixes is Church-Rosser, i.e., the deduction converges at the same fixes regardless of the order of rules applied. (3) We establish the complexity of three fundamental problems associated with certain fixes. (4) We provide (parallel) algorithms for deducing certain fixes online and offline, and guarantee to reduce running time when given more processors. Using real-life and synthetic data, we experimentally verify the effectiveness and scalability of our methods.

SummaryAccurate prediction of the solubility of chemical substances in solvents remains a challenge. The sparsity of high-quality solubility data is recognized as the biggest hurdle in the development of robust data-driven methods for practical use. Nonetheless, the effects of the quality and quantity of data on aqueous solubility predictions have not yet been scrutinized. In this study, the roles of the size and the quality of data sets on the performances of the solubility prediction models are unraveled, and the concepts of actual and observed performances are introduced. In an effort to curtail the gap between actual and observed performances, a quality-oriented data selection method, which evaluates the quality of data and extracts the most accurate part of it through statistical validation, is designed. Applying this method on the largest publicly available solubility database and using a consensus machine learning approach, a top-performing solubility prediction model is achieved.

Smart Distribution Grid and Utilization are the important support for the development of Smart City and Smart Grid, meanwhile, the effective data communication and information exchange are the key points to implement them. First, analyze the unified communication method of Smart Distribution Grid and Utilization for each specific business. Then, respectively, from the current reality and future trends, propose two typical implementation:the interface adapter pattern and the integrated pattern, and establish specific implementation patterns, deployment patterns, and pattern evaluations, so as to provide the design of pattern level for the development of Smart Distribution Grid and Utilization.

Towards a content agnostic computable knowledge repository for data quality assessment