FDTool: a Python application to mine for functional dependencies and candidate keys in tabular data.

Preserving logical and functional dependencies in synthetic tabular data

Normalization is a database design technique, which is used to design the Relational Database table up to higher normal form. The main aim of the database normalization approach is to reduce data redundancy and maintain the atomicity within the database table. In this paper, The Tabular approach algorithm method is introduced to generating candidate key from set a valid set of functional dependency. Once, it determines the candidate key of a database table from a given valid set of functional dependency, and then applying normalization algorithms we can easily achieve higher level of normal form. It also shows new tabular based automatic generation of all possible super key and candidate key of a database table. A tabular approach algorithm method have table, having three columns marked as left, middle and right and one row. This column contains the valid set of given functional dependency. Once, we feed the column of the tabular approach table marked as left, middle and right with all possible valid set of functional dependency, then using proposed tabular approach algorithm method used to determine candidate key(s) and using these key(s), we can reach higher level of normal form.

In data analysis, a significant amount of erroneous or incomplete data can hinder informed organizational decisions prompting the need for automated data cleaning. Leveraging successful artificial intelligence techniques across various domains, several initiatives have introduced machine learning models to tackle these data-quality related issues. In this paper, we study a novel strategy to enhance data cleaning using a machine learning-based model trained on unstructured data. The study involves training a machine learning model with instances that satisfy the dataset constraints (such as functional dependencies), and examining its impact on efficiency in repairing dirty data. We also suggest improving data repair by comparing the probabilities of machine learning model predictions against a threshold, replacing the actual data with higher certainty predictions. Moreover, a hybrid data cleaning approach can enhance the effectiveness of our proposed method, resulting in higher repair quality. Experiments demonstrate promising results in terms of precision (0.88) and recall (0.94), particularly when the training data are clean from constraint violations and sufficiently large.

In this paper, we propose an efficient rule discovery algorithm, called FD_Mine, for mining functional dependencies from data. By exploiting Armstrong's Axioms for functional dependencies, we identify equivalences among attributes, which can be used to reduce both the size of the dataset and the number of functional dependencies to be checked. We first describe four effective pruning rules that reduce the size of the search space. In particular, the number of functional dependencies to be checked is reduced by skipping the search for FDs that are logically implied by already discovered FDs. Then, we present the FD_Mine algorithm, which incorporates the four pruning rules into the mining process. We prove the correctness of FD_Mine, that is, we show that the pruning does not lead to the loss of useful information. We report the results of a series of experiments. These experiments show that the proposed algorithm is effective on 15 UCI datasets and synthetic data.

One essential and challenging task in data science is data cleaning - the process of identifying and eliminating data anomalies. Different data types, data domains, data acquisition methods, and final purposes of data cleaning have resulted in different approaches in defining data anomalies in the literature. This paper proposes and describes a set of basic data anomalies in the form of anomaly patterns commonly encountered in tabular data, independently of the data domain, data acquisition technique, or the purpose of data cleaning. This set of anomalies can serve as a valuable basis for developing and enhancing software products that provide general-purpose data cleaning facilities and can provide a basis for comparing different tools aimed to support tabular data cleaning capabilities. Furthermore, this paper introduces a set of corresponding data operations suitable for addressing the identified anomaly patterns and introduces Grafterizer - a software framework that implements those data operations.

Data partitioning algorithms aiming to minimize the execution time and the energy of computations in self-adaptable data-parallel applications on modern extreme-scale multicore platforms must address two critical challenges. First, they must take into account the new complexities inherent in these platforms such as severe resource contention and non-uniform memory access. Second, they must have low practical runtime and memory costs. The sequential data partitioning algorithms addressing the first challenge have a theoretical time complexity of O( $m * m * p * p$ ) where $m$ is the number of points in the discrete speed/energy function and $p$ is the number of available processors. They, however, exhibit high practical runtime cost and excessive memory footprint, therefore, rendering them impracticable for employment in self-adaptable applications executing on extreme-scale multicore platforms. We present, in this paper, the parallel data partitioning algorithms that address both the challenges. They take as input the functional models of performance and energy consumption against problem size and output workload distributions, which are globally optimal solutions. They have a low time complexity of O( $m * m * p$ ) thereby providing a linear speedup of O( $p$ ) and low memory complexity of O( $n$ ) where $n$ is the workload size expressed as a multiple of granularity. They employ dynamic programming approach, which also facilitates the easier integration of performance and energy models of communications. We experimentally study the practical cost of application of our algorithms in two data-parallel applications, matrix multiplication and fast Fourier transform, on a cluster in Grid’5000 platform. We demonstrate that their practical runtime and memory costs are low making them ideal for employment in self-adaptable applications. We also show that the parallel algorithms exhibit tremendous speedups over the sequential algorithms. Finally, using theoretical analysis for a forecast exascale platform, we demonstrate that the parallel algorithms have negligible execution times compared to the matrix multiplication application executing on the platform.

Functional dependencies (fds) express important relationships among data, which can be used for several goals, including schema normalization and data cleansing. However, to solve several issues in emerging application domains, such as the identification of data inconsistencies or patterns of semantically related data, it has been necessary to relax the fd definition through the introduction of approximations in data comparison and/or validity. Moreover, while fds were originally specified at design time, with the availability of massive data and computational power many algorithms have been devised to automatically discover them from data, including algorithms for discovering some types of relaxed fds. In this paper we present a technique that exploits lattice-based algorithms for the discovery of fds from data, in order to detect relaxed fds. Moreover, we introduce an algorithm to determine a proper distance threshold for a given relaxed fd holding over the entire database.

We present a model to visual analysis of tabular data based on functional dependencies, and a Web based tool that supports casual users in doing the following: (a) construct an analytic query visually, in a flexible, interactive manner, (b) visualize the aggregate result in a user selected mode (histogram, pie, etc.), (c) explore the query result by viewing equivalent representations at different aggregation levels or for different parameter values.KeywordsData AnalysisAnalytic QueryData VisualizationVisual Interaction

The exponential growth of data sources over the Internet or in enterprise intranets requires the development of data integration methodologies and solutions to facilitate data access by offering uniform interface to end users. Data integration is facing two challenges: (1) management of source heterogeneity and (2) consolidation of the query results. To deal with the problem of heterogeneity, several research efforts proposed the use of ontologies to explicit semantic of sources. This explicitation of source semantic facilitates the resolution of different conflicts identified during the integration process. Once an integration system is built (using mediator architecture) it shall support user queries, by first identifying the relevant sources for a given query and then conciliating the result. To accomplish this task, two trends emerge in the current work: (1) the supposition that different entities of sources representing the same concept have the same key. This hypothesis is not always true in real applications due to the autonomy of sources. (2) The use of statistical methods to identify similar instances. For some applications like banking and engineering, precise integration solutions are needed. In this chapter, we propose an integration methodology for sources referencing shared domain ontology (called ontology-based database sources) with mediation architecture. Our ontology is enriched by functional dependencies defined in each ontology class. The presence of these functional dependencies allows the generation of the lists of candidate keys for each class. Therefore, each source can choose its keys from these lists. This gives more autonomy of sources and allows consolidation of the results in the absence of a common identifier. Our approach is validated using a set of ontology based database sources in Postgres DBMS, where all mediator components are formally described.

Many techniques for cleaning dirty data are based on enforcing some set of integrity constraints. Conditional functional dependencies (CFDs) are a combination of traditional Functional dependencies (FDs) and association rules, and are widely used as a constraint formalism for data cleaning. However, the discovery of such CFDs has received limited attention. In this paper, we regard CFDs as an extension of association rules, and present three general methodologies for (approximate) CFD discovery, each using a different way of combining pattern mining for discovering the conditions (the “C” in CFD) with FD discovery. We discuss how existing algorithms fit into these three methodologies, and introduce new techniques to improve the discovery process. We show that the right choice of methodology improves performance over the traditional CFD discovery method CTane. Code related to this paper is available at: https://github.com/j-r77/cfddiscovery, https://codeocean.com/2018/06/20/discovering-conditional-functional-dependencies/code.

Recently, data-driven decision-making has attracted great interest; this requires high-quality datasets. However, real-world datasets often feature missing values for unknown or intentional reasons, rendering data-driven decision-making inaccurate. If a machine learning model is trained using incomplete datasets with missing values, the inferred results may be biased. In this case, a commonly used technique is the missing value imputation (MVI), which fills missing data with possible values estimated based on observed values. Various data imputation methods using machine learning, statistical inference, and relational database theories have been developed. Among them, conventional machine learning based imputation methods that handle tabular data can deal with only numerical columns or are time-consuming and cumbersome because they create an individualized predictive model for each column. Therefore, we have developed a novel imputational neural network that we term the Denoising Self-Attention Network (DSAN). Our proposed DSAN can deal with tabular datasets containing both numerical and categorical columns; it considers discretized numerical values as categorical values for embedding and self-attention layers. Furthermore, the DSAN learns robust feature expression vectors by combining self-attention and denoising techniques, and can predict multiple, appropriate substituted values simultaneously (via multi-task learning). To verify the validity of the method, we performed data imputation experiments after arbitrarily generating missing values for several real-world tabular datasets. We evaluated both imputational and downstream task performances, and we have seen that the DSAN outperformed the other models, especially in terms of category variable imputation.

Triple Mixup is an augmentation policy in the hidden latent space we introduced in the Contrastive Mixup Self-Semi Supervised learning framework, to address the imbalanced data problem, for Cardiovascular Heart Diseases tabular dataset. Medical tabular datasets are known to present challenges as high imbalanced class, limited annotated quality samples due to the domain nature. Recent literature in Self and Semi supervised learning, has shown tremendous progress in learning useful representations, and leveraging unlabeled dataset and labeled dataset to train a learning model. Most existing methods are not feasible for tabular data due to the data augmentation scheme. In addition, the high imbalanced problem can show lower performance on machine learning algorithms. For this work, we propose the triple data augmentation method in hidden space to attack the unbalanced challenge in self-supervised and semi-supervised learning, from the possible applications of Contrastive Mixup, thus we will study the influence of it.

We provide an efficient algorithm for computing the candidate keys of a relational database schema. The algorithm exploits the 'arrangement' of attributes in the functional dependencies to determine which attributes are essential and useful for determining the keys and which attributes should not be considered. A more generalized algorithm using attribute graphs is then provided which allows a uniform and simplified solution to find all possible keys of a relational database schema when the attribute graph of Functional Dependencies (FDs) is not strongly connected.

There is an established equivalence between relational database Functional Dependencies (FDs) and a fragment of switching algebra that is built typically of Horn clauses. This equivalence pertains to both concepts and procedures of the FD relational database domain and the switching algebraic domain. This study is an exposition of the use of switching-algebraic tools in solving problems typically encountered in the analysis and design of relational databases. The switching-algebraic tools utilized include purely-algebraic techniques, purely-visual techniques employing the Karnaugh map and intermediary techniques employing the variable-entered Karnaugh map. The problems handled include; (a) the derivation of the closure of a Dependency Set (DS), (b) the derivation of the closure of a set of attributes, (c) the determination of all candidate keys and (d) the derivation of irredundant dependency sets equivalent to a given DS and consequently the determination of the minimal cover of such a set. A relatively large example illustrates the switching-algebraic approach and demonstrates its pedagogical and computational merits over the traditional approach.

Denial constraints (DCs) are a generalization of many other integrity constraints (ICs) widely used in databases, such as key constraints, functional dependencies, or order dependencies. Therefore, they can serve as a unified reasoning framework for all of these ICs and express business rules that cannot be expressed by the more restrictive IC types. The process of formulating DCs by hand is difficult, because it requires not only domain expertise but also database knowledge, and due to DCs' inherent complexity, this process is tedious and error-prone. Hence, an automatic DC discovery is highly desirable: we search for all valid denial constraints in a given database instance. However, due to the large search space, the problem of DC discovery is computationally expensive. We propose a new algorithm H ydra , which overcomes the quadratic runtime complexity in the number of tuples of state-of-the-art DC discovery methods. The new algorithm's experimentally determined runtime grows only linearly in the number of tuples. This results in a speedup by orders of magnitude, especially for datasets with a large number of tuples. Hydra can deliver results in a matter of seconds that to date took hours to compute.