Reasoning about attribute value equivalence in relational data

Employing string similarity metrics of partners to estimate audit team continuity: determinant and its effects on audit outcomes and pricing

Traditional approaches for data anonymization consider relational data and textual data independently. We propose rx-anon, an anonymization approach for heterogeneous semi-structured documents composed of relational and textual attributes. We map sensitive terms extracted from the text to the structured data. This allows us to use concepts like k-anonymity to generate a joint, privacy-preserved version of the heterogeneous data input. We introduce the concept of redundant sensitive information to consistently anonymize the heterogeneous data. To control the influence of anonymization over unstructured textual data versus structured data attributes, we introduce a modified, parameterized Mondrian algorithm. We evaluate our approach with two real-world datasets using a Normalized Certainty Penalty score, adapted to the problem of jointly anonymizing relational and textual data. The results show that our approach is capable of reducing information loss by using the tuning parameter to control the Mondrian partitioning while guaranteeing k-anonymity. As rx-anon is a framework approach, it can be reused and extended by other anonymization algorithms, privacy models, and textual similarity metrics.

The importance of data accuracy and quality has increased with the explosion of data size. This factor is crucial to ensure the success of any cross-enterprise integration applications, business intelligence or data mining solutions. Detecting duplicate data that represent the same real world object more than once in a certain dataset is the first step to ensure the data accuracy. This operation becomes more complicated when the same object name (person, city) is represented in multiple natural languages due to several factors including spelling, typographical and pronunciation variation, dialects and special vowel and consonant distinction and other linguistic characteristics. Therefore, it is difficult to decide whether or not two syntactic values (names) are alternative designation of the same semantic entity. Up to authors’ knowledge, the previously proposed duplicate record detection (DRD) algorithms and frameworks support only single language duplicate record detection, or at most bilingual. In this paper, two available tools of duplicate record detection are compared. Then, a generic cross language based duplicate record detection solution architecture is proposed, designed and implemented to support the wide range variations of several languages. The proposed system design uses a dictionary based on phonetic algorithms and support different indexing/blocking techniques to allow fast processing. The framework proposes the use of several proximity matching algorithms, performance evaluation metrics and classifiers to suit the diversity in several languages names matching. The framework is implemented and verified empirically in several case studies. Several Experiments are executed to compare the advantages and disadvantages of the proposed system compared to other tool. Results showed that the proposed system has substantial improvements compared to the well-known tools.KeywordsDuplicate Record DetectionCross Language Systemsentity matchingdata cleaningBig Data

The problem of matching product titles is of particular interest for both users and marketers. The former, frequently search the Web with the aim of comparing prices and characteristics, or obtaining and aggregating information provided by other users. The latter, often require wide knowledge of competitive policies, prices and features to organize a promotional campaign about a group of products. To address this interesting problem, recent studies have attempted to enrich the product titles by exploiting Web search engines. More specifically, these methods suggest that for each product title a query should be submitted. After the results have been collected, the most important words which appear in the results are identified and appended in the titles. In the sequel, each word is assigned an importance score and finally, a similarity measure is applied to identify if two or more titles refer to the same product. Nonetheless, these methods have multiple problems including scalability, slow retrieval of the required additional search results, and lack of flexibility. In this paper, we present a different approach which addresses all these issues and is based on the morphological analysis of the titles of the products. In particular, our method operates in two phases. In the first phase, we compute the combinations of the words of the titles and we record several statistics such as word proximity and frequency values. In the second phase, we use this information to assign a score to each combination. The highest scoring combination is then declared as label of the cluster which contains each product. The experimental evaluation of the algorithm, in a real world dataset, demonstrated that compared to three popular string similarity metrics, our approach achieves up to 36% better matching performance and at least 13 times faster execution.

Real-world data are often stored as relational database systems with different numbers of significant attributes. Unfortunately, most classification techniques are proposed for learning from balanced non-relational data and mainly for classifying one single attribute. In this paper, we propose an approach for learning from relational data with the specific goal of classifying multiple imbalanced attributes. In our approach, we extend a relational modelling technique (PRMs-IM) designed for imbalanced relational learning to deal with multiple imbalanced attributes classification. We address the problem of classifying multiple imbalanced attributes by enriching the PRMs-IM with the Bagging classification ensemble. We evaluate our approach on real-world imbalanced student relational data and demonstrate its effectiveness in predicting student performance.

Property Graph databases (PGs) are emerging as efficient graph stores with flexible schemata. This raises the need to have a unified view over heterogenous data produced from these stores. Ontology based Data Access (OBDA) has become the most dominant approach to integrate heterogeneous data sources by providing a unified conceptual view (ontology) over them. The corner stone of any OBDA system is to define mappings between the data source and the target (domain) ontology. However, manual mapping generation is time consuming and requires great efforts. This paper proposes ProGOMap ( Pro perty G raph to O ntology M apper) system that automatically generates mappings from property graphs to a domain ontology. ProGOMap starts by generating a putative ontology with direct axioms from PG. A novel ontology learning algorithm is proposed to enrich the putative ontology with subclass axioms inferred from PG. The putative ontology is then aligned to an existing domain ontology using string similarity metrics. Another algorithm is proposed to align object properties between the two ontologies considering different modelling criteria. Finally, mappings are generated from alignment results. Experiments were done on eight data sets with different scenarios to evaluate the effectiveness of the generated mappings. The experimental results achieved mapping accuracy up to 97% and 81% when addressing PG-to-ontology terminological and structural heterogeneities, respectively. Ontology learning by inferring subclass axioms from a property graph helps to address the heterogeneity between the PG and ontology models.

Duplicate record detection is a crucial task for data cleaning process in data warehouse systems. Many approaches have been presented to address this problem: some of these rely on the accuracy of the resulted records, others focus on the efficiency of the comparison process. Following the first direction, we introduce two similarity functions based on the concept of q-grams that contribute to improve accuracy of duplicate detection process with respect to other well known measures. We also reduce the number and the running time of record comparisons by building an inverted index on a sorted list of q-grams, named q-grams array. Then, we extend this approach to perform a clustering process based on the proposed q-grams array. Finally, an experimental analysis on synthetic and real data shows the efficiency of the novel indexing method for both record comparison process and clustering.

Finding information in relational data is a key task of data management in the age of information explosion. However, state-of-the-art search paradigms in relational data, such as Structured Query Language (SQL), keyword search, and query-by-example (QBE) have their own limitations. In this paper we address the issue of enhancing the power of QBE-like search interface, i.e. form-style interface, by incorporating instant search and query autocompletion, which are useful features for traditional keyword search but have not been paid enough attention specifically for form-style interfaces by database researchers. We devise effective index and efficient algorithms that can support instant search and query autocompletion using distinct attribute string values simultaneously. In addition, we also consider the issue of query autocompletion for those attributes in which each distinct attribute string value has few occurrences in the table, and devise the algorithm that provides complete words as completion results instead. We conducted extensive experiments on real-world datasets, and results show that our methods outperform baseline methods in terms of both efficiency and scalability.

Entity matching is an important task in common data cleaning and data integration problems of determining two records that refer to the same real-world entity. Many research use string similarity as features to infer entity matching but the power of the similarity may be affected by the pairs of hard-to-classify entities, which are actually different entities but have a high similarity or the same entity with low similarity. String transformation is a good solution to solve different representations between two domains of datasets, such as abbreviations, misspellings, and other expressions.In this paper, we propose two powerful features, similarity gain and dissimilarity gain, that enables us to discriminate whether the two entities refer to the same entity after string transformation. The similarity gain is defined by the maximum amount of similarity increase among the variations in similarity before and after applying string transformations. The dissimilarity is defined by the maximum amount of similarity decrease. Moreover, the similarity gain and dissimilarity gain can also be used for selecting valuable samples in a limited labeling budget. Sufficient experiments are conducted, and our method with the proposed features improves the best accuracy in most cases.KeywordsEntity matchingEntity resolutionSupervised learningString similarityString transformationFeature engineering

Entity identification deals with matching records from different datasets or within one dataset that represent the same real-world entity when unique identifiers are not available. Enabling data integration at record level as well as the detection of duplicates, entity identification plays a major role in data preprocessing, especially concerning data quality. This paper presents a framework for statistical entity identification in particular focusing on probabilistic record linkage and string matching and its implementation in R. According to the stages of the entity identification process, the framework is structured into seven core components: data preparation, candidate selection, comparison, scoring, classification, decision, and evaluation. Samples of real-world CRM datasets serve as illustrative examples.

Entity resolution is a challenging problem with unresolved and duplicated entities common in many large real world datasets. New methods are required for addressing this problem as the use of graphs to model data continues to proliferate. In this paper we propose a general framework for the fast resolution of duplicate vertices in graphs. Our framework utilises locality sensitive hashing for the quick identification of potential duplicates based on string similarity. However it is clear that in many tasks string similarity alone is not enough to determine duplication. This motivates the second aspect of our method which discovers the community structure in the graph using an ensemble of community detection algorithms. These communities are then used to augment the string similarity in the deduplication process. We evaluate our approach on a real world graph consisting of 620885 vertices and 1129986 edges and report a high accuracy score on a commercial real world graph.

In the social computing environment, the complete information about an individual is usually distributed in heterogeneous social networks, which are presented as linked data. Synthetically recognizing and integrating these distributed and heterogeneous data for efficiently information searching is an important but challenging work. In this paper, a dynamic weight (DW)-based similarity calculation is proposed to recognize and integrate similar individuals from distributed data environments. First, each link of an individual is weighted by applying DW. Then, a semantic similarity metric is proposed to combine the DW into similarity calculation. Then, a searching system framework for a similarity-based individual is designed and tested in real-world data sets. Finally, massive experiments are conducted both in benchmark and real-world social community data sets. The results show that our approach can produce a good result in similar individual searching in social networks. In addition, it performs significantly better than the existing state-of-the-art approaches in similar individual searching.

Geospatial data is ubiquitous in today’s world, playing a crucial role in various aspects of modern life. As businesses increasingly establish and expand their online presence, they leverage location-based insights to better understand customer behavior and optimize services. Simultaneously, the proliferation of GPS-enabled devices enables users to continuously share their locations and activities online, contributing to an ever-growing pool of geospatial information. Social networks and web services, in particular, heavily rely on this data, integrating it into their platforms to provide more personalized and relevant experiences. This surge in geospatial data generation necessitates the development of novel algorithms to mine meaningful patterns and enhancing user experiences by delivering accurate information at the right place and time. This dissertation presents a pipeline of novel algorithms, spanning from the integration of geospatial data, to its effective application in real-world use cases. Specifically it introduces (1) an algorithm for deduplicating and consolidating geospatial data from multiple sources, a key step in creating a comprehensive geospatial database; it studies (2) the problem of Geospatial Knowledge Graph (GeoKG) construction, and presents an algorithm to automatically mine fine-grained spatial relationships between entities from a database, within an open-world setting; it presents (3) a framework for self-supervised pre-training of geospatial foundation models, capable of generating task-agnostic representations that are readily applicable to a wide range of downstream tasks. Finally, (4) this thesis explores the fine-tuning of pre-trained large language models (LLMs) on city-specific data, to build urban virtual assistants, capable of delivering accurate geospatial recommendations in a conversational manner while minimizing hallucinations. The study on geospatial Entity Resolution (ER) proposes a state-of-the-art approach to link records from different sources, that refer to the same real-world geospatial entity. This is a crucial step for deduplicating merged data. The algorithm, called Geo-ER, leverages textual, spatial, and contextual information, making a step forward towards automatic geospatial data integration. A novel neighborhood embedding component is designed, using Graph Attention Networks, to produce a more contextualized representation of an entity. Extensive experiments, on eight real-world datasets, are performed to demonstrate the advantages of Geo-ER compared to existing state-of-the-art solution for geospatial and traditional ER. Cross-city validation experiments showcase the algorithm's robustness in scenarios where training data for a specific city is unavailable. Subsequently, this thesis presents GTMiner, a novel framework for automatically constructing a Geospatial Knowledge Graph (GeoKG) from tabular data, derived from the merging and consolidation of multiple data sources. The architecture for geospatial relation prediction is an open-world Knowledge Graph Completion solution, consisting of a pre-trained language model, a geospatial encoder, and a Geo-Textual interaction component, which jointly model the textual and geospatial characteristics of an entity. GTMiner further includes a refinement module, specifically designed to improve both coverage and correctness of a GeoKG. The study further introduces four real-world datasets, collected from publicly available sources, and carries out extensive experiments to evaluate the performance of each of the system’s components and perform ablation studies. A study on general-purpose representation learning in the geospatial domain introduces CityFM, a new framework that leverages mutual information maximization and incorporates Nodes, Ways, and Relations into its learning frame-work to train geospatial Pre-trained Foundation Models (PFMs) for a selected area of interest. CityFM-trained models are task-agnostic and multimodal, capable of integrating an entity’s textual, visual, and spatial attributes. The study conducts experiments on road, building, and region-level downstream tasks to demonstrate the effectiveness of the embeddings produced by the foundation models in real-world applications, compared to task-specific algorithms. The last study in this dissertation presents LAMP, a virtual assistant for urban applications. It introduces a novel strategy to generate synthetic conversational data about geospatial entities, to fine-tune a pre-trained LLM. The generated data enables the language model to learn and memorize thousands of places within a city, while also equipping it with spatial (proximity) awareness specific to the urban area of interest.

Geographic location selection for river bridge construction is one of the most challenging tasks because it has to deal with both numerical data and qualitative information with uncertainty. Organizational self-assessment, ensuring safety, risk measurement, supplier appraisal and many other things depend on a proper Geo-location selection. River bridge site selection is such an example in which multiple attributes; both qualitative and quantitative criteria have to be analyzed for selecting a proper bridge location. This paper demonstrates the application of a novel method named Evidential Reasoning (ER), which is a Multi Criteria Decision Analysis method applied for decision making where many criteria have to be explicitly considered for making decision. ER approach is capable for addressing the uncertainty of multi-criteria decision problem, where there exist factors of both subjective and quantitative nature. It handles uncertainties by using a degree belief structure. The evidential reasoning approach helps in aggregating degree of belief from lower level attributes to higher level attributes. To conduct the experiments of this work, Geo-location selection for The Padma Bridge construction in Bangladesh is taken as research domain and the experiment results and analysis are discussed in this paper.

İlişkisel veri analizinde temel bir problem olan topluluk seziminde, ilişkisel veri için model seçimi hala açık bir problemdir. Bu bildiride, Bayesçi atama modelinin (BAM) genel atama çerçevesi içinde, karmaşık üyelik rastlantısal öbek modellerinin (KÜRÖB) model boyutunun tahmin edilmesi önerilmektedir. İlişkisel verilerin atama modelindeki Poisson sayıları olarak nasıl temsil edildiği açıklanmaktadır. Deney sonuçları sentetik ve gerçek veri kümeleri üzerinde gösterilmektedir. Genelleyici atama yaklaşımı, sadece model boyutunu değil, aynı zamanda en uygun ayrıştırmayı seçebilen genelleştirilmiş bir model seçimi çözümü vaat etmektedir