Instance-Based ‘One-to-Some’ Assignment of Similarity Measures to Attributes

The importance of data accuracy and quality has increased with the explosion of data size. This factor is crucial to ensure the success of any cross-enterprise integration applications, business intelligence or data mining solutions. Detecting duplicate data that represent the same real world object more than once in a certain dataset is the first step to ensure the data accuracy. This operation becomes more complicated when the same object name (person, city) is represented in multiple natural languages due to several factors including spelling, typographical and pronunciation variation, dialects and special vowel and consonant distinction and other linguistic characteristics. Therefore, it is difficult to decide whether or not two syntactic values (names) are alternative designation of the same semantic entity. Up to authors’ knowledge, the previously proposed duplicate record detection (DRD) algorithms and frameworks support only single language duplicate record detection, or at most bilingual. In this paper, two available tools of duplicate record detection are compared. Then, a generic cross language based duplicate record detection solution architecture is proposed, designed and implemented to support the wide range variations of several languages. The proposed system design uses a dictionary based on phonetic algorithms and support different indexing/blocking techniques to allow fast processing. The framework proposes the use of several proximity matching algorithms, performance evaluation metrics and classifiers to suit the diversity in several languages names matching. The framework is implemented and verified empirically in several case studies. Several Experiments are executed to compare the advantages and disadvantages of the proposed system compared to other tool. Results showed that the proposed system has substantial improvements compared to the well-known tools.KeywordsDuplicate Record DetectionCross Language Systemsentity matchingdata cleaningBig Data

Measuring the similarity of two records is a challenging problem, but necessary for fundamental tasks, such as duplicate detection and similarity search. By exploiting frequencies of attribute values, many similarity measures can be improved: In a person table with U.S. citizens, Arnold Schwarzenegger is a very rare name. If we find several Arnold Schwarzeneggers in it, it is very likely that these are duplicates. We are then less strict when comparing other attribute values, such as birth date or address. We put this intuition to use by partitioning compared record pairs according to frequencies of attribute values. For example, we could create three partitions from our data: Partition 1 contains all pairs with rare names, Partition 2 all pairs with medium frequent names, and Partition 3 all pairs with frequent names. For each partition, we learn a different similarity measure: we apply machine learning techniques to combine a set of base similarity measures into an overall measure. To determine a good partitioning, we compare different partitioning strategies. We achieved best results with a novel algorithm inspired by genetic programming.

The increasing prevalence of JSON documents as a standard format for data storage and exchange in diverse applications has led to the need for efficient methods to compare and analyze hierarchical data structures. Traditional comparison methods often struggle with scalability and fail to account for structural variations in complex data formats. These limitations become particularly problematic in applications such as duplicate detection, anomaly analysis, and data integration, where accurate and scalable comparison of large JSON datasets is essential.To address these challenges, this paper presents a scalable framework for comparing JSON documents using the Aho-Hopcroft-Ullman (AHU) algorithm and the MapReduce paradigm. By generating canonical labels for hierarchical structures, the AHU algorithm captures structural similarities between JSON trees. The framework employs structural similarity measures, such as Longest Common Substring (LCS) and Levenshtein Distance, to quantify resemblance. MapReduce enables efficient processing of large JSON datasets, with a mapping phase for parsing and labeling and a reducing phase for similarity computations. This approach offers a robust and scalable solution for hierarchical data comparison, facilitating critical applications in duplicate detection and anomaly analysis.

Duplicate detection identifies multiple records in a dataset that represent the same real-world object. Many such approaches exist, both in research and in industry. To investigate essential properties of duplicate detection algorithms, such as their result quality or runtime behavior, they must be executed on suitable test data. The quality evaluation requires that these test data are labeled, constituting a ground truth. Correctly labeled, sizable, and real or at least realistic test datasets, however, are not easy to obtain, creating an obstacle for the advancement of research. In this tutorial, we present common methods to evaluate duplicate detection algorithms and to generate labeled test data. We close with a discussion of open problems.

Duplicate bug report detection is one of the major problems in software triage systems like Bugzilla to deal with end user requests. User request contains some categorical and especially textual fields which need feature extraction for duplicate detection. Contextual and topical features are acquired using calculating cosine similarity between term frequency or inverse document frequency or BM25F technique from a pair of bug reports against some topics. This research proposes the individual Manhattan distance similarity approach instead of cosine distance similarity for every topic in contextual features to expand the feature dimension which can increase the accuracy of the duplicate bug report detection process. The four famous datasets of bug reports have used for evaluation of the proposed method including Android, Eclipse, Mozilla, and Open Office which the experimental results indicate performance improvement for four contextual features including general, cryptography, network, and Java topics.

THINGS-data reflects three large-scale neuroimaging and behavioral datasets of object processing in humans, comprising densely sampled functional MRI and magnetoencephalographic recordings, as well as 4.70 million similarity judgments in response to thousands of photographic images for up to 1854 objects.

Poor data quality is known to compromise the credibility and efficiency of commercial as well as public endeavours. Several developments from industry and academia have contributed significantly towards addressing the problem. These typically include analysts and practitioners who have contributed to the design of strategies and methodologies for data governance; solution architects including software vendors who have contributed towards appropriate system architectures that promote data integration and; and data experts who have contributed to data quality problems such as duplicate detection, identification of outliers, consistency checking and many more through the use of computational techniques. The attainment of true data quality lies at the convergence of the three aspects, namely organizational, architectural and computational.KeywordsData QualityPoor Data QualityProvenance InformationSource AttributionCollaborative SettingThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Entity identification deals with matching records from different datasets or within one dataset that represent the same real-world entity when unique identifiers are not available. Enabling data integration at record level as well as the detection of duplicates, entity identification plays a major role in data preprocessing, especially concerning data quality. This paper presents a framework for statistical entity identification in particular focusing on probabilistic record linkage and string matching and its implementation in R. According to the stages of the entity identification process, the framework is structured into seven core components: data preparation, candidate selection, comparison, scoring, classification, decision, and evaluation. Samples of real-world CRM datasets serve as illustrative examples.

A Real-World Size Organization of Object Responses in Occipitotemporal Cortex

The Information Society relies on an increasingly diverse mass of digital information, a trend accentuated by the Internet of Things in which Real World Objects (RWOs) have their own Resource Identifiers and related data. In this setting, the goals of the CONVERGENCE framework are to enhance current media handling with new functionality and extend the traditional concept of media to include digital representations of RWOs. The CONVERGENCE framework is based on the concept of Versatile Digital Item (VDI), a structured package of digital content and meta-information, inspired by the MPEG-21 standard but designed to address a broader range of application domains, including the management of RWOs in the Internet of Things. The VDI is supported by an innovative middleware and by tools and applications. The CONVERGENCE framework incorporates six innovations: (1) VDIs provide uniform mechanisms to handle different classes of information, including RWOs; (2) VDIs are intrinsically dynamic, allowing both providers and consumers to update content; (3) VDIs support “digital forgetting” (automatic “un-publishing,” automated garbage collection of VDIs after a user-defined expiry date); (4) VDIs meet security and privacy needs of both professional and nonprofessional consumers and providers; (5) VDIs support new modes of semantic search; (6) VDIs allow easy sharing of information across multiple, heterogeneous devices. In this chapter, we present the CONVERGENCE framework, its main concepts and ideas, some proposals to make it real, and some use cases and we show how our proposed concept can be a fundamental enabler for the Internet of Things.

Correlating image semantics with its low level features is a challenging task. Although, humans are adept in distinguishing object categories, both in visual as well as in semantic space, but to accomplish this computationally is yet to be fully explored. The learning based techniques do minimize the semantic gap, but unlimited possible categorization of objects in real world is a major challenge to these techniques. This work analyzes and utilizes the strength of a semantically categorized image database to assign semantics to query images. Semantics based categorization of images would result in image hierarchy. The algorithms proposed in this work exploit visual image descriptors and similarity measures in the context of a semantically categorized image database. A novel 'Branch Selection Algorithm' is developed for a highly categorized and dense image database, which drastically reduces the search space. The search space so obtained is further reduced by applying any one of the four proposed 'Pruning Algorithms'. Pruning algorithms maintain accuracy while reducing the search space. These algorithms use an adaptive combination of multiple visual features of an image database to find semantics of query images. Branch Selection Algorithm tested on a subset of 'ImageNet' database reduces search space by 75%. The best pruning algorithm further reduces this search space by 26% while maintaining 95% accuracy.

The presence of duplicate records is a major data quality concern in large databases. To detect duplicates, entity resolution also known as duplication detection or record linkage is used as a part of the data cleaning process to identify records that potentially refer to the same real-world entity. We present the Stringer system that provides an evaluation framework for understanding what barriers remain towards the goal of truly scalable and general purpose duplication detection algorithms. In this paper, we use Stringer to evaluate the quality of the clusters (groups of potential duplicates) obtained from several unconstrained clustering algorithms used in concert with approximate join techniques. Our work is motivated by the recent significant advancements that have made approximate join algorithms highly scalable. Our extensive evaluation reveals that some clustering algorithms that have never been considered for duplicate detection, perform extremely well in terms of both accuracy and scalability.

Many researchers and practitioners have been attracted to improve data quality due to its monumental importance as a key success factor. Mathematical and statistical models have been deployed to information systems to introduce constrain and transaction based mechanisms to prevent data quality related problems. Entire management of the process and roles involved in data generation has also been scrutinized. Vast amount of knowledge base has been progressed in this area; however, most of the approaches are limited from practical perspective. System development process incorporating quality modelling is rarely integrated. Quality related meta data is absent from most information system. Neither process mapping nor data modelling provides sufficient provision to measure quality or certification of data in the information systems. Furthermore, ongoing monitoring of data for quality conformance through a separate process is expensive and time consuming. Recognising this limitation and aiming to provide a practical-orient comprehensive approach, we propose a process centric quality focused system design incorporating data product quality, conformance monitoring and certification. In this paper we focus on the self certification of data quality based on our earlier work on the process centric framework for ongoing data quality monitoring.

In \em monotone classification, the input is a set P of n points in d-dimensional space, where each point carries a label 0 or 1. A point p \em dominates another point q if the coordinate of p is at least that of q on every dimension. A \em monotone classifier is a function h mapping each d-dimensional point to $\0, 1 $, subject to the condition that $h(p) \ge h(q)$ holds whenever p dominates q. The classifier h \em mis-classifies a point $p \in P$ if $h(p)$ is different from the label of p. The \em error of h is the number of points in P mis-classified by h. The objective is to find a monotone classifier with a small error. The problem is fundamental to numerous database applications in entity matching, record linkage, and duplicate detection. This paper studies two variants of the problem. In the first \em active version, all the labels are hidden in the beginning; an algorithm must pay a unit cost to \em probe (i.e., reveal) the label of a point in P. We prove that $Omega(n)$ probes are necessary to find an optimal classifier even in one-dimensional space ($d=1$). On the other hand, given an arbitrary $\eps > 0$, we show how to obtain (with high probability) a monotone classifier whose error is worse than the optimum by at most a $1 + \eps$ factor, while probing $\tO(w/\eps^2)$ labels, where w is the dominance width of P and $\tO(.)$ hides a polylogarithmic factor. For constant $\eps$, the probing cost matches an existing lower bound up to an $\tO(1)$ factor. In the second \em passive version, the point labels in P are explicitly given; the goal is to minimize CPU computation in finding an optimal classifier. We show that the problem can be settled in time polynomial to both d and n.

Recent advancements in the Internet of Things (IoT) and the Web of Things (WoT) accompany a smart life where real world objects, including sensing devices, are interconnected with each other. The Web representation of smart objects empowers innovative applications and services for various domains. To accelerate this approach, Web of Objects (WoO) focuses on the implementation aspects of bringing the assorted real world objects to the Web applications. In this paper; we propose an emergency fire management system in the WoO infrastructure. Consequently, we integrate the formation and management of Virtual Objects (ViO) which are derived from real world physical objects and are virtually connected with each other into the semantic ontology model. The charm of using the semantic ontology is that it allows information reusability, extensibility and interoperability, which enable ViOs to uphold orchestration, federation, collaboration and harmonization. Our system is context aware, as it receives contextual environmental information from distributed sensors and detects emergency situations. To handle a fire emergency, we present a decision support tool for the emergency fire management team. The previous fire incident log is the basis of the decision support system. A log repository collects all the emergency fire incident logs from ViOs and stores them in a repository.