Data De-duplication: A Review

Due to increasing volume of data in information technology, saving storage space and providing security to data has acquired more attention and popularity. In data processing and data mining, duplicates can effect severely. Data deduplication is a technique that eliminates duplicate data and store only one copy, promoting single instance storage. The main challenges are to identify maximum duplicate segment and selecting the storage nodes for distributing fragments of files. In this paper we proposed, Duplicate Detection and Fragment Placement (DDFP) a deduplication system that effectively eliminates duplicate data and fragments placement that allocates unique instances of a data file on storage nodes. For repeated data, reference pointer is used and only unique data is stored on the storage node. This increases the percentage of duplicate data detection. A fragment placement algorithm is used for placing fragments on different storage nodes. To select nodes T-coloring is used, Set T is used, which restricts the nodes that are at distance T from one another. DDFP considerably achieves duplicate elimination and obtain the high level of security on data fragments by storing fragments of the data file using T-coloring. This Selects the nodes that are not adjacent which prevent unauthorized access to data from other users.

In computing, duplicate data detection refers to identifying duplicate copies of repeating data. Identifying duplicate data items in streamed data and eliminating them before storing, is a complex job. This paper proposes a novel data structure for duplicate detection using a variant of stable Bloom filter named as FingerPrint Stable Bloom Filter (FP-SBF). The proposed approach uses counting Bloom filter with fingerprint bits along with an optimization mechanism for duplicate detection. FP-SBF uses d-left hashing which reduces the computational time and decreases the false positives as well as false negatives. FP-SBF can process unbounded data in single pass, using k hash functions, and successfully differentiate between duplicate and distinct elements in O(k+1) time, independent of the size of incoming data. The performance of FP-SBF has been compared with various Bloom Filters used for stream data duplication detection and it has been theoretically and experimentally proved that the proposed approach efficiently detects the duplicates in streaming data with less memory requirements.

Duplicate data must be properly handled in the process of intelligent search, surveillance, and tracking. To better detect and fuse this data, the authors take a new perspective on feature fusion based on the and reaction' mechanism of data objects. This mechanism can greatly facilitate differentiating duplicate and nonduplicate data in the fusion (reaction) probability. With this perspective, the fusion probability of a feature sample in a dataset is the cumulative effect of the microscopic collision (interaction) results between this sample and other samples. The operations on duplicate detection and feature fusion depend on the fusion probability and are finally realized when the collision process is finished. This new quantum-inspired feature fusion model gives interesting experimental results and could have great implications for some intelligent systems.

Entity Resolution is the process of identifying whether or not various entities from different sources are referring to the same real-world entity. Entity Resolution hasn't been extensively researched in graph databases, whereas it has been for relational databases. This paper focuses on providing comparisons of experiments on various datasets to determine the most appropriate method used in the Entity Resolution process from among literature's similarity algorithms, graph embedding techniques, and graph embedding algorithms combined to link prediction. Moreover, if the embedding algorithm employed has an impact on the given results. The results show that the Entity Resolution process performed better when graph embedding techniques were paired with link prediction, and the chosen graph embedding algorithm also has an impact on the results.

The presence of duplicate records is a major data quality concern in large databases. To detect duplicates, entity resolution also known as duplication detection or record linkage is used as a part of the data cleaning process to identify records that potentially refer to the same real-world entity. We present the Stringer system that provides an evaluation framework for understanding what barriers remain towards the goal of truly scalable and general purpose duplication detection algorithms. In this paper, we use Stringer to evaluate the quality of the clusters (groups of potential duplicates) obtained from several unconstrained clustering algorithms used in concert with approximate join techniques. Our work is motivated by the recent significant advancements that have made approximate join algorithms highly scalable. Our extensive evaluation reveals that some clustering algorithms that have never been considered for duplicate detection, perform extremely well in terms of both accuracy and scalability.

The detection of duplicate and related content is a critical data curation task in the context of digital research collections. This task can be challenging, if not impossible, to do manually in large, unstructured, and noisy collections. While there are many automated solutions for deduplicating data that contain large numbers of identical copies, it can be particularly difficult to find a solution for identifying redundancy within image-heavy collections that have evolved over a long span of time or have been created collaboratively by large groups. These types of collections, especially in academic research settings, in which the datasets are used for a wide range of publication, teaching, and research activities, can be characterized by (1) large numbers of heterogeneous file formats, (2) repetitive photographic documentation of the same subjects in a variety of conditions (3) multiple copies or subsets of images with slight modifications (e.g., cropping or color-balancing) and (4) complex file structures and naming conventions that may not be consistent throughout. In this chapter, we present a scalable and automated approach for detecting duplicate, similar, and related images, along with subimages, in digital data collections. Our approach can assist in efficiently managing redundancy in any large image collection on High Performance Computing (HPC) resources. While we illustrate the approach with a large archaeological collection, it is domain-neutral and is widely applicable to image-heavy collections within any HPC platform that has general-purpose processors.

The WHO Collaborating Centre for International Drug Monitoring in Uppsala, Sweden, maintains and analyses the world's largest database of reports on suspected adverse drug reaction (ADR) incidents that occur after drugs are on the market. The presence of duplicate case reports is an important data quality problem and their detection remains a formidable challenge, especially in the WHO drug safety database where reports are anonymised before submission. In this paper, we propose a duplicate detection method based on the hit-miss model for statistical record linkage described by Copas and Hilton, which handles the limited amount of training data well and is well suited for the available data (categorical and numerical rather than free text). We propose two extensions of the standard hit-miss model: a hit-miss mixture model for errors in numerical record fields and a new method to handle correlated record fields, and we demonstrate the effectiveness both at identifying the most likely duplicate for a given case report (94.7% accuracy) and at discriminating true duplicates from random matches (63% recall with 71% precision). The proposed method allows for more efficient data cleaning in post-marketing drug safety data sets, and perhaps other knowledge discovery applications as well.

Inter-organization knowledge management in the context of collaborative networks is a critical activity for business success. During the evolution of collaborative network specific technologies, increasingly performant instruments were created to exploit this knowledge. As a development cycle, inter-organizational knowledge is built on the foundation of information and data owned by the participants in the collaborative networks. One of the most widely used instruments to exploit this data and knowledge, with the purpose of creating new knowledge, is Data Mining. In the context of this paper, data mining is the process of discovering patterns and hidden relations in very large data collections, stored in data banks or data bases. Because only in extremely rare cases reading data tables record by record leads to the discovery of useful patterns, the information must be processed automatically, process known as Knowledge Discovery. Knowledge Discovery is a component that combines the power of computers with a human operator that has the ability to find the visual patterns revealed by the system. Using an automated data mining system, the computer finds the existing informational patterns and the human factor (the analyst) evaluates those patterns and picks the ones that are really relevant for the current analysis. Considering the current technological context, where storage devices are more and more accessible and performant, the storage capacity is no longer a barrier preventing storage of all required data. Exploitation of inter-organizational knowledge in collaborative networks leads the research to the field of business intelligence applied even on social environment. This approach belongs in literature to the general branch of social business intelligence.Keywords: Collaborative Networks, Information Technology, Inter-Organizational Knowledge, Knowledge Management, Knowledge Discovery, Data Mining1 IntroductionT he importance of knowledge and organizational networks in obtaining a competitive advantage on the market is recognized both by theoreticians and by practitioners as well. The increase of the number of collaborative networks, the accent on knowledge within the society based on knowledge and innovation come to support those who want to adhere to new organizational forms that would lead to obtaining success on the global market.Knowledge is known as one of the most important assets of management within organizations, as knowledge allows organizations to use and develop resources, to increase the competitive ability and to obtain a substantial competitive advantage [1]. Knowledge also represents an important source that allows nations, organizations, and persons to obtain benefits, such as: learning improvement, innovation, and decision making. Any organization, public or private, needs a knowledge management process in order to obtain the best performances [2].In global economy, the strong competition, the frequent changes on the market, the higher and higher demands concerning quality, lead to the necessity of new organizational forms. Organizational networks are acknowledged as organizational forms characterized by an increased flexibility and that may lead to obtaining the competitive advantage on the market. Apart from the potential advantages, organizational networks are also confronted with problems and challenges that are particularly connected to the complexity of the collaborative environment.Inter-organizational knowledge can be defined as an explicit set of knowledge that is formalized and created by organizations [3]. The interactions within the network allow organizations to develop the collaborative and relational tacit knowledge and to generate inter-organizational tacit knowledge that can be capitalized within the interorganizational memory [4]. The interorganizational knowledge allows organizations to develop distinctive abilities, which may lead to the increase of the innovation ability. …

Purpose The extraction of relevant knowledge from data is called knowledge discovery (KD). The KD process requires a large amount of data and it must be reliable before mining. Complexity is not only in deriving knowledge from data but also in improving system performance with a psycho-cognitive approach. KD demands a high level of human cognition and mental activity to generate and retrieve knowledge. Therefore, this study aims to explain how psychological knowledge is involved in KD. Design/methodology/approach By understanding the cognitive processes that lead to knowledge production, KD can be improved through interventions that target psychological processes, such as attention, learning and memory. In addition, psycho-cognitive approaches can help us to better grasp the process of KD and the factors that influence its effectiveness. The study attempted to correlate interdependence by interpreting cognitive approaches to KD from a psychological perspective. The authors of this paper draw on both primary and secondary literary warrants to empirically prove psychological bending in KD. Findings Understanding the psychological aspects of data and KD can identify the development of tools, process and environments that support individual and teams in making sense of data and extracting valuable knowledge. The study also finds that interdisciplinary collaboration, bringing together expertise in psychology, data science and domain specific knowledge fosters effective KD processes. Originality/value The KD system cannot function well and will not be able to achieve its full potential without psycho-cognitive foundation. It was found that KD in the KD system is influenced by human cognition. The authors made a contribution to KD by fusing psycho-cognitive approaches with data-driven technology and machine learning.

SummaryData Deduplication is the foremost technique used for data compression by removing redundant data. It is broadly utilized for cloud storage server, which reduces the un‐utilized memory space and optimized bandwidth. The confidentiality of sensitive information maintains during Deduplication detection. The Efficient Hash Function–based Duplication Detection (EHFDD) algorithm is proposed to encrypt the content before outsourcing in cloud environment. This paper is an attempt to enhance the data security and notify the issues of approved data Deduplication detections. Furthermore, it also presents several novel Deduplication improvisations for helping certified duplicate verification in hybrid cloud. The EHFDD method enables duplicate detection scheme with minimal overhead compared to existing functions. Based on experimental evaluations, proposed EHFDD algorithm reduces the average delay in 28.7 milliseconds, memory utilization in 8%, computation time in 457 milliseconds, communications overhead in 900 milliseconds, and improves the success rate in 3.13%.

Notice of Violation of IEEE Publication Principles<br><br>"Knowledge Discovery from Virtual Enterprise Model Based on Semantic Annotation,"<br>by Chengzhu Sun, Xiaofei Xu, Xiangyang Li, Shengchun Deng,<br>in the Proceedings of the Fifth International Conference on Fuzzy Systems and Knowledge Discovery, 2008. FSKD '08, vol.5, pp.546-551, Oct. 2008<br><br>After careful and considered review of the content and authorship of this paper by a duly constituted expert committee, this paper has been found to be in violation of IEEE's Publication Principles.<br><br>This paper contains significant portions of original text from the paper cited below. The original text was copied without attribution (including appropriate references to the original author(s) and/or paper title) and without permission.<br><br>"A Controlled Language for Semantic Annotation and Interoperability in e-Business Applications"<br>by M.Missikoff, F.Schiappelli , F.Taglino<br>in the Proceedings of the Semantic Integration Workshop, collocated with the Second International Semantic Web Conference (ISWC-03), October 2003<br><br> <br/> Discovering knowledge from virtual enterprise model is becoming increasingly important, as numerical models established for virtual enterprise are difficult to support interoperability of virtual enterprise. To solve this problem, a knowledge discovery method based on semantic annotation is put forward in this paper. A graphic ontology representation schema for virtual enterprise model is described, which is called ontology structure graph (OSG). Based on applications of semantic annotation, the process of knowledge discovery in the virtual enterprise model base is given and activities such as model selection, semantic annotation, data transformation, knowledge extraction and ontology interoperation in knowledge discovery process are illustrated in detail. Then, several critical issues influencing knowledge discovery are explained, including the organization of domain vocabulary, the definition of semantic annotation rules and semantic affinity function, the formulation of reference ontology. Finally, an instance is given to demonstrate the knowledge discovery method and results of knowledge discovery are presented.

The increasing prevalence of JSON documents as a standard format for data storage and exchange in diverse applications has led to the need for efficient methods to compare and analyze hierarchical data structures. Traditional comparison methods often struggle with scalability and fail to account for structural variations in complex data formats. These limitations become particularly problematic in applications such as duplicate detection, anomaly analysis, and data integration, where accurate and scalable comparison of large JSON datasets is essential.To address these challenges, this paper presents a scalable framework for comparing JSON documents using the Aho-Hopcroft-Ullman (AHU) algorithm and the MapReduce paradigm. By generating canonical labels for hierarchical structures, the AHU algorithm captures structural similarities between JSON trees. The framework employs structural similarity measures, such as Longest Common Substring (LCS) and Levenshtein Distance, to quantify resemblance. MapReduce enables efficient processing of large JSON datasets, with a mapping phase for parsing and labeling and a reducing phase for similarity computations. This approach offers a robust and scalable solution for hierarchical data comparison, facilitating critical applications in duplicate detection and anomaly analysis.

A major source of uncertainty in databases is the presence of duplicate items, i.e., records that refer to the same real-world entity. However, accurate deduplication is a difficult task and imperfect data cleaning may result in loss of valuable information. A reasonable alternative approach is to keep duplicates when the correct cleaning strategy is not certain, and utilize an efficient probabilistic query-answering technique to return query results along with probabilities of each answer being correct. In this paper, we present a flexible modular framework for scalably creating a probabilistic database out of a dirty relation of duplicated data and overview the challenges raised in utilizing this framework for large relations of string data. We study the problem of associating probabilities with duplicates that are detected using state-of-the-art scalable approximate join methods. We argue that standard thresholding techniques are not sufficiently robust for this task, and propose new clustering algorithms suitable for inferring duplicates and their associated probabilities. We show that the inferred probabilities accurately reflect the error in duplicate records.

Entity identification deals with matching records from different datasets or within one dataset that represent the same real-world entity when unique identifiers are not available. Enabling data integration at record level as well as the detection of duplicates, entity identification plays a major role in data preprocessing, especially concerning data quality. This paper presents a framework for statistical entity identification in particular focusing on probabilistic record linkage and string matching and its implementation in R. According to the stages of the entity identification process, the framework is structured into seven core components: data preparation, candidate selection, comparison, scoring, classification, decision, and evaluation. Samples of real-world CRM datasets serve as illustrative examples.

The problem of identifying objects in databases that refer to the same real world entity, is known, among others, as duplicate detection or record linkage. Objects may be duplicates, even though they are not identical due to errors and missing data. Typical current methods require deep understanding of the application domain or a good representative training set, which entails significant costs. In this paper we present an unsupervised, domain independent approach to duplicate detection that starts with a broad alignment of potential duplicates, and analyses the distribution of observed similarity values among these potential duplicates and among representative sample non-duplicates to improve the initial alignment. Additionally, the presented approach is not only able to align flat records, but makes also use of related objects, which may significantly increase the alignment accuracy. Evaluations show that our approach supersedes other unsupervised approaches and reaches almost the same accuracy as even fully supervised, domain dependent approaches.