Evaluation of Duplicate Detection Algorithms: From Quality Measures to Test Data Generation

The importance of data accuracy and quality has increased with the explosion of data size. This factor is crucial to ensure the success of any cross-enterprise integration applications, business intelligence or data mining solutions. Detecting duplicate data that represent the same real world object more than once in a certain dataset is the first step to ensure the data accuracy. This operation becomes more complicated when the same object name (person, city) is represented in multiple natural languages due to several factors including spelling, typographical and pronunciation variation, dialects and special vowel and consonant distinction and other linguistic characteristics. Therefore, it is difficult to decide whether or not two syntactic values (names) are alternative designation of the same semantic entity. Up to authors’ knowledge, the previously proposed duplicate record detection (DRD) algorithms and frameworks support only single language duplicate record detection, or at most bilingual. In this paper, two available tools of duplicate record detection are compared. Then, a generic cross language based duplicate record detection solution architecture is proposed, designed and implemented to support the wide range variations of several languages. The proposed system design uses a dictionary based on phonetic algorithms and support different indexing/blocking techniques to allow fast processing. The framework proposes the use of several proximity matching algorithms, performance evaluation metrics and classifiers to suit the diversity in several languages names matching. The framework is implemented and verified empirically in several case studies. Several Experiments are executed to compare the advantages and disadvantages of the proposed system compared to other tool. Results showed that the proposed system has substantial improvements compared to the well-known tools.KeywordsDuplicate Record DetectionCross Language Systemsentity matchingdata cleaningBig Data

Automatic test data generation from a software specification prepares test cases for software developers before their code development phase. Having test cases before coding, helps the developers to control their code to conform to the specification. In state-based specifications, paths from the initial state to the final state may be varied, this is called parallel paths. In order to generate test data which cover all behaviors in the specification, test data for each parallel path should be generated. This paper proposes an enhanced genetic algorithm(GA)-based approach to resolve the parallel paths from the UML state machine diagram. The proposed approach is improved from our previous study which uses a GA based test data generation method for only one path. The approach identifies parallel paths to cover all transitions. GA evolves a number of suitable test data sets - one test data set for each parallel path. The best test data for each parallel path are picked and used for calculating the overall coverage test data. The experimental results show improved coverage results with the enhanced approach for a number of case studies with parallel paths.

SummaryData Deduplication is the foremost technique used for data compression by removing redundant data. It is broadly utilized for cloud storage server, which reduces the un‐utilized memory space and optimized bandwidth. The confidentiality of sensitive information maintains during Deduplication detection. The Efficient Hash Function–based Duplication Detection (EHFDD) algorithm is proposed to encrypt the content before outsourcing in cloud environment. This paper is an attempt to enhance the data security and notify the issues of approved data Deduplication detections. Furthermore, it also presents several novel Deduplication improvisations for helping certified duplicate verification in hybrid cloud. The EHFDD method enables duplicate detection scheme with minimal overhead compared to existing functions. Based on experimental evaluations, proposed EHFDD algorithm reduces the average delay in 28.7 milliseconds, memory utilization in 8%, computation time in 457 milliseconds, communications overhead in 900 milliseconds, and improves the success rate in 3.13%.

Test data generation is one of the most technically challenging steps of testing software, but most commercial systems currently incorporate very little automation for this step. This paper presents results from a project that is trying to find ways to incorporate test data generation into practical test processes. The results include a new procedure for automatically generating test data that incorporates ideas from symbolic evaluation, constraint-based testing, and dynamic test data generation. It takes an initial set of values for each input, and dynamically ‘pushes’ the values through the control-flow graph of the program, modifying the sets of values as branches in the program are taken. The result is usually a set of values for each input parameter that has the property that any choice from the sets will cause the path to be traversed. This procedure uses new analysis techniques, offers improvements over previous research results in constraint-based testing, and combines several steps into one coherent process. The dynamic nature of this procedure yields several benefits. Moving through the control flow graph dynamically allows path constraints to be resolved immediately, which is more efficient both in space and time, and more often successful than constraint-based testing. This new procedure also incorporates an intelligent search technique based on bisection. The dynamic nature of this procedure also allows certain improvements to be made in the handling of arrays, loops, and expressions; language features that are traditionally difficult to handle in test data generation systems. The paper presents the test data generation procedure, examples to explain the working of the procedure, and results from a proof-of-concept implementation. Copyright © 1999 John Wiley & Sons, Ltd.

Data quality is a key factor for economical success. It is usually defined as a set of properties of data, such as completeness, accessibility, relevance, and conciseness. The latter includes the absence of multiple representations for same real world objects. To avoid such duplicates, there is a wide range of commercial products and customized self-coded software. These programs can be quite expensive both in acquisition and maintenance. In particular, small and medium-sized companies cannot afford these tools. Moreover, it is difficult to set up and tune all necessary parameters in these programs. Recently, web-based applications for duplicate detection have emerged. However, they are not easy to integrate into the local IT landscape and require much manual configuration effort.With DAQS (Data Quality as a Service) we present a novel approach to support duplicate detection. The approach features (1) minimal required user interaction and (2) self-configuration for the provided input data. To this end, each data cleansing task is classified to find out which metadata is available. Next, similarity measures are automatically assigned to the provided records’ attributes and a duplicate detection process is carried out. In this paper we introduce a novel matching approach, called one-to-some or 1:k assignment, to assign similarity measures to attributes. We performed an extensive evaluation on a large training corpus and ten test datasets of address data and achieved promising results.KeywordsDatabase ManagementDatabase ApplicationsData miningMatchingIntrinsicData Qualitymatchingduplicate detectionsimilarity measuresdata cleansing

Mobile Ad hoc NETworks (MANETs) are infrastructure-free, highly dynamic wireless networks, where central administration or configuration by the user is very difficult. In hardwired networks nodes usually rely on a centralized server and use a dynamic host configuration protocol, like DHCP [7], to acquire an IP address. Such a solution cannot be deployed in MANETs due to the unavailability of any centralized DHCP server. For small scale MANETs, it may be possible to allocate free IP addresses manually. However, the procedure becomes impractical for a large-scale or open system where mobile nodes are free to join and leave. Numerous dynamic addressing schemes for ad hoc networks have been proposed. These approaches differ in a wide range of aspects, such as the usage of centralized servers or full decentralization, hierarchical structure or flat network organization, and explicit or implicit duplicate address detection. In [1] we have proposed an autoconfiguration solution for OLSR, which can detect and resolve only a single address duplication in the network. In this paper, however, we will present a complete and optimized version of the autoconfiguration solution for OLSR proposed in [1]. This solution detects and resolve duplications whatever the number of address conflicts in the network, and, as in [1], it is based on an efficient Duplicate Address Detection(DAD) algorithm which takes advantage of the genuine optimization of the OLSR routing protocol [3].

The automatic detection of near duplicate video segments, such as multiple takes of a scene or different news video clips showing the same event, has received growing research interest in recent years. However, there is no agreed way of evaluating near duplicate detection algorithms. This makes it very hard to compare the performance of different algorithms, even if they are applied to the same data set. In this paper we have implemented several evaluation measures found in literature and we apply them to real algorithm outputs and a simulated result data set. We then calculate the correlation between the results obtained with the different measures in order to investigate whether they can be compared or not. The results show that the correlation between the measures is some cases quite low, and some measures are especially sensitive to certain types of deviations from the ground truth. However, a group of precision/recall type measures and two others are clearly correlated, though with moderate correlation coefficients. We also analyze the correlation between these measures and the subjective human judgment of the number of repeated segments in summary videos.

One of the ways of test data generation is using the path-oriented (path-wise) test data generator. This generator takes the program code and test adequacy criteria as input and generates the test data in order to satisfy the adequacy criteria. One of the problems of this generator in test data generation is the lack of attention to generating the diagnostic test data. In this paper a new approach has been proposed for path-oriented test data generation and by utilizing it, test data is generated automatically realizing the goal of discovering more important errors in the least amount of time. Since that some of the instructions of any programming language are more error-prone, so the paths that contain these instructions are selected for perform test data generation process. Then, the input domains of these paths variables are divided by a divide-and-conquer algorithm to the subdomains. A set of different subdomains is called hyper cuboids, and test data will be generated by these hyper cuboids. We apply our method in some programs, and compare it with some previous methods. Experimental results show proposed approach outperforms same previous approaches.

Mutation testing is a powerful software testing technique. However, it is difficult to obtain test data for killing a large number of mutants, especially for hard-to-kill mutants. Mutant branch is formed by an original statement of a program under test and its mutated statement. The true branch of a mutant branch is covered by a test datum, suggesting the corresponding mutant is killed under the criterion of weak mutation testing. This article focuses on efficiently generating test data for a large number of mutant branches. When generating test data using a search-based method, the size of the search domain is a determining factor affecting the search performance. The key observation is that only partial input variables affect whether a mutant will be killed, so the search domain can be reduced by deleting irrelevant variables. Along this line, we first group the mutant branches based on their relevant input variables, followed by formulating a multi-task optimization model of test data generation for the grouped mutant branches, in which the relevant input variables are taken as the decision variables. Finally, a multi-population genetic algorithm with individual sharing is employed to generate test data by multi-tasking. The experiments based on eight programs of various sizes show that removing related variable helps reduce the search domain, and the efficiency of test data generation by grouping and multitasking is improved.

Automatic test data generation is required to generate test cases dynamically for a specific software program. Manual generation of test data is too tedious and a time consuming task. This paper proposes a technique using Intelligent Water Drop (IWD) for automatic generation of test data. Correctly generated test data helps in reducing the effort while testing the software. This paper discusses different algorithms based on IWD to generate test data and path coverage over Control Flow Graph. Test data is generated keeping in mind all of the programming constraints like “if,” “while,” “do while,” etc., available in the program.

Software testing is the primary phase, which is performed during software development and it is carried by a sequence of instructions of test inputs followed by expected output.The Harmony Search (HS) algorithm is based on the improvisation process of music.In comparison to other algorithms, the HSA has gain popularity and superiority in the field of evolutionary computation.When musicians compose the harmony through different possible combinations of the music, at that time the pitches are stored in the harmony memory and the optimization can be done by adjusting the input pitches and generate the perfect harmony.The test case generation process is used to identify test cases with resources and also identifies critical domain requirements.In this paper, the role of Harmony search meta-heuristic search technique is analyzed in generating random test data and optimized those test data.Test data are generated and optimized by applying in a case study i.e. a withdrawal task in Bank ATM through Harmony search.It is observed that this algorithm generates suitable test cases as well as test data and gives brief details about the Harmony search method.It is used for test data generation and optimization.

4736375 Apparatus for fast generation of large quantities of test data words in a test facility

In search-based structural testing, metaheuristic search techniques have been frequently used to automate the test data generation. In Genetic Algorithms (GAs) for example, test data are rewarded on the basis of an objective function that represents generally the number of statements or branches covered. However, owing to the wide diversity of possible test data values, it is hard to find the set of test data that can satisfy a specific coverage criterion. In this paper, we introduce the use of Novelty Search (NS) algorithm to the test data generation problem based on statement-covered criteria. We believe that such approach to test data generation is attractive because it allows the exploration of the huge space of test data within the input domain. In this approach, we seek to explore the search space without regard to any objectives. In fact, instead of having a fitness-based selection, we select test cases based on a novelty score showing how different they are compared to all other solutions evaluated so far.

In software testing process, test data generation represents an important step for high quality software, even for mobile devices. As proposed in previous works, a potential source for random data generation is represented by the UI layout files that are used for almost all mobile platforms (Android, iOS, Windows Phone/Mobile). This paper continues the previous work and presents a test data generation system based on Android layout files. The test data generator uses DSL files as input and generates test data that conform to several testing principles. The generated test data could be stored in XML files or any format required by the testing frameworks.

We present a novel method for automatically generating artificialtest data that are particularly suited for testing form-centric software applications with several thousand input fields. The complex validation rules for user input are translated to a constraint satisfaction problem (CSP), which is solved using an off-the-shelf SMT-solver. In order to exert pressure onto the software under test, the generated test data have to incorporate extreme and special values (ESVs) for each field. The SMT-solver is aided by a sophisticated graph-based cluster algorithm and by other heuristic methods in order to reduce the complexity of the CSPs. With further optimizations, the test data generator now routinely generates a complete set of test data records for large form-centric applications within less than two hours. The test data generator described here is operationally being used for automated tests of form-centric Web-applications, within an iterative development process emphasizing very early testing of software applications.