Mutation testing strategies using mutant classification

The ratio of equivalent mutants: A key to analyzing mutation equivalence

Mutation testing can be used to measure the quality of a given test suite. But two flaws prevent wide acceptance. First, there are equivalent mutants - mutants that are semantically equivalent to the unmodified version and therefore unkillable. Manually identifying those mutants is time-consuming and error-prone. Second, initially there are often too few test cases. Mutation testing detects missing cases. But it is too time-consuming to manually write all the required test cases. This paper shows how to use symbolic execution to tackle both problems, i.e., to detect equivalent mutants and exclude them from further analysis, and to automatically generate test cases that kill the remaining mutants. Our evaluation uses a set of 252 publicly available mutants for which it is known that they are hard to classify. Despite the fact that detecting equivalent mutants is an undecidable problem in general, our fully automatic tool MutantDistiller correctly classifies all of them (13 equivalent, 239 non-equivalent). MutantDistiller also generates test cases that kill the nonequivalent mutants.

The use of Genetic Algorithms in evolution of mutants and test cases offers new possibilities in addressing some of the main problems of mutation testing. Most specifically the problem of equivalent mutant detection, and the problem of the large number of mutants produced. In this paper we describe the above problems in detail and introduce a new methodology based on co-evolutionary search techniques using Genetic Algorithms in order to address them effectively. Co-evolution allows the parallel evolution of mutants and test cases. We discuss the advantages of this approach over other existing mutation testing techniques, showing details of some initial experimental results carried out.KeywordsGenetic AlgorithmMutation OperatorSoftware TestMutation TestingOriginal ProgramThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Manual test case generation is an exhaustive and time-consuming process. However, automated test data generation may reduce the efforts and assist in creating an adequate test suite embracing predefined goals. The quality of a test suite depends on its fault-finding behavior. Mutants have been widely accepted for simulating the artificial faults that behave similarly to realistic ones for test data generation. In prior studies, the use of search-based techniques has been extensively reported to enhance the quality of test suites. Symmetry, however, can have a detrimental impact on the dynamics of a search-based algorithm, whose performance strongly depends on breaking the “symmetry” of search space by the evolving population. This study implements an elitist Genetic Algorithm (GA) with an improved fitness function to expose maximum faults while also minimizing the cost of testing by generating less complex and asymmetric test cases. It uses the selective mutation strategy to create low-cost artificial faults that result in a lesser number of redundant and equivalent mutants. For evolution, reproduction operator selection is repeatedly guided by the traces of test execution and mutant detection that decides whether to diversify or intensify the previous population of test cases. An iterative elimination of redundant test cases further minimizes the size of the test suite. This study uses 14 Java programs of significant sizes to validate the efficacy of the proposed approach in comparison to Initial Random tests and a widely used evolutionary framework in academia, namely Evosuite. Empirically, our approach is found to be more stable with significant improvement in the test case efficiency of the optimized test suite.

Android is an operating system source which offers flexibility and support for most mobile applications, and easy access to social networks. It is important to understand the complexity of design, development, implementation, and testing of Android apps. A number of challenges may be faced in testing android applications, including the lack of testing processes and methods, testing experts being unavailable, poor in-house testing environment, and time restrictions. Mutation testing is a fault-based testing technique, applied by generating mutants and running the application with these mutants to analyze the killed and equivalent mutants. We defined a set of mutation operators according to the features of android applications: apps with content sharing, apps with multimedia, apps with graphics, and apps with user location and maps. We identified 42 mutation operators. In addition, we implemented a new tool, “µ-Android,” which automatically generates mutants and retrieves results to prove the efficiency of the test cases and enable the new operators.

Equivalent mutants are one of the major costs of mutation testing. The undecidable nature of this problem makes a fully automated solution unattainable and necessitates the manual analysis of live mutants. This paper introduces the concept of mirrored mutants, ones that affect similar code fragments. It is argued that mirrored mutants exhibit analogous behavior with respect to their equivalence. Thus, if one of them is equivalent, then the other mirrored mutants should be too. An empirical study, conducted on real world programs, investigates this argument, focusing on both intra-method and inter-method mirrored mutants. The obtained results suggest that mirrored mutants indeed exhibit this kind of behavior and thus can be utilized to ameliorate the adverse effects of the equivalent mutant problem.

Mutation Testing originated from a classical method for digital circuit testing and today is used at program and specification levels. It can be used either to generate or to assess the quality of test sets. In spite of being very effective in detecting faults, Mutation Testing is usually considered a high cost criterion due to: i) the large number of generated mutants; ii) the time-consuming activity of determining equivalent mutants; and iii) the mutant execution time. Many initiatives aiming at reducing the Mutation Testing application cost have been conducted, most of them addressing one of the drawbacks mentioned above. In this paper, we identify and summarize some of the most relevant researches and results related to Mutation Testing cost reduction, e.g., Constrained-Mutation, Constraint-Based Testing and Bayesian Learning. Moreover, we propose a Mutation Testing process, named Muta-Pro, that synergetically integrates the related approaches and mechanisms. This process is intended to be incremental and tailorable to a specific application domain such as C programs or finite state machine models. The main ideas in this paper are illustrated using a UNIX utility program. This process is being integrated in a Mutation Testing environment, based on the authors' previous experience on implementing the Proteum Family tools, aiming at promoting the technology transfer to industry and providing the basis for improving the Muta-Pro process itself.

Testing every member of a product line individually is often impracticable due to large number of possible product configurations. Thus, feature models are frequently used to generate samples, i.e., subsets of product configurations under test. Besides the extensively studied combinatorial interaction testing (CIT) approach for coverage-driven sample generation, only few approaches exist so far adopting mutation testing to emulate faults in feature models to be detected by a sample. In this paper, we present a mutation-based sampling framework for fault-based product-line testing. We define a comprehensive catalog of atomic mutation operators on the graphical representation of feature models. This way, we are able (1) to also define complex mutation operators emulating more subtle faults, and (2) to classify operators semantically, e.g., to avoid redundant and equivalent mutants. We further introduce similarity-based mutant selection and higher order mutation strategies to reduce testing efforts. Our implementation is based on the graph transformation engine Henshin and is evaluated concerning effectiveness/efficiency trade-offs.

Traditionally, mutation testing is used as test set generation and/or test evaluation criteria once it is considered a good fault model. This paper uses mutation testing for evaluating an automated static analyzer. Since static analyzers, in general, report a substantial number of false positive warnings, the intention of this study is to define a prioritization approach of static warnings based on their correspondence with mutations. On the other hand, knowing that Mutation Test has a high application cost, another possibility is to try to identify mutations of some specific mutation operators, which an automatic static analyzer is not adequate to detect. Therefore, this information can be used to prioritize the order of incrementally applying mutation operators considering, firstly, those with no correspondence with static warnings. In both cases, contributing to the establishment of incremental strategies on using automatic static analysis or mutation testing or even a combination of them. We used mutation operators as a fault model to evaluate the direct correspondence between mutations and static warnings. The main advantage of using mutation operators is that they generate a large number of programs containing faults of different types, which can be used to decide the ones most probable to be detected by static analyzers. We provide evidences on the correspondence between mutations and some types of static warnings. The results obtained for a set of 19 open-source programs indicate that: 1) static warnings may be prioritized based on their correspondence level with mutations; 2) specific set of mutation operators and their mutations may be prioritized based on their correspondence level with warnings. It is possible to provide an incremental testing strategy aiming at reducing the cost of both static analysis and mutation testing using the correspondence information between these activities/artifacts.

Mutation testing is an effective testing technique to detect faults and improve code quality. However, few practitioners have adopted mutation testing into practice, which raises several questions: Are tests capable of killing mutants useful? What is the main hindrance to adopting mutation testing in practice? Can practitioners really integrate mutation testing into real-world agile development processes? In this paper, we present two major contributions. First, based on our analysis and knowledge of Ruby, we devised eight new mutation operators for Ruby. Second, we applied mutation testing to an industrial Ruby project at Medidata and reported the lessons learned from the study. We confirmed that mutation-adequate tests are useful and could improve code quality from the perspective of practitioners and found long mutation execution time hinders the agile process. In addition, we used an enterprise-level Amazon cloud-computing technique to reduce the computational cost of running mutants. Considering the availability of a mutation testing tool with our suggested features, we argue that mutation testing can be used in practice.

This chapter builds the foundation of software testing techniques by classifying the various testing approaches and testing coverage criteria. It gradually advances in the concepts and process of Mutation Testing and its application areas. Mutation testing has been applied at both the source code level and specification level of the software under test. Mutation testing, when applied to the source code, is named as Program Mutation. Similarly, when applied to the specifications, it is named as Specification Mutation. The relevant Mutation Testing tools available for different programming languages for both program and Specification Mutations are hereby listed. Owing to the high cost incurred in applying Mutation Testing to industrial needs, the on-going endeavors of the researchers in the area are elaborated here. Applying nature-inspired algorithms along with Mutation Testing for data generation/selection/minimization is an upcoming area of research. Search based Mutation Testing (SBMT) applies evolutionary techniques like Genetic Algorithms or other metaheuristic approaches for automating the tasks associated with mutation testing, which otherwise requires a lot of human effort, thus, making it a practical approach. This chapter concludes by giving the seminal recent advancements in the area.

Time-Critical Systems (TCS) play a crucial role in environments where strict timing constraints are essential to ensure reliability and correctness. Mutation Testing (MT) is considered a valuable strategy for quality assurance of TCS, but it suffers from the equivalent mutant problem, which is known to increase computational cost and reduce confidence in MT. To address this problem, a strong timed bisimulation equivalence (STBE) algorithm can be used when TCS are modeled as Timed Automata (TA). STBE is computationally expensive and can benefit from parallelism. We survey available STBE implementations, identify opportunities to apply parallelism, build an extension that takes advantage of them, and test its effects. The resulting solution is a Java program that receives multiple TAs expressed in UPPAAL format and determines which TAs are equivalent using an STBE implementation such as TimBrCheck or MUTES and process-based parallelism. Compared to existing solutions, our tests show that our proposal is more efficient, reducing the runtimes of STBE by more than half. This could improve the reach, reliability, and effectiveness of MT for TCS.

Mutant reduction based on dominance relation for weak mutation testing

Mutation testing has been neglected by researchers because of the high cost associated with the technique. To manage this issue, researchers have developed cost reduction strategies that aim to reduce the overall cost of mutation, while maintaining the effectiveness and the efficiency of testing. The purpose of this research paper is to present a new cost reduction strategy that cuts the cost of mutation testing through reducing the number of mutation operators used. The experimental part of the paper focuses on the implementation of this strategy on five different java applications. The results of the experiment areused to evaluate the efficiency and quantify the savings of our approach compared to two other existing mutation testing strategies.

Mutation testing in test suite generation using separate bacterial memetic evolutionary algorithm in IoT