Applying parallelism to a bisimulation algorithm to improve efficiency in software testing of time-critical systems

Mutation testing is a fault‐based technique to test the quality of test suites by inducing artificial syntactic faults or mutants in a source program. However, some mutants have the same semantics as original program and cannot be detected by any test suite input known as equivalent mutants. Equivalent mutant problem (EMP) is undecidable as it requires manual human effort to identify a mutant as equivalent or killable. The constraint‐based testing (CBT) theory suggests the use of mathematical constraints which can help reveal some equivalent mutants using mutant features. In this paper, we consider three metrics of CBT theory, ie, reachability, necessity, and sufficiency to extract feature constraints from mutant programs. Constraints are extracted using program dependency graphs. Other features such as degree of significance, semantic distance, and information entropy of mutants are also extracted to build a binary classification model. Machine learning algorithms such as Random Forest, GBT, and SVM are applied under two application scenarios (split‐project and cross‐project) on ten Java programs to predict equivalent mutants. The analysis of the study demonstrates that that the proposed techniques not only improves the efficiency of the equivalent mutant detection but also reduces the effort required to perform it with small accuracy loss.

The use of Genetic Algorithms in evolution of mutants and test cases offers new possibilities in addressing some of the main problems of mutation testing. Most specifically the problem of equivalent mutant detection, and the problem of the large number of mutants produced. In this paper we describe the above problems in detail and introduce a new methodology based on co-evolutionary search techniques using Genetic Algorithms in order to address them effectively. Co-evolution allows the parallel evolution of mutants and test cases. We discuss the advantages of this approach over other existing mutation testing techniques, showing details of some initial experimental results carried out.KeywordsGenetic AlgorithmMutation OperatorSoftware TestMutation TestingOriginal ProgramThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Mutation analysis is a popular test assessment method. It relies on the mutation score, which indicates how many mutants are revealed by a test suite. Yet, there are mutants whose behaviour is equivalent to the original system, wasting analysis resources and preventing the satisfaction of the full (100%) mutation score. For finite behavioural models, the Equivalent Mutant Problem (EMP) can be addressed through language equivalence of non-deterministic finite automata, which is a well-studied, yet computationally expensive, problem in automata theory. In this paper, we report on our preliminary assessment of a state-of-the-art exact language equivalence tool to handle the EMP against 3 models of size up to 15,000 states on 1170 mutants. We introduce random and mutation-biased simulation heuristics as baselines for comparison. Results show that the exact approach is often more than ten times faster in the weak mutation scenario. For strong mutation, our biased simulations are faster for models larger than 300 states. They can be up to 1,000 times faster while limiting the error of misclassifying non-equivalent mutants as equivalent to 10% on average. We therefore conclude that the approaches can be combined for improved efficiency.

Software Testing is one of the most significant phases within the software development life cycle since software bugs can be costly and traumatic. However, the traditional software testing process is not enough on its own as some undiscovered faults might still exist due to the test cases’ inability to detect all underlying faults. Amidst the various proposed techniques of test suites’ efficiency detection comes mutation testing, one of the most effective approaches as declared by many researchers. Nevertheless, there is not enough research on how well the mutation testing tools adhere to the theory of mutation or how well their mutation operators are performing the tasks they were developed for. This research paper presents an investigative study on two different mutation testing tools for Java programming language, namely PIT and µJava. The study aims to point out the weaknesses and strengths of each tool involved through performing mutation testing on four different open-source Java programs to identify the best mutation tool among them. The study aims to further identify and compare the mutation operators of each tool by calculating the mutation score. That is, the operators’ performance is evaluated with the mutation score, with the presumption that the more prominent the number of killed mutants is, the higher the mutation score, thus the more effective the mutation operator and the affiliated tool.Â

Mutation testing is the fault-based software testing approach that is widely applicable for assessing the effectiveness of a test suite. The test suite effectiveness is measured through artificial seeding of faults into the programs under test. Mutation testing tools (MTTs) are significant enablers of the conversion of mutation testing from the research perspective into the real life and mostly applicable testing process. Without using the automatic MTT, mutation testing can’t be really connected in this present reality and is unrealistic to be acknowledged by the industry. Authors analyze six open-source JAVA-based MTT (Jester, JavaMut, MuJava, Jumble, Judy, and Javalanche). The results are based on the performance of various JAVA programs and two real-life applications. Benchmark comparison among the MTT is presented in terms of mutants, mutation operator, mutation score, and quality output. On the basis of comparative analysis, the performance of each tool is explained with the protocol for finding the appropriate tool among the six MTTs. The results show that the MuJava performs best compared to the others.KeywordsMutation testingFault-based testingAdequacy criterionMutation testing tools

Mutation Testing originated from a classical method for digital circuit testing and today is used at program and specification levels. It can be used either to generate or to assess the quality of test sets. In spite of being very effective in detecting faults, Mutation Testing is usually considered a high cost criterion due to: i) the large number of generated mutants; ii) the time-consuming activity of determining equivalent mutants; and iii) the mutant execution time. Many initiatives aiming at reducing the Mutation Testing application cost have been conducted, most of them addressing one of the drawbacks mentioned above. In this paper, we identify and summarize some of the most relevant researches and results related to Mutation Testing cost reduction, e.g., Constrained-Mutation, Constraint-Based Testing and Bayesian Learning. Moreover, we propose a Mutation Testing process, named Muta-Pro, that synergetically integrates the related approaches and mechanisms. This process is intended to be incremental and tailorable to a specific application domain such as C programs or finite state machine models. The main ideas in this paper are illustrated using a UNIX utility program. This process is being integrated in a Mutation Testing environment, based on the authors' previous experience on implementing the Proteum Family tools, aiming at promoting the technology transfer to industry and providing the basis for improving the Muta-Pro process itself.

The paper is devoted to the mutation testing technique that is widely used when testing different software tools. This paper presents a comparative analysis of two mutation testing tools for Java programs, namely Pitest and /Java. They both allow automatically introducing faults into a software code. The analysis has revealed their pros and cons, as well as specific features of their launch and application. Such comparison can help to select the choose the most appropriate mutation tool or the combination of these tools when testing Java programs. We further discuss how these two tools might be combined and which of the two tools better for the mutation generation.

Subrecursive program schemata I & II: I. Undecidable equivalence problems; II. Decidable equivalence problems

Manual test case generation is an exhaustive and time-consuming process. However, automated test data generation may reduce the efforts and assist in creating an adequate test suite embracing predefined goals. The quality of a test suite depends on its fault-finding behavior. Mutants have been widely accepted for simulating the artificial faults that behave similarly to realistic ones for test data generation. In prior studies, the use of search-based techniques has been extensively reported to enhance the quality of test suites. Symmetry, however, can have a detrimental impact on the dynamics of a search-based algorithm, whose performance strongly depends on breaking the “symmetry” of search space by the evolving population. This study implements an elitist Genetic Algorithm (GA) with an improved fitness function to expose maximum faults while also minimizing the cost of testing by generating less complex and asymmetric test cases. It uses the selective mutation strategy to create low-cost artificial faults that result in a lesser number of redundant and equivalent mutants. For evolution, reproduction operator selection is repeatedly guided by the traces of test execution and mutant detection that decides whether to diversify or intensify the previous population of test cases. An iterative elimination of redundant test cases further minimizes the size of the test suite. This study uses 14 Java programs of significant sizes to validate the efficacy of the proposed approach in comparison to Initial Random tests and a widely used evolutionary framework in academia, namely Evosuite. Empirically, our approach is found to be more stable with significant improvement in the test case efficiency of the optimized test suite.

Aspect-Oriented Programming has introduced new types of software faults that may be systematically tackled with mutation testing. However, such testing approach requires adequate tooling support in order to be properly performed. This paper addresses this issue, introducing a novel tool named Proteum/AJ. Proteum/AJ realises a set of requirements for mutation-based testing tools and overcomes some limitations identified in previous tools for aspect-oriented programs. Through an example, we show how Proteum/AJ was designed to support the main steps of mutation testing. This preliminary use of the tool in a full test cycle provided evidences of the feasibility of using it in real software development processes and helped us to reason about the current functionalities and to identify future needs.

Mutation testing has a widespread reputation of being a rather powerful testing technique. However, its practical application requires the detection of equivalent mutants. Detecting equivalent mutants is cumbersome since it requires manual analysis, resulting in unbearable testing cost. To overcome this difficulty, researchers have proposed the use of mutant classification, an approach that helps isolating equivalent mutants. From this perspective, the present paper establishes and assesses possible mutant classification strategies. The conducted study suggests that while mutant classification can be useful in isolating equivalent mutants, it fails to kill some mutants. Indeed, the experimental results show that the proposed strategies achieve to kill approximately 95% of the introduced killable mutants.

The ratio of equivalent mutants: A key to analyzing mutation equivalence

Mutation testing can be used to measure the quality of a given test suite. But two flaws prevent wide acceptance. First, there are equivalent mutants - mutants that are semantically equivalent to the unmodified version and therefore unkillable. Manually identifying those mutants is time-consuming and error-prone. Second, initially there are often too few test cases. Mutation testing detects missing cases. But it is too time-consuming to manually write all the required test cases. This paper shows how to use symbolic execution to tackle both problems, i.e., to detect equivalent mutants and exclude them from further analysis, and to automatically generate test cases that kill the remaining mutants. Our evaluation uses a set of 252 publicly available mutants for which it is known that they are hard to classify. Despite the fact that detecting equivalent mutants is an undecidable problem in general, our fully automatic tool MutantDistiller correctly classifies all of them (13 equivalent, 239 non-equivalent). MutantDistiller also generates test cases that kill the nonequivalent mutants.

Equivalent mutants are one of the major costs of mutation testing. The undecidable nature of this problem makes a fully automated solution unattainable and necessitates the manual analysis of live mutants. This paper introduces the concept of mirrored mutants, ones that affect similar code fragments. It is argued that mirrored mutants exhibit analogous behavior with respect to their equivalence. Thus, if one of them is equivalent, then the other mirrored mutants should be too. An empirical study, conducted on real world programs, investigates this argument, focusing on both intra-method and inter-method mirrored mutants. The obtained results suggest that mirrored mutants indeed exhibit this kind of behavior and thus can be utilized to ameliorate the adverse effects of the equivalent mutant problem.

Android is an operating system source which offers flexibility and support for most mobile applications, and easy access to social networks. It is important to understand the complexity of design, development, implementation, and testing of Android apps. A number of challenges may be faced in testing android applications, including the lack of testing processes and methods, testing experts being unavailable, poor in-house testing environment, and time restrictions. Mutation testing is a fault-based testing technique, applied by generating mutants and running the application with these mutants to analyze the killed and equivalent mutants. We defined a set of mutation operators according to the features of android applications: apps with content sharing, apps with multimedia, apps with graphics, and apps with user location and maps. We identified 42 mutation operators. In addition, we implemented a new tool, “µ-Android,” which automatically generates mutants and retrieves results to prove the efficiency of the test cases and enable the new operators.