Contract-based Mutation Testing in the Refinement Calculus

This paper presents a theory of testing that integrates into Hoare and He’s Unifying Theory of Programming (UTP). We give test cases a denotational semantics by viewing them as specification predicates. This reformulation of test cases allows for relating test cases via refinement to specifications and programs. Having such a refinement order that integrates test cases, we develop a testing theory for fault-based testing. Fault-based testing uses test data designed to demonstrate the absence of a set of pre-specified faults. A well-known fault-based technique is mutation testing. In mutation testing, first, faults are injected into a program by altering (mutating) its source code. Then, test cases that can detect these errors are designed. The assumption is that other faults will be caught, too. In this paper, we apply the mutation technique to both, specifications and programs. Using our theory of testing, two new test case generation laws for detecting injected (anticipated) faults are presented: one is based on the semantic level of UTP design predicates, the other on the algebraic properties of a small programming language.

Mutation testing is an effective testing technique to detect faults and improve code quality. However, few practitioners have adopted mutation testing into practice, which raises several questions: Are tests capable of killing mutants useful? What is the main hindrance to adopting mutation testing in practice? Can practitioners really integrate mutation testing into real-world agile development processes? In this paper, we present two major contributions. First, based on our analysis and knowledge of Ruby, we devised eight new mutation operators for Ruby. Second, we applied mutation testing to an industrial Ruby project at Medidata and reported the lessons learned from the study. We confirmed that mutation-adequate tests are useful and could improve code quality from the perspective of practitioners and found long mutation execution time hinders the agile process. In addition, we used an enterprise-level Amazon cloud-computing technique to reduce the computational cost of running mutants. Considering the availability of a mutation testing tool with our suggested features, we argue that mutation testing can be used in practice.

This chapter builds the foundation of software testing techniques by classifying the various testing approaches and testing coverage criteria. It gradually advances in the concepts and process of Mutation Testing and its application areas. Mutation testing has been applied at both the source code level and specification level of the software under test. Mutation testing, when applied to the source code, is named as Program Mutation. Similarly, when applied to the specifications, it is named as Specification Mutation. The relevant Mutation Testing tools available for different programming languages for both program and Specification Mutations are hereby listed. Owing to the high cost incurred in applying Mutation Testing to industrial needs, the on-going endeavors of the researchers in the area are elaborated here. Applying nature-inspired algorithms along with Mutation Testing for data generation/selection/minimization is an upcoming area of research. Search based Mutation Testing (SBMT) applies evolutionary techniques like Genetic Algorithms or other metaheuristic approaches for automating the tasks associated with mutation testing, which otherwise requires a lot of human effort, thus, making it a practical approach. This chapter concludes by giving the seminal recent advancements in the area.

Traditionally, mutation testing is used as test set generation and/or test evaluation criteria once it is considered a good fault model. This paper uses mutation testing for evaluating an automated static analyzer. Since static analyzers, in general, report a substantial number of false positive warnings, the intention of this study is to define a prioritization approach of static warnings based on their correspondence with mutations. On the other hand, knowing that Mutation Test has a high application cost, another possibility is to try to identify mutations of some specific mutation operators, which an automatic static analyzer is not adequate to detect. Therefore, this information can be used to prioritize the order of incrementally applying mutation operators considering, firstly, those with no correspondence with static warnings. In both cases, contributing to the establishment of incremental strategies on using automatic static analysis or mutation testing or even a combination of them. We used mutation operators as a fault model to evaluate the direct correspondence between mutations and static warnings. The main advantage of using mutation operators is that they generate a large number of programs containing faults of different types, which can be used to decide the ones most probable to be detected by static analyzers. We provide evidences on the correspondence between mutations and some types of static warnings. The results obtained for a set of 19 open-source programs indicate that: 1) static warnings may be prioritized based on their correspondence level with mutations; 2) specific set of mutation operators and their mutations may be prioritized based on their correspondence level with warnings. It is possible to provide an incremental testing strategy aiming at reducing the cost of both static analysis and mutation testing using the correspondence information between these activities/artifacts.

Semantic mutation testing

Testing Deep Learning (DL) systems is a complex task as they do not behave like traditional systems would, notably because of their stochastic nature. Nonetheless, being able to adapt existing testing techniques such as Mutation Testing (MT) to DL settings would greatly improve their potential verifiability. While some efforts have been made to extend MT to the Supervised Learning paradigm, little work has gone into extending it to Reinforcement Learning (RL) which is also an important component of the DL ecosystem but behaves very differently from SL. This paper builds on the existing approach of MT in order to propose a framework, RLMutation, for MT applied to RL. Notably, we use existing taxonomies of faults to build a set of mutation operators relevant to RL and use a simple heuristic to generate test cases for RL. This allows us to compare different mutation killing definitions based on existing approaches, as well as to analyze the behavior of the obtained mutation operators and their potential combinations called Higher Order Mutation(s) (HOM). We show that the design choice of the mutation killing definition can affect whether or not a mutation is killed as well as the generated test cases. Moreover, we found that even with a relatively small number of test cases and operators we manage to generate HOM with interesting properties which can enhance testing capability in RL systems.

Test case minimization approach using fault detection and combinatorial optimization techniques for configuration-aware structural testing

Mutation testing has been neglected by researchers because of the high cost associated with the technique. To manage this issue, researchers have developed cost reduction strategies that aim to reduce the overall cost of mutation, while maintaining the effectiveness and the efficiency of testing. The purpose of this research paper is to present a new cost reduction strategy that cuts the cost of mutation testing through reducing the number of mutation operators used. The experimental part of the paper focuses on the implementation of this strategy on five different java applications. The results of the experiment areused to evaluate the efficiency and quantify the savings of our approach compared to two other existing mutation testing strategies.

Mutant reduction based on dominance relation for weak mutation testing

Mutation testing is a fault based testing technique which examines the adequacy of test cases by generating faulty programs, called mutants, to be run against them. However, one of the pitfalls of mutation testing is its high computational cost, rendering it impractical for industry use. In this survey, we study mutation generation techniques, cost reduction methods and test-data generation approaches found in the literature, before presenting some tools and applications for mutation testing.

IntroductionMalignant pleural effusions are common in non–small cell lung cancer (NSCLC). Molecular testing is among the most critical steps in the management of patients with advanced NSCLC. However, the optimal approach for epidermal growth factor receptor (EGFR) mutation testing in such effusion samples remains unclear.MethodsWe prospectively collected effusion samples from patients with EGFR–mutant NSCLC. Following sample centrifugation, genomic DNA and cell–free DNA were respectively extracted from the sediment and supernatants. EGFR mutation was detected through a real–time PCR assay.ResultsA total of 108 effusions from 78 patients were examined, with 12 and 96 obtained before and after EGFR tyrosine kinase inhibitor treatment, respectively. Carcinoma cells or atypical cells were identified in 73 effusions (67.6%). EGFR mutations were detected in 86 (79.6%) sediment and 84 (77.8%) supernatant samples. Among the effusions with positive cytological findings, the EGFR mutation detection rates were 95.9% (70/73) and 86.3% (63/73) in the sediment and supernatants, respectively. Among the effusions with negative cytological findings, the corresponding detection rates were 45.7% (16/35) and 60% (21/35), respectively. Current clinical practice is to arrange EGFR mutation testing only for sediment from cytologically positive effusions. Through the proposed cytology–based specimen triage, wherein sediment and supernatants with positive and negative cytological findings, respectively, are tested, the detection rate was increased from 64.8% (70/108) to 84.3% (91/108). At half of the cost, this strategy provided a detection rate only slightly lower than the rate in a combined test of all the sediment and supernatants (87.0%, 94/108).ConclusionsThe separate extraction of DNA from sediment and supernatants obtained from centrifuged effusion samples can improve the overall EGFR mutation detection rate. The present cytology–based specimen triage is an efficient strategy for EGFR mutation testing in patients with EGFR–mutant NSCLC.

The paper is devoted to the mutation testing technique that is widely used when testing different software tools. This paper presents a comparative analysis of two mutation testing tools for Java programs, namely Pitest and /Java. They both allow automatically introducing faults into a software code. The analysis has revealed their pros and cons, as well as specific features of their launch and application. Such comparison can help to select the choose the most appropriate mutation tool or the combination of these tools when testing Java programs. We further discuss how these two tools might be combined and which of the two tools better for the mutation generation.

SummaryWhen applied to mobile software, mutation testing is particularly costly due to the deployment of the app under test onto the device: if one deployment is made for each generated mutant, the execution time becomes unapproachable. This paper analyses how the combination of different cost reduction techniques improves the execution time of mutation testing in mobile apps. The techniques reviewed and combined are mutant schema, parallel execution and two different ways of executing tests against the mutants (all against all and all against mutants remaining alive), as well as greedy algorithm for reducing the test suite size. This paper also presents a mathematical model of cost reduction and checks its validity with several experiments. Furthermore, the exhaustive and long experimentation has led the authors to compile a set of good practices which are also presented in a set of lessons learned.

Software Testing is one of the most significant phases within the software development life cycle since software bugs can be costly and traumatic. However, the traditional software testing process is not enough on its own as some undiscovered faults might still exist due to the test cases’ inability to detect all underlying faults. Amidst the various proposed techniques of test suites’ efficiency detection comes mutation testing, one of the most effective approaches as declared by many researchers. Nevertheless, there is not enough research on how well the mutation testing tools adhere to the theory of mutation or how well their mutation operators are performing the tasks they were developed for. This research paper presents an investigative study on two different mutation testing tools for Java programming language, namely PIT and µJava. The study aims to point out the weaknesses and strengths of each tool involved through performing mutation testing on four different open-source Java programs to identify the best mutation tool among them. The study aims to further identify and compare the mutation operators of each tool by calculating the mutation score. That is, the operators’ performance is evaluated with the mutation score, with the presumption that the more prominent the number of killed mutants is, the higher the mutation score, thus the more effective the mutation operator and the affiliated tool.Â

Mutation testing is a powerful software testing technique. However, it is difficult to obtain test data for killing a large number of mutants, especially for hard-to-kill mutants. Mutant branch is formed by an original statement of a program under test and its mutated statement. The true branch of a mutant branch is covered by a test datum, suggesting the corresponding mutant is killed under the criterion of weak mutation testing. This article focuses on efficiently generating test data for a large number of mutant branches. When generating test data using a search-based method, the size of the search domain is a determining factor affecting the search performance. The key observation is that only partial input variables affect whether a mutant will be killed, so the search domain can be reduced by deleting irrelevant variables. Along this line, we first group the mutant branches based on their relevant input variables, followed by formulating a multi-task optimization model of test data generation for the grouped mutant branches, in which the relevant input variables are taken as the decision variables. Finally, a multi-population genetic algorithm with individual sharing is employed to generate test data by multi-tasking. The experiments based on eight programs of various sizes show that removing related variable helps reduce the search domain, and the efficiency of test data generation by grouping and multitasking is improved.