Mutant Execution Cost Reduction: Through MUSIC (Mutant Schema Improved with Extra Code)

The work on mutation testing has attracted a lot of attention during the last decades. Mutation testing is a powerful mechanism to improve the quality of test suites based on the injection of syntactic changes into the code of the original program. Several studies have focused on reducing the high computational cost of applying this technique and increasing its efficiency. Only some of them have tried to do it through the application of genetic algorithms. Genetic algorithms can guide through the generation of a reduced subset of mutants without significant loss of information. In this paper, we analyse recent advances in mutation testing that contribute to reduce the cost associated to this technique and propose to apply them for addressing current drawbacks in Evolutionary Mutation Testing (EMT), a genetic algorithm based technique with promising experimental results so far.

Traditionally, mutation testing is used as test set generation and/or test evaluation criteria once it is considered a good fault model. This paper uses mutation testing for evaluating an automated static analyzer. Since static analyzers, in general, report a substantial number of false positive warnings, the intention of this study is to define a prioritization approach of static warnings based on their correspondence with mutations. On the other hand, knowing that Mutation Test has a high application cost, another possibility is to try to identify mutations of some specific mutation operators, which an automatic static analyzer is not adequate to detect. Therefore, this information can be used to prioritize the order of incrementally applying mutation operators considering, firstly, those with no correspondence with static warnings. In both cases, contributing to the establishment of incremental strategies on using automatic static analysis or mutation testing or even a combination of them. We used mutation operators as a fault model to evaluate the direct correspondence between mutations and static warnings. The main advantage of using mutation operators is that they generate a large number of programs containing faults of different types, which can be used to decide the ones most probable to be detected by static analyzers. We provide evidences on the correspondence between mutations and some types of static warnings. The results obtained for a set of 19 open-source programs indicate that: 1) static warnings may be prioritized based on their correspondence level with mutations; 2) specific set of mutation operators and their mutations may be prioritized based on their correspondence level with warnings. It is possible to provide an incremental testing strategy aiming at reducing the cost of both static analysis and mutation testing using the correspondence information between these activities/artifacts.

One of the main problems to perform the Software Testing is to find a set of tests (subset from input domain of the problem) which is effective to detect the remaining bugs in the software. The Search-Based Software Testing (SBST) approach uses metaheuristics to find low cost set of tests with a high effectiveness to detect bugs. From several existing test criteria, Mutation Testing is considered quite promising to reveal bugs, despite its high computational cost, due to the great quantity of mutant programs generated. Therefore, this paper addresses the problem of selecting mutant programs and test cases in Mutation Testing context. To this end, it is proposed a Coevolutionary Genetic Algorithm (CGA) and the concept of Genetic Effectiveness, describing a new representation and implementing new genetic operators. The CGA is applied in five benchmarks and the results are compared to other five methods, showing a better performance of the proposed algorithm in subsets automatic selection with better mutation score and greater reduction of computational cost, specifically the amount of testing, when compared with exhaustive test.

SummaryWhen applied to mobile software, mutation testing is particularly costly due to the deployment of the app under test onto the device: if one deployment is made for each generated mutant, the execution time becomes unapproachable. This paper analyses how the combination of different cost reduction techniques improves the execution time of mutation testing in mobile apps. The techniques reviewed and combined are mutant schema, parallel execution and two different ways of executing tests against the mutants (all against all and all against mutants remaining alive), as well as greedy algorithm for reducing the test suite size. This paper also presents a mathematical model of cost reduction and checks its validity with several experiments. Furthermore, the exhaustive and long experimentation has led the authors to compile a set of good practices which are also presented in a set of lessons learned.

Mutant reduction based on dominance relation for weak mutation testing

Mutation testing has been neglected by researchers because of the high cost associated with the technique. To manage this issue, researchers have developed cost reduction strategies that aim to reduce the overall cost of mutation, while maintaining the effectiveness and the efficiency of testing. The purpose of this research paper is to present a new cost reduction strategy that cuts the cost of mutation testing through reducing the number of mutation operators used. The experimental part of the paper focuses on the implementation of this strategy on five different java applications. The results of the experiment areused to evaluate the efficiency and quantify the savings of our approach compared to two other existing mutation testing strategies.

Android applications have experienced an outstanding growth in recent years with a projected figure of 378 billion in the number of application downloads by 2020. As a result, verification and validation of Android apps and their graphical user interface through testing is crucial for quality assurance. One way to assess quality assurance is through the use of mutation testing. Mutation testing modifies the code of the System Under Test in small ways in order to produce faulty versions of the program called mutants and compare them to the original version. However, the cost of mutation testing can be expensive and time consuming. We propose a graphical user-friendly Android mutation tool called Edroid whose primary purpose is to mutate Android's main components such as activities, services, content providers and broadcast receivers using the source code of XML files. We introduce fourteen Android mutation operators, where ten of them are new, that are derived from operators used in different fields such as mutation testing for Graphical User Interface and XML schema. We evaluate Edroid on five real world Android applications using testing methodologies such as random testing and black box testing. Results from these experiments demonstrate that Edroid facilitates the generation of mutants. In addition and equally important, the set of proposed mutation operators are effective in revealing errors that improve the quality of a given test suite.

Mutation Testing originated from a classical method for digital circuit testing and today is used at program and specification levels. It can be used either to generate or to assess the quality of test sets. In spite of being very effective in detecting faults, Mutation Testing is usually considered a high cost criterion due to: i) the large number of generated mutants; ii) the time-consuming activity of determining equivalent mutants; and iii) the mutant execution time. Many initiatives aiming at reducing the Mutation Testing application cost have been conducted, most of them addressing one of the drawbacks mentioned above. In this paper, we identify and summarize some of the most relevant researches and results related to Mutation Testing cost reduction, e.g., Constrained-Mutation, Constraint-Based Testing and Bayesian Learning. Moreover, we propose a Mutation Testing process, named Muta-Pro, that synergetically integrates the related approaches and mechanisms. This process is intended to be incremental and tailorable to a specific application domain such as C programs or finite state machine models. The main ideas in this paper are illustrated using a UNIX utility program. This process is being integrated in a Mutation Testing environment, based on the authors' previous experience on implementing the Proteum Family tools, aiming at promoting the technology transfer to industry and providing the basis for improving the Muta-Pro process itself.

Mutation testing in test suite generation using separate bacterial memetic evolutionary algorithm in IoT

Mutation testing is an effective testing technique to detect faults and improve code quality. However, few practitioners have adopted mutation testing into practice, which raises several questions: Are tests capable of killing mutants useful? What is the main hindrance to adopting mutation testing in practice? Can practitioners really integrate mutation testing into real-world agile development processes? In this paper, we present two major contributions. First, based on our analysis and knowledge of Ruby, we devised eight new mutation operators for Ruby. Second, we applied mutation testing to an industrial Ruby project at Medidata and reported the lessons learned from the study. We confirmed that mutation-adequate tests are useful and could improve code quality from the perspective of practitioners and found long mutation execution time hinders the agile process. In addition, we used an enterprise-level Amazon cloud-computing technique to reduce the computational cost of running mutants. Considering the availability of a mutation testing tool with our suggested features, we argue that mutation testing can be used in practice.

We use transactional data from the USD and EUR segments of the plain vanilla interest rate swap market to assess the impact of the Dodd-Frank mandate that US persons must trade certain swap contracts on Swap Execution Facilities (SEFs). We find that, as a result of SEF trading, activity increases and liquidity improves across the swap market, with the improvement being largest for USD mandated contracts which are most affected by the mandate. The associated reduction in execution costs is economically significant. For example, execution costs in USD mandated contracts, where SEF penetration is highest, drop, for market end-users alone, by $3 million-$4 million daily relative to EUR mandated contracts and in total by about $7 million-$13 million daily. We also find that inter-dealer activity drops concurrently with the improvement in liquidity suggesting that execution costs may have fallen because dealer intermediation chains became shorter. Finally, we document that the Dodd-Frank mandate caused the activity of the EUR segment of the market to geographically fragment. However, this does not appear to have compromised liquidity. Overall, our results suggest that the improvements in transparency brought about by the Dodd-Frank trading mandate have substantially improved interest rate swap market liquidity.

The high cost of access to space begins with the current availability of very expensive ($3,000 $5,000/lb payload to LEO) launch systems. This paper discusses and compares two distinct alternatives for making significant (nearly order-of-magnitude) reductions in launch costs. An additional benefit to dramatic reductions in the cost of space transportation would also be significant cost reductions of the payloads (i.e., spacecraft) by designing for minimum cost and not for minimum weight and maximum performance. One proposes a reusable single-stage-to-orbit and return system with high development cost but very low recurring costs leading to very low life-cycle costs. The other proposes a very simplified expendable launch vehicle with extremely low both non-recurring and recurring costs, also leading to a very low life-cycle cost and early IOC (with low risk). This paper presents physical and costbenefit arguments which result in recommending that the Ultra Low-Cost Expendable Launch Vehicle (LCELV) concept be seriously considered as the next logical step in achieving cost-effective access to space with minimum fiscal risk. Such an would greatly reduce the cost of U.S. commercial, civil and military space systems and would go a long ways towards restoring the international competitiveness of U.S. Launch Vehicles. 1TRW, Manager, Propulsion and Combustion Center, Space & Technology Division TRW, Technical Consultant 1. Background Among the various goals enunciated in the National Space Transportation Study documents (Reference 1), the statement substantial reduction in space transportation costs must be attained if the nation is to meet the demanding needs of the future is most significant and forms the basis for a redirection in the philosophy of this country should fundamentally develop and procure space hardware. During the Apollo Lunar Landing period (1968-1973) efforts were being initiated to reduce the costs of space transportation. Primary emphasis was placed on reusable systems and studies were undertaken which ultimately resulted in the development of the space shuttle. However, during that same period James Webb, who was administrator of NASA, asked the question how cheaply can build expendable launch vehicles? This question was raised because some people felt that reuse of the launch vehicle might introduce additional costs due to the demanding requirements for more durable hardware, retrieval, and rapid turnaround refurbishment. The basic concern embodied in this minority position was that a reusable vehicle might not be as cost effective as an expendable system designed for minimum cost. It must be remembered that the stable of expendable launch vehicles existing at that time was designed for minimum weight and maximum performance with costs being a derived or resulting quantity rather than minimum cost being a primary design criterion. Even in studies being conducted today, some people still feel that minimum weight and advanced equates to minimum cost. This is not the of design-for-minimum cost which guides the design to use whatever weight or gives minimum cost. In any event, the decision was made in the late 1960's to proceed with a reusable system because it was felt that a shuttle would be even more economical than the simplified expendable launch vehicle on a life-cycle basis and was better operationally when manned by astronauts. Most steadfast believers in the economics of reuse often cite the fact that we don't throw away airplanes after one use. However, the analogy with an airplane is not totally valid. The structure and control surfaces 1048 American Institute of Aeronautics and Astronautics of an airplane are always necessary and critical for an airplane mission during its entire flight operation. A launch vehicle spends less than five percent of its flight time in the atmosphere; hence, winged structures are not needed. The tanks of a simplified expendable launch vehicle merely contain the propellant and can be very inexpensive, hence expendable. In any case, the operational history of the shuttle indicates that reusability has not shown itself to be cost effective, as of this date. 2. Performance Considerations Today there is an urgent need to provide timely access to space at minimum cost. Considerations are being given to both nearterm solutions as well as long-term goals. Terms such as evolutionary approach to leap-frog technology have been used by various proponents. The various launch vehicle candidates can be grouped into four classes with the following general rationale for each: a. Low-Cost Expendable Launch Vehicle (LCELV)-See Figure 1) The emphasis here is to design the simplest minimum cost design with little reliance on advanced technology. If the design is simple, development costs will be low as well as manufacturing and operating costs. b. Fully Reusable (including Single-Stage to Orbit) (SSTO-R, TSTO-R) (See Figure 2) Development costs (and hence risk) will be high, but if done properly, recurring costs should be very low (if not the lowest). This was the thesis of the partially reusable shuttle, which has not proven itself. Proponents feel that with proper use of advanced propulsion systems and materials, this can be achieved. c. National Launch System (formerly ALS, STAS, Spacelifter) This puts great emphasis on improvements in infrastructure, reduction in operations and paperwork costs and reliance on advanced and TQM. These involve relaxing performance requirements (i.e., STME vs. SSME) while still using LOX-Hydrogen. The concepts are higher-performance oriented than the Low-Cost Expendable Launch Vehicles. This is currently on hold, as an option, because of the lack of funding and appears to be dead. d. Interim Efficiency Improvements of Existing Systems (EELV) see Figure 3) The idea here is to spend money to improve the current systems, operationally, until new systems are rendered operational, or to continue with these improved systems indefinitely (since development of new systems may prove too costly). These systems (Thor, Atlas and Titan derivatives) have been evolving over the past 35 or more years to more complicated systems and one certainly might wonder many improvements are left to evolve and at what cost effectiveness? Within each of these classes there exists a minimum-cost design option. Among all of the classes there may be a minimum-cost concept. Options a) and b) have clear-cut differences with respect to performance requirements and are discussed in the following analysis. Single-stage to earth orbit and return, for reuse, appears desirable from an operational and logistics basis, but demands minimum weight of all inert components such as tanks, structure, engines, etc., because all of these items are accelerated to orbital speed and must later return to earth. At the same time, propellant combinations such as LOXHydrogen, are used to achieve high propulsion performance (high Isp). Because of the current state-of-the art in materials and propellants, the payload percentage efficiency will be very low and requires significant and costly efforts to develop designs based upon very lightweight inert materials, which must still survive multiple reuse without major refurbishing costs or failures. When go to multiple-staged vehicles are getting rid of inert material such as tanks, structure, engines, etc., in a quantized way as use up propellant. In other words, get rid of inerts before accelerate them unnecessarily to higher speeds. The most efficient way would 1049 American Institute of Aeronautics and Astronautics

Semantic mutation testing

Test suite reduction techniques decrease the cost of software testing by removing the redundant test cases from the test suite while still producing a reduced set of tests that yields the same level of code coverage as the original suite. Most of the existing approaches to reduction aim to decrease the size of the test suite. Yet, the difference in the execution cost of the tests is often significant and it may be costly to use a test suite consisting of a few long-running test cases. Thus, this paper proposes an algorithm, based on the concept of test irreplaceability, which creates a reduced test suite with a decreased execution cost. Leveraging widely used benchmark programs, the empirical study shows that, in comparison to existing techniques, the presented algorithm is the most effective at reducing the cost of running a test suite.

Mutation testing is a fault based testing technique which examines the adequacy of test cases by generating faulty programs, called mutants, to be run against them. However, one of the pitfalls of mutation testing is its high computational cost, rendering it impractical for industry use. In this survey, we study mutation generation techniques, cost reduction methods and test-data generation approaches found in the literature, before presenting some tools and applications for mutation testing.