On the Effectiveness of Combinatorial Interaction Testing: A Case Study

Combinatorial interaction testing (CIT) is an important technique for testing highly configurable software systems with demonstrated effectiveness in practice. The goal of CIT is to generate test cases covering the interactions of configuration options, under certain hard constraints. In this context, constrained covering arrays (CCAs) are frequently used as test cases in CIT. Constrained Covering Array Generation (CCAG) is an NP-hard combinatorial optimization problem, solving which requires an effective method for generating small CCAs. In particular, effectively solving t-way CCAG with t>=4 is even more challenging. Inspired by the success of automated algorithm configuration and automated algorithm selection in solving combinatorial optimization problems, in this paper, we investigate the efficacy of automated algorithm configuration and automated algorithm selection for the CCAG problem, and propose a novel, automated CCAG approach called AutoCCAG. Extensive experiments on public benchmarks show that AutoCCAG can find much smaller-sized CCAs than current state-of-the-art approaches, indicating the effectiveness of AutoCCAG. More encouragingly, to our best knowledge, our paper reports the first results for CCAG with a high coverage strength (i.e., 5-way CCAG) on public benchmarks. Our results demonstrate that AutoCCAG can bring considerable benefits in testing highly configurable software systems.

System testing of configurable software is an expensive and resource constrained process. Insufficient testing often leads to escaped faults in the field where failures impact customers and are costly to repair. Prior work has shown that it is possible to efficiently sample configurations for testing using combinatorial interaction testing, and to prioritize these configurations to increase the rate of early fault detection. The underlying assumption to date has been that there is no added complexity to configuring a system level environment over a user configurable one; i.e. the time required to setup and test each individual configuration is nominal. In this paper we examine prioritization of system configurable software driven not only by fault detection but also by the cost of configuration and setup time that moving between different configurations incurs. We present a case study on two releases of an enterprise software system using failures reported in the field. We examine the most effective prioritization technique and conclude that (1) using failure history of configurations can improve the early fault detection rate, but that (2) we must consider fault detection rate over time, not by the number of configurations tested. It is better to test related configurations which incur minimal setup time than to test fewer, more diverse configurations.

Testing every member of a product line individually is often impracticable due to large number of possible product configurations. Thus, feature models are frequently used to generate samples, i.e., subsets of product configurations under test. Besides the extensively studied combinatorial interaction testing (CIT) approach for coverage-driven sample generation, only few approaches exist so far adopting mutation testing to emulate faults in feature models to be detected by a sample. In this paper, we present a mutation-based sampling framework for fault-based product-line testing. We define a comprehensive catalog of atomic mutation operators on the graphical representation of feature models. This way, we are able (1) to also define complex mutation operators emulating more subtle faults, and (2) to classify operators semantically, e.g., to avoid redundant and equivalent mutants. We further introduce similarity-based mutant selection and higher order mutation strategies to reduce testing efforts. Our implementation is based on the graph transformation engine Henshin and is evaluated concerning effectiveness/efficiency trade-offs.

Extensive work on Search-Based Software Testing for Software Product Lines has been published in the last few years. Salient among them is the use of similarity as a surrogate metric for t-wise coverage whenever higher strengths are needed or whenever the size of the test suites is infeasible because of technological or budget limitations. Though promising, this metric has not been assessed with real fault data. In this paper, we address this limitation by using Drupal, a widely used open source web content management system, as an industry-strength case study for which both variability information and fault data have been recently made available. Our preliminary assessment corroborates some of the previous findings but also raises issues on some assumptions and claims made. We hope our work encourages further empirical evaluations of Combinatorial Interaction Testing approaches for Software Product Lines.

Test Case Generation for Mutation-based Testing of Timeliness

In model-based mutation testing, a test model is mutated for test case generation. The resulting test cases are able to detect whether the faults in the mutated models have been implemented in the system under test. For this purpose, a conformance check between the original and the mutated model is required. We have developed an approach for conformance checking of action systems, which are well-suited to specify reactive and non-deterministic systems. We rely on constraint solving techniques. Both, the conformance relation and the transition relation are encoded as constraint satisfaction problems. Earlier results showed the potential of our constraint-based approach to outperform explicit conformance checking techniques, which often face state space explosion. In this work, we go one step further and show optimisations that really boost our performance. In our experiments, we could reduce our runtimes by 80%.

SUMMARYCombinatorial interaction testing (CIT) is a testing technique that requires covering all t‐sized tuples of values out of n parameter attributes or properties modelled after the input parameters or the configuration domain of a system under test. CIT test suites have shown to be very effective in software testing already at pairwise (t = 2) level, and the effectiveness of CIT grows with the tuple width t. Unfortunately, the number of tuples to be tested also does grow. In order to reduce the testing effort, researchers addressed the issue of computing minimal‐sized CIT test suites with effective and scalable algorithms. However, still very few generally applicable t‐wise covering construction algorithms (and tools) do exist in literature. This paper presents an original greedy algorithm to compute t‐wise covering mixed covering arrays with constant space complexity, irrespective of the number of involved parameters and strength of interaction. The proposed algorithm has been implemented in a prototype tool, featuring also support for user constraints over the inputs. Assessment of the tool performance on a set of large, real‐world test systems is reported, with results encouraging its adoption in industrial production environments. Copyright © 2011 John Wiley & Sons, Ltd.

Software Testing is one of the most significant phases within the software development life cycle since software bugs can be costly and traumatic. However, the traditional software testing process is not enough on its own as some undiscovered faults might still exist due to the test cases’ inability to detect all underlying faults. Amidst the various proposed techniques of test suites’ efficiency detection comes mutation testing, one of the most effective approaches as declared by many researchers. Nevertheless, there is not enough research on how well the mutation testing tools adhere to the theory of mutation or how well their mutation operators are performing the tasks they were developed for. This research paper presents an investigative study on two different mutation testing tools for Java programming language, namely PIT and µJava. The study aims to point out the weaknesses and strengths of each tool involved through performing mutation testing on four different open-source Java programs to identify the best mutation tool among them. The study aims to further identify and compare the mutation operators of each tool by calculating the mutation score. That is, the operators’ performance is evaluated with the mutation score, with the presumption that the more prominent the number of killed mutants is, the higher the mutation score, thus the more effective the mutation operator and the affiliated tool.Â

Mutation testing is an effective testing technique to detect faults and improve code quality. However, few practitioners have adopted mutation testing into practice, which raises several questions: Are tests capable of killing mutants useful? What is the main hindrance to adopting mutation testing in practice? Can practitioners really integrate mutation testing into real-world agile development processes? In this paper, we present two major contributions. First, based on our analysis and knowledge of Ruby, we devised eight new mutation operators for Ruby. Second, we applied mutation testing to an industrial Ruby project at Medidata and reported the lessons learned from the study. We confirmed that mutation-adequate tests are useful and could improve code quality from the perspective of practitioners and found long mutation execution time hinders the agile process. In addition, we used an enterprise-level Amazon cloud-computing technique to reduce the computational cost of running mutants. Considering the availability of a mutation testing tool with our suggested features, we argue that mutation testing can be used in practice.

Semantic mutation testing

Introduction: Perfusion abnormalities that have been associated with collateral status include the hypoperfusion intensity ratio (HIR) and reduction of cerebral blood volume within the ischemic lesion (CBV index). We hypothesized that assessment of both of these parameters would predict poor CTA collaterals with greater accuracy than a single parameter. This hypothesis was assessed among patients enrolled in the DEFUSE 3 trial that evaluated the efficacy of thrombectomy in patients with imaging evidence of salvageable tissue in the late 6-16 hour window. Methods: Patients were eligible for this analysis if they had adequate quality CTA and CT perfusion. An expert neuroradiologist assessed collateral status on CTA using the Tan scale. The HIR index (fraction of Tmax>10s delay within the Tmax>6s lesion) and the CBV index (mean CBV within the Tmax>6s lesion relative to normal CBV) were automatically calculated by RAPID software. ROC curve analysis was used to identify thresholds for both indices that predicted presence of poor collaterals on CTA. Proportions were compared using χ 2 and McNemar tests. Results: Of the 128 cases eligible for this analysis, 26% had poor collaterals on CTA. HIR ≥0.39 and CBV index ≤0.77 were the optimal thresholds for predicting poor collaterals on CTA. Good collaterals were present on CTA in 87% when neither of the perfusion indices were unfavorable, 77% when a single index was unfavorable, and 54% when both were unfavorable (p=0.001 for none vs both and p=0.029 for single vs both). Sensitivity for predicting poor CTA collaterals was 0.70 (95% CI 0.51-0.84) with HIR ≥0.39, 0.61 (0.42-0.77) with CBV index ≤0.77, and 0.82 (0.64-0.92) for the combined criteria of ‘either index unfavorable’ (p=0.016 for ‘either’ criteria vs CBV index alone). The presence of unfavorable criteria on both indices produced a higher specificity of 0.80 (0.70-0.87) vs. HIR alone 0.59 (0.48-0.69) or CBV alone 0.62 (0.52-0.72), both p<0.001. Conclusion: The combination of 2 fully automated perfusion parameters were more accurate for predicting CTA collaterals than a single perfusion parameter in the DEFUSE 3 population. Automated perfusion parameters do not require an expert neuroradiologist for interpretation and have no concerns regarding inter-rater agreement.

Mutant generation for embedded systems using kernel-based software and hardware fault simulation

Mutation Testing originated from a classical method for digital circuit testing and today is used at program and specification levels. It can be used either to generate or to assess the quality of test sets. In spite of being very effective in detecting faults, Mutation Testing is usually considered a high cost criterion due to: i) the large number of generated mutants; ii) the time-consuming activity of determining equivalent mutants; and iii) the mutant execution time. Many initiatives aiming at reducing the Mutation Testing application cost have been conducted, most of them addressing one of the drawbacks mentioned above. In this paper, we identify and summarize some of the most relevant researches and results related to Mutation Testing cost reduction, e.g., Constrained-Mutation, Constraint-Based Testing and Bayesian Learning. Moreover, we propose a Mutation Testing process, named Muta-Pro, that synergetically integrates the related approaches and mechanisms. This process is intended to be incremental and tailorable to a specific application domain such as C programs or finite state machine models. The main ideas in this paper are illustrated using a UNIX utility program. This process is being integrated in a Mutation Testing environment, based on the authors' previous experience on implementing the Proteum Family tools, aiming at promoting the technology transfer to industry and providing the basis for improving the Muta-Pro process itself.

This paper used both methods of single parameter and Multi-parameter combination sensitivity analysis in groundwater numerical model parameter sensitivity analysis. With the examples of Jiaozuo, Henan province, choosing coefficient of permeability, porosity and longitudinal dispersion degree as factor, the paper analysis the main controlling factors of contaminant migration of Hexavalent chromium in groundwater. Single parameter sensitivity analysis results show that the permeability coefficient gives largest impact to the migration of hexavalent chromium. It is positively correlates with groundwater numerical model. The rest of the parameters assume the inverse correlation. Multi-parameter combination sensitivity analysis results show that the permeability coefficient has absolute advantage among various parameter combinations. Another conclusion is that longitudinal dispersion degree affects smaller on the process of hexavalent chromium pollutants migration, but the impact on the final density is bigger.

This chapter builds the foundation of software testing techniques by classifying the various testing approaches and testing coverage criteria. It gradually advances in the concepts and process of Mutation Testing and its application areas. Mutation testing has been applied at both the source code level and specification level of the software under test. Mutation testing, when applied to the source code, is named as Program Mutation. Similarly, when applied to the specifications, it is named as Specification Mutation. The relevant Mutation Testing tools available for different programming languages for both program and Specification Mutations are hereby listed. Owing to the high cost incurred in applying Mutation Testing to industrial needs, the on-going endeavors of the researchers in the area are elaborated here. Applying nature-inspired algorithms along with Mutation Testing for data generation/selection/minimization is an upcoming area of research. Search based Mutation Testing (SBMT) applies evolutionary techniques like Genetic Algorithms or other metaheuristic approaches for automating the tasks associated with mutation testing, which otherwise requires a lot of human effort, thus, making it a practical approach. This chapter concludes by giving the seminal recent advancements in the area.