AutoCCAG: An Automated Approach to Constrained Covering Array Generation

Researchers have explored the application of combinatorial interaction testing (CIT) methods to construct samples to drive systematic testing of software system configurations. Applying CIT to highly-configurable software systems is complicated by the fact that, in many such systems, there are constraints between specific configuration parameters that render certain combinations invalid. Many CIT algorithms lack a mechanism to avoid these. In recent work, automated constraint solving methods have been combined with search-based CIT construction methods to address the constraint problem with promising results. However, these techniques can incur a non-trivial overhead. In this paper, we build upon our previous work to develop a family of greedy CIT sample generation algorithms that exploit calculations made by modern Boolean satisfiability (SAT) solvers to prune the search space of the CIT problem. We perform a comparative evaluation of the cost-effectiveness of these algorithms on four real-world highly-configurable software systems and on a population of synthetic examples that share the characteristics of those systems. In combination our techniques reduce the cost of CIT in the presence of constraints to 30 percent of the cost of widely-used unconstrained CIT methods without sacrificing the quality of the solutions.

Highly-configurable software systems can be easily adapted to address user's needs. Modelling parameter configurations and their relationships can facilitate software reuse. Combinatorial Interaction Testing (CIT) methods are already often used to drive systematic testing of software system configurations. However, a model of the system's configurations not conforming with respect to its software implementation, must be repaired in order to restore conformance. In this paper we extend CIT by devising a new search-based technique able to repair a model composed of a set of constraints among the various software system's parameters. Our technique can be used to detect and fix faults both in the model and in the real software system. Experiments for five real-world systems show that our approach can repair on average 37% of conformance faults. Moreover, we also show it can infer parameter constraints in a large real-world software system, hence it can be used for automated creation of CIT models.

Testing every member of a product line individually is often impracticable due to large number of possible product configurations. Thus, feature models are frequently used to generate samples, i.e., subsets of product configurations under test. Besides the extensively studied combinatorial interaction testing (CIT) approach for coverage-driven sample generation, only few approaches exist so far adopting mutation testing to emulate faults in feature models to be detected by a sample. In this paper, we present a mutation-based sampling framework for fault-based product-line testing. We define a comprehensive catalog of atomic mutation operators on the graphical representation of feature models. This way, we are able (1) to also define complex mutation operators emulating more subtle faults, and (2) to classify operators semantically, e.g., to avoid redundant and equivalent mutants. We further introduce similarity-based mutant selection and higher order mutation strategies to reduce testing efforts. Our implementation is based on the graph transformation engine Henshin and is evaluated concerning effectiveness/efficiency trade-offs.

Combinatorial interaction testing has been well studied in recent years, and has been widely applied in practice. It generally aims at generating an effective test suite (an interaction test suite) in order to identify faults that are caused by parameter interactions. Due to some constraints in practical applications (e.g. limited testing resources), for example in combinatorial interaction regression testing, prioritized interaction test suites (called interaction test sequences) are often employed. Consequently, many strategies have been proposed to guide the interaction test suite prioritization. It is, therefore, important to be able to evaluate the different interaction test sequences that have been created by different strategies. A well-known metric is the Average Percentage of Combinatorial Coverage (shortly APCCλ), which assesses the rate of interaction coverage of a strength λ (level of interaction among parameters) covered by a given interaction test sequence S. However, APCCλ has two drawbacks: firstly, it has two requirements (that all test cases in S be executed, and that all possible λ-wise parameter value combinations be covered by S); and secondly, it can only use a single strength λ (rather than multiple strengths) to evaluate the interaction test sequence - which means that it is not a comprehensive evaluation. To overcome the first drawback, we propose an enhanced metric Normalized APCCλ (NAPCC) to replace the APCCλ Additionally, to overcome the second drawback, we propose three new metrics: the Average Percentage of Strengths Satisfied (APSS); the Average Percentage of Weighted Multiple Interaction Coverage (APWMIC); and the Normalized APWMIC (NAPWMIC). These metrics comprehensively assess a given interaction test sequence by considering different interaction coverage at different strengths. Empirical studies show that the proposed metrics can be used to distinguish different interaction test sequences, and hence can be used to compare different test prioritization strategies.

Test‐case prioritisation (TCP) attempts to schedule the order of test‐case execution such that faults can be detected as quickly as possible. TCP has been widely applied in many testing scenarios such as regression testing and fault localisation. Abstract test cases (ATCs) are derived from models of the system under test and have been applied to many testing environments such as model‐based testing and combinatorial interaction testing. Although various empirical and analytical comparisons for some ATC prioritisation (ATCP) techniques have been conducted, to the best of the authors’ knowledge, no comparative study focusing on the most current techniques has yet been reported. In this study, they investigated 18 ATCP techniques, categorised into four classes. They conducted a comprehensive empirical study to compare 16 of the 18 ATCP techniques in terms of their testing effectiveness and efficiency. They found that different ATCP techniques could be cost‐effective in different testing scenarios, allowing us to present recommendations and guidelines for which techniques to use under what conditions.

The Traveling-Salesperson-Problem (TSP) is arguably one of the best-known NP-hard combinatorial optimization problems. The two sophisticated heuristic solvers LKH and EAX and respective (restart) variants manage to calculate close-to optimal or even optimal solutions, also for large instances with several thousand nodes in reasonable time. In this work we extend existing benchmarking studies by addressing anytime behaviour of inexact TSP solvers based on empirical runtime distributions leading to an increased understanding of solver behaviour and the respective relation to problem hardness. It turns out that performance ranking of solvers is highly dependent on the focused approximation quality. Insights on intersection points of performances offer huge potential for the construction of hybridized solvers depending on instance features. Moreover, instance features tailored to anytime performance and corresponding performance indicators will highly improve automated algorithm selection models by including comprehensive information on solver quality.

Maximum Satisfiability (MaxSAT) is a general model for formulating combinatorial optimization problems. MaxSAT formulas encoded from different domains have different features, yet most MaxSAT solvers are designed for general formulas. This work considers an important subclass of MaxSAT, named as Pure MaxSAT, which characterizes a wide range of combinatorial optimization problems particularly subset problems. We design a novel local search method for Pure MaxSAT, which combines the idea of linear search and local search and is dubbed as linear local search. Our algorithm LinearLS significantly outperforms state of the art MaxSAT solvers on Pure MaxSAT instances, including instances from MaxSAT Evaluations and those encoded from three famous NP hard combinatorial optimization problems. Moreover, LinearLS outperforms state of the art algorithms for each tested combinatorial optimization problem on the popular benchmarks.

Extensive work on Search-Based Software Testing for Software Product Lines has been published in the last few years. Salient among them is the use of similarity as a surrogate metric for t-wise coverage whenever higher strengths are needed or whenever the size of the test suites is infeasible because of technological or budget limitations. Though promising, this metric has not been assessed with real fault data. In this paper, we address this limitation by using Drupal, a widely used open source web content management system, as an industry-strength case study for which both variability information and fault data have been recently made available. Our preliminary assessment corroborates some of the previous findings but also raises issues on some assumptions and claims made. We hope our work encourages further empirical evaluations of Combinatorial Interaction Testing approaches for Software Product Lines.

System testing of configurable software is an expensive and resource constrained process. Insufficient testing often leads to escaped faults in the field where failures impact customers and are costly to repair. Prior work has shown that it is possible to efficiently sample configurations for testing using combinatorial interaction testing, and to prioritize these configurations to increase the rate of early fault detection. The underlying assumption to date has been that there is no added complexity to configuring a system level environment over a user configurable one; i.e. the time required to setup and test each individual configuration is nominal. In this paper we examine prioritization of system configurable software driven not only by fault detection but also by the cost of configuration and setup time that moving between different configurations incurs. We present a case study on two releases of an enterprise software system using failures reported in the field. We examine the most effective prioritization technique and conclude that (1) using failure history of configurations can improve the early fault detection rate, but that (2) we must consider fault detection rate over time, not by the number of configurations tested. It is better to test related configurations which incur minimal setup time than to test fewer, more diverse configurations.

Vertex coloring is an assignment of colors to vertex of an undirected graph such that no two vertices sharing the same edge have the same color. The vertex coloring problem is to find the minimum number of colors necessary to color a graph given, which is an NP-hard problem in combinatorial optimization. Ant Colony Optimization (ACO) is a well-known meta-heuristic in which a colony of artificial ants cooperates in exploring good solutions to a combinatorial optimization problem. Several methods applying ACO to the vertex coloring problem have been proposed. The main contribution of this paper is to propose a GPU implementation to accelerate the computation of the ACO algorithm for the vertex coloring problem. In our implementation, we have considered programming issues of the GPU architecture, such as coalescing access of the global memory, bank conflict of the shared memory, etc. The experimental results show that on NVIDIA GeForce GTX 1080, our implementation for 1000 vertices runs in 2.740s, while the CPU implementation on Intel Core i7-4790 runs in 100.866s. Thus, our GPU implementation attains a speed-up factor of 36.81.

In this Chapter, we review the virtues and limitations of the Hopfield neural network for tackling NP-hard combinatorial optimization problems (COPs). Then we discuss two new neural network models based on the noisy chaotic neural network, and applied the two methods to solving two different NP-hard COPs in communication networks. The simulation results show that our methods are superior to previous methods in solution quality. We also point out several future challenges and possible directions in this domain.

The objective of this paper is to develop a controller-centric combinatorial wrap-around interaction testing methodology for a stateful path computation element (PCE)-based transport network architecture. By exploiting the internal contexts of the controller-centric network in conjunction with combinatorial, wrap-around, and interaction testing methodologies, the proposed testing methodology helps test engineers build a highly configurable testing environment, select high-quality test cases to obtain the best possible combination of interactions, and prune fault cases or useless cases from all possible test cases throughout the entire development process as gray-box testing. The experimental results verify that the combined testing methodology effectively evaluates the controller-centric network for all testing processes in a completely controlled environment.

The 0-1 knapsack problem (01-KP) is a NP-hard combinatorial optimization problems (COPs) with several applications. Because of its non-convexity, its search space contains several local and/or global optimum solutions. In this situation, both the classical and metaheuristic global optimization approaches often failed to locate the optimal solution to the 01-KP. Therefore, this research develops a clustering-based niching (CBN) method for maintaining and locating multiple solutions within an optimization run, increasing the possibility of locating the global optimal solution to the 01-KP. To do this, CBN method divides the population individuals into a number of clusters (similar to niches in a biology or ecology system) by measuring the Hamming distance between individuals. During the optimization, the individuals in the formed clusters independently explore different regions of the combinatorial search space in order to find promising solutions. For simplicity, the proposed CBN method is implemented using a modified binary PSO (lbest-BPSO) method. The numerical results show that the CBN method is superior than the well-known metaheuristic methods on the basis of solution quality, with a better success rate and average function evaluations, over widely used sets of 0-1 knapsack instance. This guarantees that CBN method is a suitable optimization method for determining the optimal solution to combinatorial optimization problems.

The Traveling Salesman Problem (TSP) is one of the most well-known NP-hard combinatorial optimization problems, widely studied due to its numerous industrial applications in logistics, supply chain management, and resource planning. Over the years, various metaheuristic algorithms have been developed to provide near-optimal solutions within a reasonable computational time. While traditional methods such as Genetic Algorithms (GA) and Simulated Annealing (SA) have been applied widely to the TSP, recent developments in metaheuristics have introduced new methods that can ensure improved performance. In this study, we compare three new metaheuristics: Arctic Puffin Optimization (APO), Human Evolutionary Optimization (HEO), and Ship Rescue Optimization (SRO). These methods, inspired by puffin's behavior, human evolutions, and naval rescue, respectively, have demonstrated high optimization capability in various areas. However, these have not been extensively tried on solving the TSP yet. This research seeks to establish the performance of these algorithms by comparing their convergence rate, solution quality, and computational efficiency with well-established metaheuristics. The results of the experiment expose the strengths and weaknesses of such innovative approaches, showing their potential as competitive alternatives to solve complex combinatorial optimization problems

A survey of approaches for university course timetabling problem