A New Algorithm for Post-Processing Covering Arrays

A greedy-metaheuristic 3-stage approach to construct covering arrays

The construction of covering arrays, the combinatorial structures underlying combinatorial test suites, is a highly researched topic. In previous works, various metaheuristic algorithms, such as Simulated Annealing and Tabu Search, were used to successfully construct covering arrays with a small number of rows. In this paper, we propose for the first time a quantum-inspired evolutionary algorithm for covering array generation. For this purpose, we introduce a simpler and more natural qubit representation as well as new rotation and mutation operators. We implemented different versions of our algorithm employing the different operators. We evaluate the different implementations against selected (optimal) covering array instances.

Covering arrays have become a key piece in Combinatorial Testing. In particular, we focus on the efficient construction of Covering Arrays with Constraints of high strength. SAT solving technology has been proven to be well suited when solving Covering Arrays with Constraints. However, the size of the SAT reformulations rapidly grows up with higher strengths. To this end, we present a new incomplete algorithm that mitigates substantially memory blow-ups. The experimental results confirm the goodness of the approach, opening avenues for new practical applications.

Covering array EXtender

Covering arrays are combinatorial structures, that can be considered generalizations of orthogonal arrays and find application in the field of automated software testing amongst others. The construction of covering arrays is a highly researched topic, with existing works focusing on heuristic, metaheuristic and combinatorial algorithms to successfully construct covering arrays with a small number of rows. In this paper, we introduce the IPO-Q algorithm which combines a recently introduced quantum-inspired evolutionary algorithm with the widely used in-parameter order (IPO) strategy for covering array generation. We implemented different versions of this algorithm and evaluate them, by means of selected covering array instances, against each other and against an algorithm implementing the IPO strategy.

Test engineers are often faced with the challenge of selecting test cases that maximize the chance of discovering faults while working within a limited budget. Combinatorial testing is an effective test case selection strategy to address this challenge. The basic idea is to select test cases that ensure that all possible combinations of settings from two (or more) inputs are accounted for, regardless of which subset of inputs are selected. This is accomplished by using a covering array as the test case selection mechanism. However, for any given testing scenario, there are usually several alternative covering arrays that may satisfy budgetary and coverage requirements. Unfortunately, it is often unclear how to choose from these alternatives. Practitioners often want a way to explore how the input space is being covered to decide which alternative is best suited for the testing scenario. In this paper we propose a graphical method that may be used to evaluate binary covering arrays. The method allows practitioners to visualize input space coverage for inputs of interest. We illustrate the method with several examples.

Software testing is required to verify and validate systems. Combinatorial testing in one of the significant testing techniques. Design and select test cases for combinatorial testing considered as combinatorial problem. Even though there are some existing optimization algorithm based combinatorial testing strategies, that minimize the number of test cases, but most of these strategies based on one-test-at-a-time (OTAT) approach and none of them has adopted the one-parameter-at-a-time (OPAT) approach. Therefore, this paper will propose a new OPAT strategy based on Harmony Search Algorithm (HS) called OPAT-HS. OPAT-HS was originally designed only to support Covering Array (CA) and Mixed Covering Array (MCA) for uniform interaction strength. The result obtained in the experiments appears that OPAT-HS is always best at configurations with MCA notations.

Covering arrays (CAs) are interesting objects in combinatorics and they also play an important role in software testing. It is a challenging task to generate small CAs automatically and efficiently. In this paper, we propose a new approach which generates a CA column by column. A kind of balancing heuristic is adopted to guide the searching procedure. At each step (column extension), some pseudo Boolean constraints are generated and solved by a PBO solver. A prototype tool is implemented, which turns out to be able to find smaller CAs than other tools, for some cases.

A covering array CA(N;t, k, v) is an N × k array with entries in {1,2,…, v}, for which every N × t subarray contains each t-tuple of {1,2,…, v} t among its rows. Covering arrays find application in interaction testing, including software and hardware testing, advanced materials development, and biological systems. A central question is to determine or bound CAN(t, k, v), the minimum number N of rows of a CA(N;t, k, v). The well known bound CAN(t, k, v) = O((t − 1)v t log k) is not too far from being asymptotically optimal. Sensible relaxations of the covering requirement arise when (1) the set {1,2,…, v} t need only be contained among the rows of at least $(1-\epsilon )\binom {k}{t}$ of the N × t subarrays and (2) the rows of every N × t subarray need only contain a (large) subset of {1,2,…, v} t . In this paper, using probabilistic methods, significant improvements on the covering array upper bound are established for both relaxations, and for the conjunction of the two. In each case, a randomized algorithm constructs such arrays in expected polynomial time.

A covering array CA(N;t,k,v) is an N × k array such that every N × t sub‐array contains all t‐tuples from v symbols at least once, where t is the strength of the array. Covering arrays are used to generate software test suites to cover all t‐sets of component interactions. We introduce a combinatorial technique for their construction, focussing on covering arrays of strength 3 and 4. With a computer search, covering arrays with improved parameters have been found. © 2005 Wiley Periodicals, Inc. J Combin Designs 14: 202–213, 2006

Covering Arrays(CA) are combinatorial objects that have been used succesfully to automate the generation of test cases for software testing. The CAs have the features of being of minimal cardinality (i.e. minimize the number of test cases), and maximum coverage (i.e. they guarantee to cover all combinations of certain size between the input parameters). Only in few cases there is known an optimal solution to construct CAs, but in general the problem of constructing optimal CAs is a hard combinatorial optimization problem. For this reason, a number of methods to solve the construction of covering arrays have been developed. This paper gives a survey of the state of the art of the methods to construct covering arrays. The methods analyzed were grouped in four categories: exact methods (Section II), greedy methods (Section III), metaheuristic methods (Section IV), and algebraic methods (Section V). The paper ends with a summary of the methods analyzed.

<p>In combinatorial testing development, the fabrication of covering arrays is the key challenge by the multiple aspects that influence it. A wide range of combinatorial problems can be solved using metaheuristic and greedy techniques. Combining the greedy technique utilizing a metaheuristic search technique like hill climbing (HC), can produce feasible results for combinatorial tests. Methods based on metaheuristics are used to deal with tuples that may be left after redundancy using greedy strategies; then the result utilization is assured to be near-optimal using a metaheuristic algorithm. As a result, the use of both greedy and HC algorithms in a single test generation system is a good candidate if constructed correctly. This study presents a hybrid greedy hill climbing algorithm (HGHC) that ensures both effectiveness and near-optimal results for generating a small number of test data. To make certain that the suggested HGHC outperforms the most used techniques in terms of test size. It is compared to others in order to determine its effectiveness. In contrast to recent practices utilized for the production of covering arrays (CAs) and mixed covering arrays (MCAs), this hybrid strategy is superior since allowing it to provide the utmost outcome while reducing the size and limit the loss of unique pairings in the CA/MCA generation.</p>

Testing is considered as indispensable part in the development process life cycle of any software. Testing is performed to improve the performance, quality and reliability of the software. Designing of relevant test cases plays a major role in deciding the quality of testing. Hence test case designing activity is crucial for the success of testing. Combinatorial strategies are one of those most extensively used methods for generating test cases. These strategies detect failures which are caused by interactions of parameters in the Software under Test(SUT) and by using some sampling mechanisms a covering array test suite is generated. With increase in the number of parameter coverage, the t-way test size sets also increase exponentially which would result in explosion problem. This paper presents a technique to generate minimal test suites in Combinatorial testing(CT). Optimization technique used is Greedy computation strategy for mutation and selection of test cases, used in selecting the locally worst test case for elimination. That is the test case that does not cover maximum number of uncovered interactions from a pool of candidate test cases. Additionally, Branch and bound technique is used for cross over. It is used to branch for the possibilities of number of test cases as cross over inputs. If a particular test case is not possible the bounding happens, else branched for further selection of test cases as input. Experimental results show that Recast Ipog-D generates better/comparable results as compared to the existing algorithm. Additionally, Recast IPOG-D has also contributed to enhance many known Variable Strength Covering array(VSCA) that exist in literature.

Generating combinatorial test sets for complex systems, where the input model contains hundreds or thousands of parameters is a very challenging task. In [1], we developed a new approach that performs this task very effectively by modifying the popular In-Parameter-Order (IPO) strategy to generate a combinatorial structure called Covering Perfect Hash Families (CPHFs), which can be seen as an efficient packing of certain families of Covering Arrays (CAs). The introduced methods not only manage to generate CAs with a large number of columns within a short amount of time, in many cases the produced CAs are smaller than the previously best known ones.

This systematic literature review (SLR) investigates search-based strategies for generating combinatorial test suites using covering arrays (CAs) to efficiently test system interactions. Conducted following PRISMA guidelines, the review analyzes 91 primary studies published between 2003 and 2025, selected through a rigorous process from major academic databases. The identified strategies are categorized into five types: standard, mix, adaptive, hybrid, and hyper-heuristic, based on their underlying algorithmic approaches, including swarm intelligence, evolutionary algorithms, and hyper-heuristic techniques. Each strategy is examined in depth, evaluating its effectiveness in generating high-quality combinatorial test suites. The review also highlights challenges in applying these strategies to varying software testing scenarios. Based on the findings, it provides practical insights to enhance their application and effectiveness in real-world contexts. This work supports broader adoption of search-based testing to improve software quality and reduce defect rates.