A Logic-Based Approach to Combinatorial Testing with Constraints

Testing is considered as indispensable part in the development process life cycle of any software. Testing is performed to improve the performance, quality and reliability of the software. Designing of relevant test cases plays a major role in deciding the quality of testing. Hence test case designing activity is crucial for the success of testing. Combinatorial strategies are one of those most extensively used methods for generating test cases. These strategies detect failures which are caused by interactions of parameters in the Software under Test(SUT) and by using some sampling mechanisms a covering array test suite is generated. With increase in the number of parameter coverage, the t-way test size sets also increase exponentially which would result in explosion problem. This paper presents a technique to generate minimal test suites in Combinatorial testing(CT). Optimization technique used is Greedy computation strategy for mutation and selection of test cases, used in selecting the locally worst test case for elimination. That is the test case that does not cover maximum number of uncovered interactions from a pool of candidate test cases. Additionally, Branch and bound technique is used for cross over. It is used to branch for the possibilities of number of test cases as cross over inputs. If a particular test case is not possible the bounding happens, else branched for further selection of test cases as input. Experimental results show that Recast Ipog-D generates better/comparable results as compared to the existing algorithm. Additionally, Recast IPOG-D has also contributed to enhance many known Variable Strength Covering array(VSCA) that exist in literature.

Testing is an important activity to detect faults before software deployment. We focus on black-box combinatorial testing, where fault detection is one of the main objectives. In this paper, we argue that input model abstraction notably impacts the fault detection capability of a combinatorial test suite. First, we present experiments from previous work that support this argument. We then perform new experiments on a more diverse set of programs. These experiments use mutation testing to estimate fault detection capability, but we also include structural coverage measures in the new experiments. Finally, we elaborate on two possible improvements to obtain an optimal input abstraction strategy for not just continuous but all input domains. Both experiments suggest that input abstraction affects the fault detection capability. We claim that the improvements will produce a better input abstraction with which we can achieve better fault detection capability without increasing the test suite size.

Although combinatorial testing has been widely studied and used, there are still some situations and requirements that combinatorial testing does not apply to well, such as a system under test whose test cases need to be performed contiguously. For thorough testing, the testing requirements are not only to cover all the interactions among factors but also to cover all the value sequences of every factor. Generally, systems under test always involve constraints and dependencies in or among test cases. The constraints among test cases have not been effectively specified. First, we introduce extended covering arrays that can achieve both t-way combinatorial coverage and t-wise sequence coverage, and propose a clocked computation tree logic-based formal specification method for specifying constraints. Then, Particle Swarm Optimization (PSO) based Extended covering array Generator (PEG) is elaborated. To evaluate the constructed test suites, a method for verifying the constraints’ validity is presented, and kernel functions for measuring the coverage are also introduced. Finally, the performance of the proposed PEG is evaluated using several sets of benchmark experiments for some common constraints, and the feasibility and usefulness of PEG is validated.

Combinatorial testing (CT) is a well-known technique for improving the quality of test plans while reducing testing costs. Traditionally, CT is used by testers at testing phase to design a test plan based on a manual definition of the test space. In this work, we extend the traditional use of CT to other parts of the development life cycle. We use CT at early design phase to improve design quality. We also use CT after test cases have been created and executed, in order to find gaps between design and test. For the latter use case we deploy a novel technique for a semi-automated definition of the test space, which significantly reduces the effort associated with manual test space definition. We report on our practical experience in applying CT for these use cases to three large and heavily deployed industrial products. We demonstrate the value gained from extending the use of CT by (1) discovering latent design flaws with high potential impact, and (2) correlating CT-uncovered gaps between design and test with field reported problems.

Combinatorial test case generation faces a problem on how to reduce the test cases by uncover the unnecessary test cases. So, there is a need for expert applications or strategies that generate the most optimum test cases keeping in mind the most important combinations. Complementing existing work on combinatorial test case generation strategies, also known as t-way testing strategies (i.e., where t represents interaction degree), this paper presents the design and the implementation of a new combinatorial test list generation strategy based on Harmony Search (HS) algorithm, called General T-way Harmony Search-based Strategy (GTHS). HS has been chosen to be the main engine of test generation because it could balance between intensification and diversification. Benchmarking experimental results show that GTHS produces competitive results as compared to other existing well-known optimization-based strategies and provides the support for high combination degrees (i.e., t ≤ 12).

The Target Speed Monitor (TSM), which performs the safety critical function of the Chinese Train Control System Level 3 (CTCS-3), is characterized by large parameter input field and fault modes triggered by complicated combinations of parameters. Although combinatorial testing has been successfully applying in testing TSM, redundant test cases are generated because of covering the parameter combinations without interaction, which leads to the problem of high test execution costs. In this paper, an automatic reduction methodology based on decision tree equivalence and test suite output labels is presented. With previous combinatorial test suite generation experiment results of CTCS-3 TSM function, after data preprocessing, some data set have been constructed by test suites and corresponding output labels and they can be used to infer decision trees. After that, the reduction algorithm was used to reduce the data set to realize the test suite reduction. Related experimental results show that this method can achieve a reduction rate of up to 90%, and at the same time, the low-level combination coverage and the fault detection capability of reduced test suites are almost the same as that before the reduction.

Combinatorial testing has lots of test cases, but software testers hope to get the best test coverage with the smallest test case suite. For the scale of produced test cases is so large that researchers have considered the implementation of the critical test cases. This article researches the classic combinatorial test methods and proposes methods to generate pair-wise testing cases with a priority. Firstly, we design formulas to compute the weights of priorities. Secondly, we adopt a greed algorithm to solve the combinatorial testing problems. Furthermore, we integrate the greed strategy into a genetic algorithm to improve the efficiency. It improves the testing efficiency while securing the detection rate of defects under limited resources.

Combinatorial testing (CT) is an important black-box testing method. In CT, the behavior of the system under test (SUT) is affected by several parameters/components. Then CT generates a combinatorial test suite. After the user executes a test suite and starts debugging, some test cases fail and some pass. From the perspective of a black box, the failures are caused by interaction of several parameters. It will be helpful if we can identify a small set of interacting parameters that caused the failures. This paper proposes a new automatic approach to identifying faulty interactions. It uses (pseudo-Boolean) constraint solving and optimization techniques to analyze the execution results of the combinatorial test suite. Experimental results show that the method is quite efficient and it can find faulty combinatorial interactions quickly. They also shed some light on the relation between the size of test suite and the ability of fault localization.

OpenDaylight is an open-source Software-Defined Network controller that has made programmable networks possible. It has got together both industry and the OpenDaylight community in delivering programmable and interoperable networks to service providers, enterprises, universities and organizations. It is a continuously evolving project with many different releases, where every new release requires extensive testing, to ensure quality software. Effective regression testing is required to ensure the existing features are not affected. Also a project of this scale will have many test cases, hence an effective selection method is required to ensure a reduced set of test cases for regression testing. The current paper proposes to adopt Combinatorial testing for generating test suite which has reduced number of test cases and better test coverage. Upon successful execution of the combinatorial test suite, the faults identified needs to be localized to help debug them effectively and in less time. This can be achieved through an efficient fault localization technique.KeywordsCombinatorial testingFault localizationOpenDaylightSoftware Defined Networks

Software Testing is an important aspect of the real time software development process. Software testing always assures the quality of software product. As associated with software testing, there are few very important issues where there is a need to pay attention on it in the process of software development test. These issues are generation of effective test case and test suite as well as optimization of test case and suite while doing testing of software product. The important issue is that testing time of the test case and test suite. It is very much important that after development of software product effective testing should be performed. So to overcome these issues of optimization, we have proposed new approach for test suite optimization using genetic algorithm (GA). Genetic algorithm is evolutionary in nature so it is often used for optimization of problem by researcher. In this paper, our aim is to study various selections methods like tournament selection, rank selection and roulette wheel selection and then we apply this genetic algorithm (GA) on various programs which will generate optimized test suite with parameters like fitness value of test case, test suite and take minimum amount of time for execution after certain preset generation. In this paper our main objectives as per the experimental investigation, we show that tournament selection works very fine as compared to other methods with respect fitness selection of test case and test suites, testing time of test case and test suites as well as number of requirements.

Guest Editorial: Special Section from the 11th International Conference on Quality Software (QSIC 2011)

Combinatorial testing is a practical software testing approach that has been widely used in practice. Most research and applications of such approach focus on N-way combinatorial testing that provides a minimum coverage of all N-way interactions among factor. However, the strengths of different interactions may not be a fixed integer N, but a variable. Therefore, variable strength combinatorial testing approach is necessary in applications. Existing variable strength combinatorial testing, which allows some interactions have a higher strength than others, has a limitation that such higher-strength interactions must be disjoint. To avoid such a limitation, an improved variable strength combinatorial testing approach, which makes a more sufficient consideration on actual interaction relationship, is proposed in this article. Furthermore, two greedy heuristic algorithms are also proposed to generate combinatorial test suite. Compared to some existing algorithms and tools, the proposed algorithms have advantages on both the execution effectiveness and the optimality of generated test suite. Experimental results can prove such advantages.

Database Management System (DBMS) developers have implemented extensive test suites to test their DBMSs. For example, the SQLite test suites contain over 92 million lines of code. Despite these extensive efforts, test suites are not systematically reused across DBMSs, leading to wasted effort. Integration is challenging, as test suites use various test case formats and rely on unstandardized test runner features. We present a unified test suite, SQuaLity, in which we integrated test cases from three widely-used DBMSs, SQLite, PostgreSQL, and DuckDB. In addition, we present an empirical study to determine the potential of reusing these systems' test suites. Our results indicate that reusing test suites is challenging: First, test formats and test runner commands vary widely; for example, SQLite has 4 test runner commands, while MySQL has 112 commands with additional features, to, for example, execute file operations or interact with a shell. Second, while some test suites contain mostly standard-compliant statements (e.g., 99% in SQLite), other test suites mostly test non-standardized functionality (e.g., 31% of statements in the PostgreSQL test suite are nonstandardized). Third, test reuse is complicated by various explicit and implicit dependencies, such as the need to set variables and configurations, certain test cases requiring extensions not present by default, and query results depending on specific clients. Despite the above findings, we have identified 3 crashes, 3 hangs, and multiple compatibility issues across four different DBMSs by executing test suites across DBMSs, indicating the benefits of reuse. Overall, this work represents the first step towards test-case reuse in the context of DBMSs, and we hope that it will inspire follow-up work on this important topic.

Combinatorial testing is a prevailing approach to finding out faults in the extremely complex systems of train operation for railway. While combinatorial testing is applied to Chinese Train Control Systems (CTCS) for revealing faults, some potential faults tend to be arduous to trigger due to the interaction of multiple test cases. For the sake of accurate and efficient fault localization of CTCS under the circumstances of masking effects, a novel method of locating the faults of CTCS using the relationship matrix with frequent terms has been proposed in the paper. To begin with, failed test cases from CTCS were made full use of to calculate the frequentness of failure‐inducing elements for ordered tables. Then, the relationship matrix with frequent terms for CTCS was established to work out the suspiciousness of failure‐inducing combinations by the following relation of failure‐inducing elements in the above tables. Furthermore, an iterative approach to locating failure‐inducing combinations was adopted to obtain the full list of rankings of failure‐inducing combinations, among which top combinations, especially MFS (Minimal Failure‐causing Schema), were prone to discover the faults of CTCS. Finally, the effectiveness and accuracy of the proposed method for CTCS was confirmed on the simulation platform of CTCS for Beijing‐Shenyang High‐speed Rail. Empirical studies suggest that compared to the method of fault forest, the average suspiciousness in the proposed method is 12.17% higher than that and the number of additional test cases are reduced by 31.7% on average, which can produce more precise and efficient results of identifying the faults of CTCS in case of masking effects. The research achievements can provide some references for the safety evaluation and system optimization of CTCS for high‐speed rail. © 2022 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.

A test scenario generation method based on combinatorial testing (CT) and Bayesian Network for autonomous vehicles is proposed in this paper. Firstly, some parameters are selected to describe the test scenarios which are classified according to road types and driving tasks. Then, the constraint sets for the scenarios with forbidden tuples are established to avoid the generated cases do not conform to the reality, in which the construct constraint set (CCS) algorithm is utilized to compute implied constraints. Furthermore, the Bayesian networks is used as the probabilistic models of the scenarios, where the traffic participants are represented as object nodes and the relative relationships between the participants are converted into the network structures. Finally, an improved automatic efficient test case generator (AETG) is developed to generate test cases. By considering both probability and frequency, the select function is designed for determining the values of scenario parameters. And the generation mode can be changed by modifying the weight and target parameters. The effectiveness of the proposed method is evaluated by generating six typical test scenarios. Compared with other algorithms, the numbers of test cases in the sets generated by this method are less and the probability deviations are smaller.