Recast IPOG-D Algorithm with Constraint Handling for Combinatorial Testing

Background: In today’s era, modifications in a software is a common requirement by customers. When changes are made to existing software, re-testing of all the test cases is required to ensure that the newly introduced changes do not have any unwanted effect on the behavior of the software. However, retesting of all the test cases would not only be time consuming but also expensive. Therefore, there is a need for a technique that reduces the number of tests to be performed. Regression testing is one of the ways to reduce the number of test cases. Selection technique is one such method which seeks to identify the test cases that are relevant to some set of recent changes. Objective: It is evident that most of the studies have used different selection techniques and have focused only on one parameter for achieving reduced test suite size without compromising the performance of regression testing. However, to the best of our knowledge, no study has taken two or more parameters of coverage, and/or execution time in a single testing. This paper presents a hybrid technique that combines both regression test selection using slicing technique and minimization of test cases using modified firefly algorithm with combination of parameters coverage and execution time in a single testing. Methods: A hybrid technique has been described that combines both selection and minimization. Selection of test cases is based upon slicing technique while minimization is done using firefly algorithm. Hybrid technique selects and minimizes the test suite using information on statement coverage and execution time. Results: The proposed technique gives 43.33% much superior result as compared to the other hybrid approach in terms of significantly reduced number of test cases. It shows that the resultant test cases were effective enough to cover 100% of the statements, for all the programs. The proposed technique was also tested on four different programs namely Quadratic, Triangle, Next day, Commission respectively for test suite selection and minimization which gave comparatively superior result in terms of reduction (%) in number of test cases required for testing. Conclusion: The combination of parameters used in slicing based approach, reduces the number of test cases making software testing an economical, feasible and time saving option without any fault in the source code. This proposed technique can be used by software practitioners/experts to reduce time, efforts and resources for selection and minimization of test cases.

Test engineers are often faced with the challenge of selecting test cases that maximize the chance of discovering faults while working within a limited budget. Combinatorial testing is an effective test case selection strategy to address this challenge. The basic idea is to select test cases that ensure that all possible combinations of settings from two (or more) inputs are accounted for, regardless of which subset of inputs are selected. This is accomplished by using a covering array as the test case selection mechanism. However, for any given testing scenario, there are usually several alternative covering arrays that may satisfy budgetary and coverage requirements. Unfortunately, it is often unclear how to choose from these alternatives. Practitioners often want a way to explore how the input space is being covered to decide which alternative is best suited for the testing scenario. In this paper we propose a graphical method that may be used to evaluate binary covering arrays. The method allows practitioners to visualize input space coverage for inputs of interest. We illustrate the method with several examples.

The process of regression testing is generally done under critical time and resource constraints. In the existing approaches, Test Case Prioritization (TCP), Test Case Selection (TCS), and Test Suite Reduction (TSR) have been treated as different optimization problems. This is based on simple heuristics that do not use valuable information that can be extracted from the program under test. Moreover, they do not allow for making internally consistent budget-conscious decisions. This paper introduces a unified approach that combines various variants of Graph Neural Networks (GNNs) and Reinforcement Learning (RL) for the solutions of TCP, TCS, and TSR. In our proposal, the representation of regression artifacts such as test cases, code entities, and faults is used as nodes in the typed graph, while the edges are used to show the relations between these artifacts. The relation-aware GNN is used for generating test case embeddings, reflecting the distance between the test cases and the changed areas as well as the areas known to have issues in the past. The actor-critic RL agent uses these test case embeddings and the budget to decide whether to run, skip, or discard test cases. The performance of our proposal outstrips coverage-based heuristics, history-based ranking, a GA-based search, an enhanced QPSO, and two learning-based ablations, as validated by our experiments on four Java projects from Defects4J. In terms of various programs and budgets, using GNN–RL has led to an improvement in the Average Percentage of Faults Detected (APFD) and Cost-cognizant Average Percentage of Faults Detected (APFDc) by approximately 6–9 percentage points on lower budgets. In addition, it is possible to obtain a 42% suite reduction and a 48% cost reduction while still retaining 98% of all fault-revealing tests in the final suite. The results show that it is possible to obtain promising solutions for adaptive regression testing using budget-aware reinforcement learning and graph-based representation learning.

Software needs modifications and requires revisions regularly. Owing to these revisions, retesting software becomes essential to ensure that the enhancements made, have not affected its bug-free functioning. The time and cost incurred in this process, need to be reduced by the method of test case selection and prioritization. It is observed that many nature-inspired techniques are applied in this area. African Buffalo Optimization is one such approach, applied to regression test selection and prioritization. In this paper, the proposed work explains and proves the applicability of the African Buffalo Optimization approach to test case selection and prioritization. The proposed algorithm converges in polynomial time (O(n2)). In this paper, the empirical evaluation of applying African Buffalo Optimization for test case prioritization is done on sample data set with multiple iterations. An astounding 62.5% drop in size and a 48.57% drop in the runtime of the original test suite were recorded. The obtained results are compared with Ant Colony Optimization. The comparative analysis indicates that African Buffalo Optimization and Ant Colony Optimization exhibit similar fault detection capabilities (80%), and a reduction in the overall execution time and size of the resultant test suite. The results and analysis, hence, advocate and encourages the use of African Buffalo Optimization in the area of test case selection and prioritization.

In recent years, Model-Based Testing (MBT) has attracted an increasingly wide interest from industry and academia. MBT allows automatic generation of a large and comprehensive set of test cases from system models (e.g., state machines), which leads to the systematic testing of the system. However, even when using simple test strategies, applying MBT in large industrial systems often leads to generating large sets of test cases that cannot possibly be executed within time and cost constraints. In this situation, test case selection techniques are employed to select a subset from the entire test suite such that the selected subset conforms to available resources while maximizing fault detection. In this paper, we propose a new similarity-based selection technique for state machine-based test case selection, which includes a new similarity function using triggers and guards on transitions of state machines and a genetic algorithm-based selection algorithm. Applying this technique on an industrial case study, we show that our proposed approach is more effective in detecting real faults than existing alternatives. We also assess the overall benefits of model-based test case selection in our case study by comparing the fault detection rate of the selected subset with the maximum possible fault detection rate of the original test suite.

In this paper, we present a controlled experiment for combinatorial designs algorithms aiming at software test case generation. We compare our recently proposed algorithm, TTR, to generate Mixed-Level Covering Array (MCA) with four other well-known combinatorial designs algorithms/tools regarding two aspects: cost in terms of the size of the set of test cases, and cost in terms of the time to generate the test suites. We used a set of 27 instances for this experiment. Results show that our algorithm was the best in terms of the size of the test suite, but was the poorest in terms of the time to generate the test cases. However, the not so good performance of our algorithm regarding the time to generate the test suite can be alleviated by the fact that TTR produces shorter set of test cases to be executed. We also made a comparison about the similarity of the test cases, i.e. to realize how similar are the test suites (test input data) of TTR compared with the other four algorithms/tools. We conclude that the TTR's test suite is not similar to any other test suites meaning that our algorithm has the potential to uncover different software defects by exercising different parts of the Software Under Test (SUT).

In many cases, Software Product Line Testing (SPLT) targets only the selection of test cases which cover product features or feature interactions. However, higher testing efficiency can be achieved through the selection of test cases with improved fault-revealing capabilities. By associating each test case a priority-value representing (or aggregating) different criteria, such as importance (in terms of fault discovered in previous test campaigns), duration or cost, it becomes possible to select a feature-covering test suite with improved capabilities. A crucial objective in SPLT then becomes to identify a test suite that optimizes reaching a specific goal (lower test duration or cost), while preserving full feature coverage.

Web application testing based on user session reduces artificial efforts while designing and generating test cases. It attracts more and more researchers and forms a hot spot of web application testing. Reduction, prioritization, and selection of test cases are widely used for the testing based on user session. In this paper, web application test cases optimization based on clustering is researched, and a novel method named USCHC (User Sessions Clustering based on Hierarchical Clustering algorithm for test cases optimization) is proposed. This method firstly gives the function to calculate the distance between the user sessions, and then employs the bottom-up agglutinate hierarchical clustering algorithm to cluster the initial testing cases and produces different kinds of test suites. From the clustered test suites, representative test cases are selected to replace the original test cases in each kind of test suites to test the functionality of web applications. The experiments show that the number of test cases is reduced through USCHC while the testing efficiency is improved.

In the testing-retraining pipeline for enhancing the robustness property of deep learning (DL) models, many state-of-the-art robustness-oriented fuzzing techniques are metric-oriented. The pipeline generates adversarial examples as test cases via such a DL testing technique and retrains the DL model under test with test suites that contain these test cases. On the one hand, the strategies of these fuzzing techniques tightly integrate the key characteristics of their testing metrics. On the other hand, they are often unaware of whether their generated test cases are different from the samples surrounding these test cases and whether there are relevant test cases of other seeds when generating the current one. We propose a novel testing metric called Contextual Confidence (CC). CC measures a test case through the surrounding samples of a test case in terms of their mean probability predicted to the prediction label of the test case. Based on this metric, we further propose a novel fuzzing technique Clover as a DL testing technique for the pipeline. In each fuzzing round, Clover first finds a set of seeds whose labels are the same as the label of the seed under fuzzing. At the same time, it locates the corresponding test case that achieves the highest CC values among the existing test cases of each seed in this set of seeds and shares the same prediction label as the existing test case of the seed under fuzzing that achieves the highest CC value. Clover computes the piece of difference between each such pair of a seed and a test case. It incrementally applies these pieces of differences to perturb the current test case of the seed under fuzzing that achieves the highest CC value and to perturb the resulting samples along the gradient to generate new test cases for the seed under fuzzing. Clover finally selects test cases among the generated test cases of all seeds as much as possible and with a preference to select test cases with higher CC values for improving model robustness. The experiments show that Clover outperforms the state-of-the-art coverage-based technique Adapt and loss-based fuzzing technique RobOT by 67%–129% and 48%–100% in terms of robustness improvement ratio, respectively, delivered through the same testing-retraining pipeline. For test case generation, in terms of numbers of unique adversarial labels and unique categories for the constructed test suites, Clover outperforms Adapt by \(2.0\times\) and \(3.5\times\) and RobOT by \(1.6\times\) and \(1.7\times\) on fuzzing clean models, and also outperforms Adapt by \(3.4\times\) and \(4.5\times\) and RobOT by \(9.8\times\) and \(11.0\times\) on fuzzing adversarially trained models, respectively.

Software testing is required to verify and validate systems. Combinatorial testing in one of the significant testing techniques. Design and select test cases for combinatorial testing considered as combinatorial problem. Even though there are some existing optimization algorithm based combinatorial testing strategies, that minimize the number of test cases, but most of these strategies based on one-test-at-a-time (OTAT) approach and none of them has adopted the one-parameter-at-a-time (OPAT) approach. Therefore, this paper will propose a new OPAT strategy based on Harmony Search Algorithm (HS) called OPAT-HS. OPAT-HS was originally designed only to support Covering Array (CA) and Mixed Covering Array (MCA) for uniform interaction strength. The result obtained in the experiments appears that OPAT-HS is always best at configurations with MCA notations.

This work introduces a model-based approach for minimization of test costs for interactive systems. Results known from state-based conformance testing and graph theory are used and extended to construct algorithms for test case generation and selection to cover the behavioral model of the system under test (SUT). The test case selection on the basis of the established model is ruled by an adequacy criterion, which provides a measure of how effective a given set of test cases is in terms of its potential to reveal faults. The approach is specification-oriented; i.e., the underlying model represents the system behavior interacting with the user's actions, that are viewed here as events.

Test case selection is the maintenance technique regarding the concept of choosing the appropriate representative of the modified program that has to be changed depending upon the new requirements added for the next modification. The main problem of updating software is the amounts of the test suite that is generated for the tests increases, which can drop the performance of using the new code that added to the previous program. According to this, execution and testing time will increase, including the complexity of integrating the codes also increase. Therefore, the test suite minimization approaches are offered to solve these problems. The objective of proposing the design based technique is to refine the competency of using test case selection to control the large programs by applying four algorithms, which are testing test case, classification, deletion, and selection. The result of determining the size reduction is greater than random selection, general regression technique, and test case prioritization as about 28.00%, 17.00%, and 11.00% approximately.

Ensuring the quality of Highly Configurable Systems (HCSs) during its evolution and maintenance is challenging. As an HCS evolves, new features are added, changed, or removed, which makes the test case selection for regression testing a difficult task. The use of test traceability can help in this task, but there is a lack of studies exploring the use of trace links for HCS testing. Existing work is usually based on the variability model, which is not always available or updated. Yet, the few existing approaches rely on links between test cases and files/lines of code, limiting the selection to test cases related to file changes, not considering the whole implementation of features, which can be spread in many files other than the changed ones. Considering this limitation, this work presents a test case selection approach, namely FeaTestSel, that links test cases to features using HCS pre-processor directives. Then, the selection of test cases is based on features affected by changes in each commit. In addition to the selected test cases, the approach also produces the following reports to support the test activity: the lines of code that correspond to each feature, the lines exercised by each test case, and the test cases linked to each feature. To validate the approach, we rely on Libssh, a real open-source HCS in constant evolution. By adding the execution time of the approach to the execution time of the selected test cases, we achieved a reduction of approximately ≈50%, in comparison with the retest-all technique. Furthermore, the approach was able to maintain quality by selecting 100% of failed test files. The traceability and reports produced by our approach can also be used for further work by researchers, analysis of the test quality by engineers, or as a source of information for tool builders.

In the software industry, software testing becomes more important in the entire software development life cycle. Software testing is one of the fundamental components of software quality assurances. Software Testing Life Cycle (STLC)is a process involved in testing the complete software, which includes Regression Testing, Unit Testing, Smoke Testing, Integration Testing, Interface Testing, System Testing & etc. In the STLC of Regression testing, test case selection is one of the most important concerns for effective testing as well as cost of the testing process. During the Regression testing, executing all the test cases from existing test suite is not possible because that takes more time to test the modified software. This paper proposes new Hybrid approach that consists of modified Greedy approach for handling the test case selection and Genetic Algorithm uses effective parameter like Initial Population, Fitness Value, Test Case Combination, Test Case Crossover and Test Case Mutation for optimizing the tied test suite. By doing this, effective test cases are selected and minimized the tied test suite to reduce the cost of the testing process. Finally the result of proposed approach compared with conventional greedy approach and proved that our approach is more effective than other existing approach.

Testing a group of software artifacts that implement the same specification can be time consuming, especially when the test case repository is large. In the meantime, some test cases may cover the same aspects in the software, thus it is not necessary to apply all the test cases. This paper proposes a model-based adaptive test (MAT) case selection and ranking technique to eliminate duplicate test cases, i.e., test cases with the similar coverage, and rank the test cases according to their potency and coverage. This technique can be applied in various domains where multiple versions of applications are available to test, such as web service testing, n-version applications, regression testing, and standard-based testing. The MAT is based a statistical model based on earlier testing results, and the model can determine the next sets of test cases to minimize the testing effort. The MAT is then applied to testing of multi-versioned web services and the results shows that the MAT can reduce testing effort while still maintain the effectiveness of testing.