An enhanced test case selection approach for model-based testing

Background: In today’s era, modifications in a software is a common requirement by customers. When changes are made to existing software, re-testing of all the test cases is required to ensure that the newly introduced changes do not have any unwanted effect on the behavior of the software. However, retesting of all the test cases would not only be time consuming but also expensive. Therefore, there is a need for a technique that reduces the number of tests to be performed. Regression testing is one of the ways to reduce the number of test cases. Selection technique is one such method which seeks to identify the test cases that are relevant to some set of recent changes. Objective: It is evident that most of the studies have used different selection techniques and have focused only on one parameter for achieving reduced test suite size without compromising the performance of regression testing. However, to the best of our knowledge, no study has taken two or more parameters of coverage, and/or execution time in a single testing. This paper presents a hybrid technique that combines both regression test selection using slicing technique and minimization of test cases using modified firefly algorithm with combination of parameters coverage and execution time in a single testing. Methods: A hybrid technique has been described that combines both selection and minimization. Selection of test cases is based upon slicing technique while minimization is done using firefly algorithm. Hybrid technique selects and minimizes the test suite using information on statement coverage and execution time. Results: The proposed technique gives 43.33% much superior result as compared to the other hybrid approach in terms of significantly reduced number of test cases. It shows that the resultant test cases were effective enough to cover 100% of the statements, for all the programs. The proposed technique was also tested on four different programs namely Quadratic, Triangle, Next day, Commission respectively for test suite selection and minimization which gave comparatively superior result in terms of reduction (%) in number of test cases required for testing. Conclusion: The combination of parameters used in slicing based approach, reduces the number of test cases making software testing an economical, feasible and time saving option without any fault in the source code. This proposed technique can be used by software practitioners/experts to reduce time, efforts and resources for selection and minimization of test cases.

Testing a group of software artifacts that implement the same specification can be time consuming, especially when the test case repository is large. In the meantime, some test cases may cover the same aspects in the software, thus it is not necessary to apply all the test cases. This paper proposes a model-based adaptive test (MAT) case selection and ranking technique to eliminate duplicate test cases, i.e., test cases with the similar coverage, and rank the test cases according to their potency and coverage. This technique can be applied in various domains where multiple versions of applications are available to test, such as web service testing, n-version applications, regression testing, and standard-based testing. The MAT is based a statistical model based on earlier testing results, and the model can determine the next sets of test cases to minimize the testing effort. The MAT is then applied to testing of multi-versioned web services and the results shows that the MAT can reduce testing effort while still maintain the effectiveness of testing.

In software testing, different testers focus on different aspects of the software such as functionality, performance, design, and other attributes. While many tools and coverage metrics exist to support testers at the code level, not much support is targeted for testers who want to inspect the output of a program such as a dynamic web application. To support this category of testers, we propose a family of output-coverage metrics (similar to statement, branch, and path coverage metrics on code) that measure how much of the possible output has been produced by a test suite and what parts of the output are still uncovered. To do that, we first approximate the output universe using our existing symbolic execution technique. Then, given a set of test cases, we map the produced outputs onto the output universe to identify the covered and uncovered parts and compute output-coverage metrics. In our empirical evaluation on seven real-world PHP web applications, we show that selecting test cases by output coverage is more effective at identifying presentation faults such as HTML validation errors and spelling errors than selecting test cases by traditional code coverage. In addition, to help testers understand output coverage and augment test cases, we also develop a tool called WebTest that displays the output universe in one single web page and allows testers to visually explore covered and uncovered parts of the output.

This work introduces a model-based approach for minimization of test costs for interactive systems. Results known from state-based conformance testing and graph theory are used and extended to construct algorithms for test case generation and selection to cover the behavioral model of the system under test (SUT). The test case selection on the basis of the established model is ruled by an adequacy criterion, which provides a measure of how effective a given set of test cases is in terms of its potential to reveal faults. The approach is specification-oriented; i.e., the underlying model represents the system behavior interacting with the user's actions, that are viewed here as events.

The process of regression testing is generally done under critical time and resource constraints. In the existing approaches, Test Case Prioritization (TCP), Test Case Selection (TCS), and Test Suite Reduction (TSR) have been treated as different optimization problems. This is based on simple heuristics that do not use valuable information that can be extracted from the program under test. Moreover, they do not allow for making internally consistent budget-conscious decisions. This paper introduces a unified approach that combines various variants of Graph Neural Networks (GNNs) and Reinforcement Learning (RL) for the solutions of TCP, TCS, and TSR. In our proposal, the representation of regression artifacts such as test cases, code entities, and faults is used as nodes in the typed graph, while the edges are used to show the relations between these artifacts. The relation-aware GNN is used for generating test case embeddings, reflecting the distance between the test cases and the changed areas as well as the areas known to have issues in the past. The actor-critic RL agent uses these test case embeddings and the budget to decide whether to run, skip, or discard test cases. The performance of our proposal outstrips coverage-based heuristics, history-based ranking, a GA-based search, an enhanced QPSO, and two learning-based ablations, as validated by our experiments on four Java projects from Defects4J. In terms of various programs and budgets, using GNN–RL has led to an improvement in the Average Percentage of Faults Detected (APFD) and Cost-cognizant Average Percentage of Faults Detected (APFDc) by approximately 6–9 percentage points on lower budgets. In addition, it is possible to obtain a 42% suite reduction and a 48% cost reduction while still retaining 98% of all fault-revealing tests in the final suite. The results show that it is possible to obtain promising solutions for adaptive regression testing using budget-aware reinforcement learning and graph-based representation learning.

Software needs modifications and requires revisions regularly. Owing to these revisions, retesting software becomes essential to ensure that the enhancements made, have not affected its bug-free functioning. The time and cost incurred in this process, need to be reduced by the method of test case selection and prioritization. It is observed that many nature-inspired techniques are applied in this area. African Buffalo Optimization is one such approach, applied to regression test selection and prioritization. In this paper, the proposed work explains and proves the applicability of the African Buffalo Optimization approach to test case selection and prioritization. The proposed algorithm converges in polynomial time (O(n2)). In this paper, the empirical evaluation of applying African Buffalo Optimization for test case prioritization is done on sample data set with multiple iterations. An astounding 62.5% drop in size and a 48.57% drop in the runtime of the original test suite were recorded. The obtained results are compared with Ant Colony Optimization. The comparative analysis indicates that African Buffalo Optimization and Ant Colony Optimization exhibit similar fault detection capabilities (80%), and a reduction in the overall execution time and size of the resultant test suite. The results and analysis, hence, advocate and encourages the use of African Buffalo Optimization in the area of test case selection and prioritization.

Software testing consists in the dynamic verification of the behavior of a program on a set of test cases. When a program is modified, it must be tested to verify if the changes did not imply undesirable effects on its functionality. The rerunning of all test cases can be impossible, due to cost, time and resource constraints. So, it is required the creation of a test cases subset before the test execution. This is a hard problem and the use of standard Software Engineering techniques could not be suitable. This work presents an approach for test case prioritization and selection, based in relevant inputs obtained from a software development environment. The approach uses Software Quality Function Deployment (SQFD) to deploy the features relevance among the system components, Mamdani fuzzy inference systems to infer the criticality of each class and Ant Colony Optimization to select test cases. An evaluation of the approach is presented, using data from simulations with different number of tests.

Testing is considered as indispensable part in the development process life cycle of any software. Testing is performed to improve the performance, quality and reliability of the software. Designing of relevant test cases plays a major role in deciding the quality of testing. Hence test case designing activity is crucial for the success of testing. Combinatorial strategies are one of those most extensively used methods for generating test cases. These strategies detect failures which are caused by interactions of parameters in the Software under Test(SUT) and by using some sampling mechanisms a covering array test suite is generated. With increase in the number of parameter coverage, the t-way test size sets also increase exponentially which would result in explosion problem. This paper presents a technique to generate minimal test suites in Combinatorial testing(CT). Optimization technique used is Greedy computation strategy for mutation and selection of test cases, used in selecting the locally worst test case for elimination. That is the test case that does not cover maximum number of uncovered interactions from a pool of candidate test cases. Additionally, Branch and bound technique is used for cross over. It is used to branch for the possibilities of number of test cases as cross over inputs. If a particular test case is not possible the bounding happens, else branched for further selection of test cases as input. Experimental results show that Recast Ipog-D generates better/comparable results as compared to the existing algorithm. Additionally, Recast IPOG-D has also contributed to enhance many known Variable Strength Covering array(VSCA) that exist in literature.

Test case generation, selection and coverage from natural language

In many cases, Software Product Line Testing (SPLT) targets only the selection of test cases which cover product features or feature interactions. However, higher testing efficiency can be achieved through the selection of test cases with improved fault-revealing capabilities. By associating each test case a priority-value representing (or aggregating) different criteria, such as importance (in terms of fault discovered in previous test campaigns), duration or cost, it becomes possible to select a feature-covering test suite with improved capabilities. A crucial objective in SPLT then becomes to identify a test suite that optimizes reaching a specific goal (lower test duration or cost), while preserving full feature coverage.

In the testing-retraining pipeline for enhancing the robustness property of deep learning (DL) models, many state-of-the-art robustness-oriented fuzzing techniques are metric-oriented. The pipeline generates adversarial examples as test cases via such a DL testing technique and retrains the DL model under test with test suites that contain these test cases. On the one hand, the strategies of these fuzzing techniques tightly integrate the key characteristics of their testing metrics. On the other hand, they are often unaware of whether their generated test cases are different from the samples surrounding these test cases and whether there are relevant test cases of other seeds when generating the current one. We propose a novel testing metric called Contextual Confidence (CC). CC measures a test case through the surrounding samples of a test case in terms of their mean probability predicted to the prediction label of the test case. Based on this metric, we further propose a novel fuzzing technique Clover as a DL testing technique for the pipeline. In each fuzzing round, Clover first finds a set of seeds whose labels are the same as the label of the seed under fuzzing. At the same time, it locates the corresponding test case that achieves the highest CC values among the existing test cases of each seed in this set of seeds and shares the same prediction label as the existing test case of the seed under fuzzing that achieves the highest CC value. Clover computes the piece of difference between each such pair of a seed and a test case. It incrementally applies these pieces of differences to perturb the current test case of the seed under fuzzing that achieves the highest CC value and to perturb the resulting samples along the gradient to generate new test cases for the seed under fuzzing. Clover finally selects test cases among the generated test cases of all seeds as much as possible and with a preference to select test cases with higher CC values for improving model robustness. The experiments show that Clover outperforms the state-of-the-art coverage-based technique Adapt and loss-based fuzzing technique RobOT by 67%–129% and 48%–100% in terms of robustness improvement ratio, respectively, delivered through the same testing-retraining pipeline. For test case generation, in terms of numbers of unique adversarial labels and unique categories for the constructed test suites, Clover outperforms Adapt by \(2.0\times\) and \(3.5\times\) and RobOT by \(1.6\times\) and \(1.7\times\) on fuzzing clean models, and also outperforms Adapt by \(3.4\times\) and \(4.5\times\) and RobOT by \(9.8\times\) and \(11.0\times\) on fuzzing adversarially trained models, respectively.

On the order of test goals in specification-based testing

Test case selection is the maintenance technique regarding the concept of choosing the appropriate representative of the modified program that has to be changed depending upon the new requirements added for the next modification. The main problem of updating software is the amounts of the test suite that is generated for the tests increases, which can drop the performance of using the new code that added to the previous program. According to this, execution and testing time will increase, including the complexity of integrating the codes also increase. Therefore, the test suite minimization approaches are offered to solve these problems. The objective of proposing the design based technique is to refine the competency of using test case selection to control the large programs by applying four algorithms, which are testing test case, classification, deletion, and selection. The result of determining the size reduction is greater than random selection, general regression technique, and test case prioritization as about 28.00%, 17.00%, and 11.00% approximately.

Web application testing based on user session reduces artificial efforts while designing and generating test cases. It attracts more and more researchers and forms a hot spot of web application testing. Reduction, prioritization, and selection of test cases are widely used for the testing based on user session. In this paper, web application test cases optimization based on clustering is researched, and a novel method named USCHC (User Sessions Clustering based on Hierarchical Clustering algorithm for test cases optimization) is proposed. This method firstly gives the function to calculate the distance between the user sessions, and then employs the bottom-up agglutinate hierarchical clustering algorithm to cluster the initial testing cases and produces different kinds of test suites. From the clustered test suites, representative test cases are selected to replace the original test cases in each kind of test suites to test the functionality of web applications. The experiments show that the number of test cases is reduced through USCHC while the testing efficiency is improved.

This research focuses on the Kawn Subscriptions Manager app, which was developed to help Food and Beverage (F&B) businesses manage customer subscriptions. Currently, the Kawn Subscriptions Manager application does not have a standardized test suite that has been validated. Therefore, in this research, Model-Based Testing (MBT) is applied in testing the Kawn Subscriptions Manager application as a testing method to produce a specific and technical test suite. MBT is a testing method based on the behavioral model of the system being tested. Then, to measure the adequacy of test cases generated by MBT, mutation testing is used. The model is created using the GraphWalker tool and testing is done automatically using Selenium and testNG in the Eclipse IDE text editor. From the results of the study, it is known that MBT testing generates test cases from the behavior model of the function being tested. Testing with MBT shows that no failures were found in the tested functions. Then the average mutation score is 72.18% which indicates that the test cases generated by MBT are adequate or in the medium category. However, it is important to note that while MBT was successful in generating test cases that examined the application's behavior, it was found that the method was unable to detect failures caused by mutants that did not significantly affect the application's behavior. This finding suggests that while MBT can be an effective tool for testing key functions, additional methods are needed to identify failures that may result from minor or insignificant changes to the application code.