Reduction of Model Checking-based Test Generation using Satisfiability

On the order of test goals in specification-based testing

Test suites play an important role in developing reliable software applications. Generally, the behaviour of software applications is verified by executing test suites to find defects. The quality of a test suite needs to be evaluated and enriched (if needed) especially for testing critical systems, such as plane-navigation system. This paper presents a novel method for comparing concrete and executable test suites using equivalence classes. This comparison identifies gaps in test suites with respect to each other. These gaps indicate potential weaknesses in the test suites. Furthermore, this method provides a mechanism to enrich the test suites using these gaps. In this method, we devise equivalence classes, and associate each test case to an equivalence class. We, then, simulate the comparison of test suites by comparing sets of equivalence classes. The method compares test suites in a platform independent manner. The test suites, which are compared, are smaller than the original test suites because the redundant test cases are removed from the test suites, which makes it efficient. We exercise our method over three case studies to demonstrate its viability and effectiveness. The first case study illustrates the application of the method and evaluates its effectiveness using a mutation analysis. The second case study evaluates its effectiveness using mutation and coverage analyses. The final case study evaluates it on a real case study, which is Lucene search engine.

Software Testing is an important aspect of the real time software development process. Software testing always assures the quality of software product. As associated with software testing, there are few very important issues where there is a need to pay attention on it in the process of software development test. These issues are generation of effective test case and test suite as well as optimization of test case and suite while doing testing of software product. The important issue is that testing time of the test case and test suite. It is very much important that after development of software product effective testing should be performed. So to overcome these issues of optimization, we have proposed new approach for test suite optimization using genetic algorithm (GA). Genetic algorithm is evolutionary in nature so it is often used for optimization of problem by researcher. In this paper, our aim is to study various selections methods like tournament selection, rank selection and roulette wheel selection and then we apply this genetic algorithm (GA) on various programs which will generate optimized test suite with parameters like fitness value of test case, test suite and take minimum amount of time for execution after certain preset generation. In this paper our main objectives as per the experimental investigation, we show that tournament selection works very fine as compared to other methods with respect fitness selection of test case and test suites, testing time of test case and test suites as well as number of requirements.

Software testing is one of the important stages of software development. In software development, developers always depend on testing to reveal bugs. In the maintenance stage test suite size grow because of integration of new technique. Addition of new technique force to create new test case which increase the size of test suite. In regression testing new test case may be added to the test suite during the whole testing process. These additions of test cases create possibility of presence of redundant test cases. Due to limitation of time and resource, reduction techniques should be used to identify and remove them. Research shows that a subset of the test case in a suit may still satisfy all the test objectives which is called as representative set. Redundant test case increase the execution cost of the test suite, in spite of NP-completeness of the problem there are few good reduction techniques have been available. In this paper a new approach for test case reduction is proposed. This algorithm use genetic algorithm technique iteratively with varying chromosome length to reduce test case in a test suit by finding representative set of test cases that are fulfill the testing criteria.

Generating test cases with a model checker provides an effective means to perform test automation. However, there are usually much redundant calls to the model checker such that degrade the performance of test case generation, and also the generated test suit is often much redundant. In this paper, we propose a reduction approach to test case generation by using property rewriting technique. The approach includes twice rewritings: one is for eliminating redundant test goals represented in LTL properties, and the other for eliminating redundant test cases. A simple example, withdrawing money from ATM, is employed to illustrate our approach.

In the testing-retraining pipeline for enhancing the robustness property of deep learning (DL) models, many state-of-the-art robustness-oriented fuzzing techniques are metric-oriented. The pipeline generates adversarial examples as test cases via such a DL testing technique and retrains the DL model under test with test suites that contain these test cases. On the one hand, the strategies of these fuzzing techniques tightly integrate the key characteristics of their testing metrics. On the other hand, they are often unaware of whether their generated test cases are different from the samples surrounding these test cases and whether there are relevant test cases of other seeds when generating the current one. We propose a novel testing metric called Contextual Confidence (CC). CC measures a test case through the surrounding samples of a test case in terms of their mean probability predicted to the prediction label of the test case. Based on this metric, we further propose a novel fuzzing technique Clover as a DL testing technique for the pipeline. In each fuzzing round, Clover first finds a set of seeds whose labels are the same as the label of the seed under fuzzing. At the same time, it locates the corresponding test case that achieves the highest CC values among the existing test cases of each seed in this set of seeds and shares the same prediction label as the existing test case of the seed under fuzzing that achieves the highest CC value. Clover computes the piece of difference between each such pair of a seed and a test case. It incrementally applies these pieces of differences to perturb the current test case of the seed under fuzzing that achieves the highest CC value and to perturb the resulting samples along the gradient to generate new test cases for the seed under fuzzing. Clover finally selects test cases among the generated test cases of all seeds as much as possible and with a preference to select test cases with higher CC values for improving model robustness. The experiments show that Clover outperforms the state-of-the-art coverage-based technique Adapt and loss-based fuzzing technique RobOT by 67%–129% and 48%–100% in terms of robustness improvement ratio, respectively, delivered through the same testing-retraining pipeline. For test case generation, in terms of numbers of unique adversarial labels and unique categories for the constructed test suites, Clover outperforms Adapt by \(2.0\times\) and \(3.5\times\) and RobOT by \(1.6\times\) and \(1.7\times\) on fuzzing clean models, and also outperforms Adapt by \(3.4\times\) and \(4.5\times\) and RobOT by \(9.8\times\) and \(11.0\times\) on fuzzing adversarially trained models, respectively.

Software testing and retesting occurs continuously during the software development lifecycle to detect errors as early as possible. As the software evolves the size of test suites also grows. When the no of test cases generated are more, obviously size of the test suite will also be more. So the testing time is to be minimized by reducing the execution time of the algorithm used for test data generation and also by introducing minimization procedure for test suite reduction. Due to limited resources and timing constraints for testing, test suite minimization techniques are needed to eliminate redundant test cases as possible. By considering multiple objectives rather than the coverage alone, the test cases are being generated which satisfies the testing requirements. Most of the existing techniques are code-based. In this article we present an approach by modifying an existing heuristic for test suite minimization. Genetic algorithm has been used for random test data generation and the output of GA is given to the minimization procedure for reducing the total no of generated test cases, collectively named as Hybrid Algorithm (HA). The results are satisfactory and show significant improvements in reducing test suite size with minimum execution time. Experiments have been done for simple to medium complexity java programs taken from SIR and execution time is reduced to 5,685ms for a test set. The results are compared with existing method Mutant Gene Algorithm and size of test suite is minimized upto 13.6% using Hybrid Algorithm.

Many different techniques have been proposed to address the problem of automated test case generation, varying in a range of properties and resulting in very different test cases. In this paper we investigate the effects of the test case length on resulting test suites: Intuitively, longer test cases should serve to find more difficult faults but will reduce the number of test cases necessary to achieve the test objectives. On the other hand longer test cases have disadvantages such as higher computational costs and they are more difficult to interpret manually. Consequently, should one aim to generate many short test cases or fewer but longer test cases? We present the results of a set of experiments performed in a scenario of specification based testing for reactive systems. As expected, a long test case can achieve higher coverage and fault detecting capability than a short one, while giving preference to longer test cases in general can help reduce the size of test suites but can also have the opposite effect, for example, if minimization is applied.

Redundant test cases in newly generated test suites often remain undetected until execution and waste scarce project resources. In model-based testing, the testing process starts early on in the developmental phases and enables early fault detection. The redundancy in the test suites generated from models can be detected earlier as well and removed prior to its execution. The article presents a novel model-based test suite optimization technique involving UML activity diagrams by formulating the test suite optimization problem as an Equality Knapsack Problem. The aim here is the development of a test suite optimization framework that could optimize the model-based test suites by removing the redundant test cases. An evolution-based algorithm is incorporated into the framework and is compared with the performances of two other algorithms. An empirical study is conducted with four synthetic and industrial scale Activity Diagram models and results are presented.

The Target Speed Monitor (TSM), which performs the safety critical function of the Chinese Train Control System Level 3 (CTCS-3), is characterized by large parameter input field and fault modes triggered by complicated combinations of parameters. Although combinatorial testing has been successfully applying in testing TSM, redundant test cases are generated because of covering the parameter combinations without interaction, which leads to the problem of high test execution costs. In this paper, an automatic reduction methodology based on decision tree equivalence and test suite output labels is presented. With previous combinatorial test suite generation experiment results of CTCS-3 TSM function, after data preprocessing, some data set have been constructed by test suites and corresponding output labels and they can be used to infer decision trees. After that, the reduction algorithm was used to reduce the data set to realize the test suite reduction. Related experimental results show that this method can achieve a reduction rate of up to 90%, and at the same time, the low-level combination coverage and the fault detection capability of reduced test suites are almost the same as that before the reduction.

Database Management System (DBMS) developers have implemented extensive test suites to test their DBMSs. For example, the SQLite test suites contain over 92 million lines of code. Despite these extensive efforts, test suites are not systematically reused across DBMSs, leading to wasted effort. Integration is challenging, as test suites use various test case formats and rely on unstandardized test runner features. We present a unified test suite, SQuaLity, in which we integrated test cases from three widely-used DBMSs, SQLite, PostgreSQL, and DuckDB. In addition, we present an empirical study to determine the potential of reusing these systems' test suites. Our results indicate that reusing test suites is challenging: First, test formats and test runner commands vary widely; for example, SQLite has 4 test runner commands, while MySQL has 112 commands with additional features, to, for example, execute file operations or interact with a shell. Second, while some test suites contain mostly standard-compliant statements (e.g., 99% in SQLite), other test suites mostly test non-standardized functionality (e.g., 31% of statements in the PostgreSQL test suite are nonstandardized). Third, test reuse is complicated by various explicit and implicit dependencies, such as the need to set variables and configurations, certain test cases requiring extensions not present by default, and query results depending on specific clients. Despite the above findings, we have identified 3 crashes, 3 hangs, and multiple compatibility issues across four different DBMSs by executing test suites across DBMSs, indicating the benefits of reuse. Overall, this work represents the first step towards test-case reuse in the context of DBMSs, and we hope that it will inspire follow-up work on this important topic.

Web application testing based on user session reduces artificial efforts while designing and generating test cases. It attracts more and more researchers and forms a hot spot of web application testing. Reduction, prioritization, and selection of test cases are widely used for the testing based on user session. In this paper, web application test cases optimization based on clustering is researched, and a novel method named USCHC (User Sessions Clustering based on Hierarchical Clustering algorithm for test cases optimization) is proposed. This method firstly gives the function to calculate the distance between the user sessions, and then employs the bottom-up agglutinate hierarchical clustering algorithm to cluster the initial testing cases and produces different kinds of test suites. From the clustered test suites, representative test cases are selected to replace the original test cases in each kind of test suites to test the functionality of web applications. The experiments show that the number of test cases is reduced through USCHC while the testing efficiency is improved.

Guest Editorial: Special Section from the 11th International Conference on Quality Software (QSIC 2011)

Testing is the fundamental technique to assess the correctness of software systems, but it is cost-labored to generate test cases. One solution to change the situation is to automatize some parts of the testing process, especially the generation of test cases using formal theory and technology. The research work in the direction shows the good perspective. This paper targets on the automatic generation of test cases based on IOSTS, which is widely used to model reactive systems with data. When selecting test cases based on a set of test purposes specified by IOSTS or temporal logic, in general, the redundancy phenomena are unavoidable in the derived test suite. Hence, some strategies are presented for eliminating the redundancies in order to reduce the cost of implementing testing. More importantly, the strategies are directly applied to test cases in form of IOSTS, such that it can reduce not only the size of test suite, but also the cost of deriving test cases.

Test suite reduction techniques decrease the cost of software testing by removing the redundant test cases from the test suite while still producing a reduced set of tests that yields the same level of code coverage as the original suite. Most of the existing approaches to reduction aim to decrease the size of the test suite. Yet, the difference in the execution cost of the tests is often significant and it may be costly to use a test suite consisting of a few long-running test cases. Thus, this paper proposes an algorithm, based on the concept of test irreplaceability, which creates a reduced test suite with a decreased execution cost. Leveraging widely used benchmark programs, the empirical study shows that, in comparison to existing techniques, the presented algorithm is the most effective at reducing the cost of running a test suite.