Application of Pairwise Testing into BWDM which is a Test Case Generation Tool for the VDM++ Specification

Pairwise testing is a combinatorial testing technique that tests all possible pairs of input values. Although, finding a smallest set of test cases for pairwise testing is NP-complete, pairwise testing is regarded as a reasonable cost-benefit compromise among combinatorial testing methods. In this paper we formulate the problem of finding a pairwise test set as a search problem and apply a genetic algorithm to solve it. We also describe an open-source tool called PWiseGen for generating pairwise test sets. PWiseGen produces competitive results compared with existing pairwise testing tools. Besides, it provides a framework and a research platform for generating pairwise test sets using genetic algorithms; it is configurable, extensible, and reusable.

Usually, statistical tests for interactions are applied to investigate potential effect modifiers. If an effect modifier, consisting of three categories, is found to be statistically significant, the application of pairwise interaction tests is indicated. In this case, the problem of non-transitive relations may occur if the significance level is fixed at 0.05 for all tests. To develop an algorithm for which non-transitive relations do not occur. A hierarchical testing procedure is applied, based on the heterogeneity statistic Q. In a first step the interaction will be tested for the three trial subgroups altogether, applying the significance level α = 0.05 (global test). If a significant interaction is proven in the first step, pairwise tests for interaction will be applied in a second step. Theoretical data scenarios will be considered and p-values will be calculated for the pairwise tests. Based on these results the significance level for pairwise testing will be determined. Fixing the significance level at 0.05 for all tests, the problem of non-transitive relations is mostly relevant, if the difference in the effects between the three trial subgroups is approximately 3.5 standard errors and the effect of the 'middle' trial subgroup is not close to one of the other two effects. This problem vanishes when the significance level is set to α = 0.22. We propose to select α = 0.20 to get a more 'even' and simple value. By increasing the significance level for the pairwise tests to 0.20, non-transitive relations are virtually avoidable. The proposed hierarchical testing procedure represents a clear practical guidance to perform subgroup analyses in the framework of systematic reviews.

In this paper, we report on the effectiveness of the testing approach known as pairwise or orthogonal testing in verifying the correctness of the LTL specifications generated by the PROperty SPECification (Prospec) tool. This tool assists the user in generating a large number (over 34,000) of formal specifications in formal languages, including Linear Temporal Logic (LTL). Pairwise testing is a technique that aims at, significantly, reducing the amount of test cases required for testing a particular software system while providing assurance of adequate coverage of the problem space.

While pair-wise testing has shown a high level of fault detection in many situations, it has not done so for testing logical expressions in software applications. The modified condition/decision coverage (MC/DC) approach was especially developed for testing logical expressions, but test generation according to this approach can be complicated and time-consuming. From a practical point of view, combining pair-wise and MC/DC testing would integrate the benefits of both approaches; however, this would require solving many research problems. As an initial step in this direction, our paper evaluates the level of MC/DC coverage for pair-wise test cases in different situations and compares this level to MC/DC coverage for random test cases. To confirm this, we conducted experimental testing of logical expressions of different sizes, complexities, and numbers of input variables. Our experimental results show that the pair-wise test cases had a higher level of MC/DC coverage compared to the random test cases, achieving more than 70% MC/DC coverage. This allows us to suggest pair-wise testing as a promising method to be used as the basis for MC/DC test generation.

Pair-wise testing is a practical and effective method which has already been used in the software testing. Extensive research has been made on the generation of pair-wise testing. In order to make it easy to analyze the current generation methods, we propose a method to ease the process. That is we transform the problem of pair-wise testing to a graphic one. The IPO algorithm is based on parameters and can ensure the optimization of test cases in each expansion. Though it has many advantages, it is still not sustainable enough because of its flexibility. We studied the three elements which affect its sustainability. The three elements are the horizontal growth of pair-wise testing, the combination of pair-wise testing cases and the extension sequence of the parameters to be extended. Thus we propose a HIPO algorithm based on IPO algorithm to solve those problems. The HIPO algorithm inherits the merits of high extension of IPO algorithm and introduces a new concept of contribution extent. It adopts the methods of preferential sequence as well as minimization algorithm to optimize the problems above. We develop the test case generation tool based on the HIPO algorithm by means of .Net technology. And we also prove its effectiveness in our experiment.

For most software systems, pair-wise combinatorial testing is highly practical and effective. This article mainly starts with the specific research status of the pair-wise covering combinatorial test and the main content of the software test, analyzes and compares the existing main implementation algorithms, and then evaluates their specific performance.

Pairwise testing can greatly minimize the cost of software testing and also increase the ability of fault detection. Nevertheless, generating the most optimal test suite is an NP-complete problem and still an open area for research. The test case generation is the most active area of the pairwise testing research. Metaheuristic algorithms have been broadly used for solving difficult optimization problems as well as proving their effectiveness to get most optimal solutions. Kidney algorithm (KA) is a recent metaheuristic algorithm. This study introduces a new pairwise strategy by adapting KA; which is the first time to adapt KA in generating the test suite. The proposed strategy is called Pairwise Kidney Strategy (PKS). This study also highlights the PKS design; in addition, compare its performance with other reported strategies in the literature in terms of test suite size. Experiment results show that PKS has very competitive results as compared with other strategies.

In vivo rodent erythrocyte micronucleus assay

Aiming at the construction of driverless vehicle test scenario, this paper proposes a construction research of driverless vehicle test scenario library in campus environment. By analyzing the unique characteristics of the campus environment, the definition of different from the open/closed road test scenario is proposed. Combining with the characteristics of the scenario, the scenario elements are analyzed in layers. The combination analysis of scenario elements at different levels was carried out by using PICT(Pairwise Independent Combinatorial Testing) combination test tool, and random sampling method is used to extract different levels of sample sets. Through the analysis of scene elements and driving task complexity, this paper proposes a generation system of campus driverless vehicle combination test scenarios. According to the construction method of campus test scenario proposed, an obstacle avoidance test case is generated, and its simulation verification is carried out, the results show that the test vehicle can effectively decelerate and stop and avoid collision when encountering obstacles in front, which verifies the effectiveness of the test scenario.

Flight control software is the nerve center of flight control system, which has high functional complexity and difficult test. In this paper, a test method based on input field model is proposed. The input space of flight control software is identified, and input domain modeling is carried out. According to the pairwise combination test coverage strategy, a heuristic test case generation algorithm with constraints is used to generate test cases so as to guide the test execution. The example shows that the characteristic modeling of flight control software input domain and the test based on the model can meet the requirements of the sufficiency and coverage of flight control software test, greatly improve the effectiveness of software testing and reduce the test cost.)

Based on our experience of collaborating with industry, we observed that test case generation usually relies on test case specifications (TCSs), commonly written in natural language, specifying test cases of a System Under Test at a high level of abstraction. In practice, TCSs are commonly used by test engineers as reference documents to perform these activities: 1) Manually executing test cases in TCSs; 2) Manually coding test cases in a test scripting language for automated test case execution. In the latter case, the gap between TCSs and executable test cases has to be filled by test engineers, requiring a significant amount of coding effort and domain knowledge. Motivated by the above observations from the industry, we first propose, in this paper, a TCS language, named as Restricted Test Case Modeling (RTCM), which is based on natural language and composed of an easy-to-use template, a set of restriction rules and keywords. Second, we propose a test case generation tool (aToucan4Test), which takes TCSs in RTCM as input and generates either manual test cases or automatically executable test cases, based on various coverage criteria defined on RTCM. To assess the applicability of RTCM, we manually modeled two industrial case studies and examined 30 automatically generated TCSs. To evaluate aToucan4Test, we modeled three subsystems of a Video Conferencing System developed by Cisco Systems, Norway and automatically generated executable test cases. These test cases were successfully executed on two commercial software versions. In the paper, we also discuss our experience of applying RTCM and aToucan4Test in an industrial context and compare our approach with other model-based testing methodologies.

Despite recent progress in automated test generation research, significant challenges remain for applying these techniques on large-scale software systems. These systems under test often require structurally complex test inputs within a large input domain. It is challenging to automatically generate a reasonable number of tests that are both legal and behaviorally-diverse to exercise these systems. Constraint-based test generation is an effective approach for generating structurally complex inputs for systematic testing. While this approach can typically generate large numbers of tests, it has limited scalability – tests generated are usually only up to a small bound on input size. Combinatorial test generation, e.g., pair-wise testing, is a more scalable approach but is challenging to apply on commercial software systems that require complex input structures that cannot be formed by using arbitrary combinations. This paper introduces comKorat, which unifies constraint-based generation of structurally complex tests with combinatorial testing. Specifically, comKorat integrates Korat and ACTS test generators to generate test suites for large scale software systems with structurally complex test inputs. We have successfully applied comKorat on four software applications developed at eBay and Yahoo!. The experimental results show that comKorat outperforms existing solutions in execution time and test coverage. Furthermore, comKorat found a total of 59 previously unknown bugs in the four applications.

Combinatorial testing has lots of test cases, but software testers hope to get the best test coverage with the smallest test case suite. For the scale of produced test cases is so large that researchers have considered the implementation of the critical test cases. This article researches the classic combinatorial test methods and proposes methods to generate pair-wise testing cases with a priority. Firstly, we design formulas to compute the weights of priorities. Secondly, we adopt a greed algorithm to solve the combinatorial testing problems. Furthermore, we integrate the greed strategy into a genetic algorithm to improve the efficiency. It improves the testing efficiency while securing the detection rate of defects under limited resources.

Quality is a key property of modern software systems. In addition, to quality improvement strategies such as CMM or CMMI and manual checking approaches such as reviews or inspections, software testing is the major quality assurance activity in most projects. Literally speaking, software testing is nearly as essential as writing code itself. Due to its prominent role there are quite a number of commercial and open-source test tools available. However, the question remains if an organisation should rely on a single tool or if it should follow a more general strategy using several tools? Practical experience has shown that the latter strategy is more promising by covering multiple quality aspects that cannot be covered by a single tool, although it requires a means of communication between tools (i.e., a tool independent test language). This paper introduces a XML-based and test tool independent test modeling language that support creating general test cases. The language separates test logic from test tool specific code. Thus, a test case only has to be specified once and can then be executed by several test tools.

In the testing-retraining pipeline for enhancing the robustness property of deep learning (DL) models, many state-of-the-art robustness-oriented fuzzing techniques are metric-oriented. The pipeline generates adversarial examples as test cases via such a DL testing technique and retrains the DL model under test with test suites that contain these test cases. On the one hand, the strategies of these fuzzing techniques tightly integrate the key characteristics of their testing metrics. On the other hand, they are often unaware of whether their generated test cases are different from the samples surrounding these test cases and whether there are relevant test cases of other seeds when generating the current one. We propose a novel testing metric called Contextual Confidence (CC). CC measures a test case through the surrounding samples of a test case in terms of their mean probability predicted to the prediction label of the test case. Based on this metric, we further propose a novel fuzzing technique Clover as a DL testing technique for the pipeline. In each fuzzing round, Clover first finds a set of seeds whose labels are the same as the label of the seed under fuzzing. At the same time, it locates the corresponding test case that achieves the highest CC values among the existing test cases of each seed in this set of seeds and shares the same prediction label as the existing test case of the seed under fuzzing that achieves the highest CC value. Clover computes the piece of difference between each such pair of a seed and a test case. It incrementally applies these pieces of differences to perturb the current test case of the seed under fuzzing that achieves the highest CC value and to perturb the resulting samples along the gradient to generate new test cases for the seed under fuzzing. Clover finally selects test cases among the generated test cases of all seeds as much as possible and with a preference to select test cases with higher CC values for improving model robustness. The experiments show that Clover outperforms the state-of-the-art coverage-based technique Adapt and loss-based fuzzing technique RobOT by 67%–129% and 48%–100% in terms of robustness improvement ratio, respectively, delivered through the same testing-retraining pipeline. For test case generation, in terms of numbers of unique adversarial labels and unique categories for the constructed test suites, Clover outperforms Adapt by \(2.0\times\) and \(3.5\times\) and RobOT by \(1.6\times\) and \(1.7\times\) on fuzzing clean models, and also outperforms Adapt by \(3.4\times\) and \(4.5\times\) and RobOT by \(9.8\times\) and \(11.0\times\) on fuzzing adversarially trained models, respectively.