A Method of Testing Generation Based on Prioritized Pair

Regression testing (RT) is an expensive activity. It is applied on a modified program to enhance confidence and reliability by ensuring that the changes are accurately true and have not affected the unmodified portions of the SUT. Due to limited resources, it is not practical to re-run each test cases (TC). To improve the regression testing's effectiveness, the TCs should be arranged according to some objective function or criteria. Test case prioritization (TCP) arranges TCs in an order for execution that enhances their effectiveness by satisfying some testing goals. The highest priority assigned to TCs must execute before the TCs with low priority by virtue of some performance goal. Numerous goals are possible to achieve of which one such goal is rate of fault detection (RFT) in which the faults are surfaced as quickly as possible within the testing process. In this paper, a novel technique is suggested to prioritize the TCs that increase its effectiveness in detecting faults. The effectiveness of the proposed method is compared and matched with other prioritization approaches with the help of Average Percentage of Fault Detection (APFD) metric from which charts have been prepared.

Combinatorial testing (CT) is a well-known technique for improving the quality of test plans while reducing testing costs. Traditionally, CT is used by testers at testing phase to design a test plan based on a manual definition of the test space. In this work, we extend the traditional use of CT to other parts of the development life cycle. We use CT at early design phase to improve design quality. We also use CT after test cases have been created and executed, in order to find gaps between design and test. For the latter use case we deploy a novel technique for a semi-automated definition of the test space, which significantly reduces the effort associated with manual test space definition. We report on our practical experience in applying CT for these use cases to three large and heavily deployed industrial products. We demonstrate the value gained from extending the use of CT by (1) discovering latent design flaws with high potential impact, and (2) correlating CT-uncovered gaps between design and test with field reported problems.

Software testing is required to verify and validate systems. Combinatorial testing in one of the significant testing techniques. Design and select test cases for combinatorial testing considered as combinatorial problem. Even though there are some existing optimization algorithm based combinatorial testing strategies, that minimize the number of test cases, but most of these strategies based on one-test-at-a-time (OTAT) approach and none of them has adopted the one-parameter-at-a-time (OPAT) approach. Therefore, this paper will propose a new OPAT strategy based on Harmony Search Algorithm (HS) called OPAT-HS. OPAT-HS was originally designed only to support Covering Array (CA) and Mixed Covering Array (MCA) for uniform interaction strength. The result obtained in the experiments appears that OPAT-HS is always best at configurations with MCA notations.

Testing is considered as indispensable part in the development process life cycle of any software. Testing is performed to improve the performance, quality and reliability of the software. Designing of relevant test cases plays a major role in deciding the quality of testing. Hence test case designing activity is crucial for the success of testing. Combinatorial strategies are one of those most extensively used methods for generating test cases. These strategies detect failures which are caused by interactions of parameters in the Software under Test(SUT) and by using some sampling mechanisms a covering array test suite is generated. With increase in the number of parameter coverage, the t-way test size sets also increase exponentially which would result in explosion problem. This paper presents a technique to generate minimal test suites in Combinatorial testing(CT). Optimization technique used is Greedy computation strategy for mutation and selection of test cases, used in selecting the locally worst test case for elimination. That is the test case that does not cover maximum number of uncovered interactions from a pool of candidate test cases. Additionally, Branch and bound technique is used for cross over. It is used to branch for the possibilities of number of test cases as cross over inputs. If a particular test case is not possible the bounding happens, else branched for further selection of test cases as input. Experimental results show that Recast Ipog-D generates better/comparable results as compared to the existing algorithm. Additionally, Recast IPOG-D has also contributed to enhance many known Variable Strength Covering array(VSCA) that exist in literature.

Combinatorial test case generation faces a problem on how to reduce the test cases by uncover the unnecessary test cases. So, there is a need for expert applications or strategies that generate the most optimum test cases keeping in mind the most important combinations. Complementing existing work on combinatorial test case generation strategies, also known as t-way testing strategies (i.e., where t represents interaction degree), this paper presents the design and the implementation of a new combinatorial test list generation strategy based on Harmony Search (HS) algorithm, called General T-way Harmony Search-based Strategy (GTHS). HS has been chosen to be the main engine of test generation because it could balance between intensification and diversification. Benchmarking experimental results show that GTHS produces competitive results as compared to other existing well-known optimization-based strategies and provides the support for high combination degrees (i.e., t ≤ 12).

Although combinatorial testing has been widely studied and used, there are still some situations and requirements that combinatorial testing does not apply to well, such as a system under test whose test cases need to be performed contiguously. For thorough testing, the testing requirements are not only to cover all the interactions among factors but also to cover all the value sequences of every factor. Generally, systems under test always involve constraints and dependencies in or among test cases. The constraints among test cases have not been effectively specified. First, we introduce extended covering arrays that can achieve both t-way combinatorial coverage and t-wise sequence coverage, and propose a clocked computation tree logic-based formal specification method for specifying constraints. Then, Particle Swarm Optimization (PSO) based Extended covering array Generator (PEG) is elaborated. To evaluate the constructed test suites, a method for verifying the constraints’ validity is presented, and kernel functions for measuring the coverage are also introduced. Finally, the performance of the proposed PEG is evaluated using several sets of benchmark experiments for some common constraints, and the feasibility and usefulness of PEG is validated.

A test scenario generation method based on combinatorial testing (CT) and Bayesian Network for autonomous vehicles is proposed in this paper. Firstly, some parameters are selected to describe the test scenarios which are classified according to road types and driving tasks. Then, the constraint sets for the scenarios with forbidden tuples are established to avoid the generated cases do not conform to the reality, in which the construct constraint set (CCS) algorithm is utilized to compute implied constraints. Furthermore, the Bayesian networks is used as the probabilistic models of the scenarios, where the traffic participants are represented as object nodes and the relative relationships between the participants are converted into the network structures. Finally, an improved automatic efficient test case generator (AETG) is developed to generate test cases. By considering both probability and frequency, the select function is designed for determining the values of scenario parameters. And the generation mode can be changed by modifying the weight and target parameters. The effectiveness of the proposed method is evaluated by generating six typical test scenarios. Compared with other algorithms, the numbers of test cases in the sets generated by this method are less and the probability deviations are smaller.

Test case prioritization involves reordering the test cases in an order that helps in attaining certain performance goals. The rate of fault detection is one of the prime goals that we tend to achieve while doing prioritization. Test cases should run in an order to increase the possibility of fault detection and it should be achieved early during the test life cycle. To reduce the cost and time of regression testing, test case prioritization should be done with the intention of periodically modifying the test suite. The humongous set of test cases makes it redundant and cumbersome for the testers who ensure quality for an end application. The fault detection capability of a prioritized test suite is improved up to 15% using Modified PSO which forms the base algorithms for prioritization. The algorithm illustrated detects serious errors at earlier phases of testing process and effectiveness between prioritized and unprioritized test cases.

Background/Objective: The objective is to minimize ambiguous test cases by using Adaptive Genetic Algorithm (AGA). Problem description/Proposed Method: Here we are concerned with the problem of randomly generated test cases. It can contain some ambiguous test cases, which lead to problems at the organizational level. A random algorithm will generate random test cases each time it is run, and it will have resemblance each time. Another problem related to random algorithms is that running them can take a lot of time. To minimize these issues we propose a new technique, which will reduce the given drawbacks. We proposed an Adaptive Genetic Algorithm (AGA), which will provide legal input in each case where it applied. Thus the problem of ambiguity will decrease. Results/Findings: In this research, the near optimal inputs will be generated based on the Adaptive Genetic Algorithm (AGA), which will reduce the illegal inputs. The fault detection rate is used as the fitness function in AGA. To remove the fault proneness, our AGA uses the coverage metrics of the test cases. Conclusion: Random algorithms will generate low cost test cases in large number but problem is that it will consists ambiguous test cases ,to reduce these here we are using AGA which will further reduce test cases by moderating the illegal inputs.

Abstract—Abstract Test Cases (ATCs) have been widely used in practice, including in combinatorial testing and in software product line testing. When constructing a set of ATCs, due to limited testing resources in practice (for example in regression testing), Test Case Prioritization (TCP) has been proposed to improve the testing quality, aiming at ordering test cases to increase the speed with which faults are detected. One intuitive and extensively studied TCP technique for ATCs is λ-wise Level-combination Coverage based Prioritization (λLCP), a static, black-box prioritization technique that only uses the ATC information to guide the prioritization process. A challenge facing λLCP, however, is the necessity for the selection of the fixed prioritization strength λ before testing — testers need to choose an appropriate λ value before testing begins. Choosing higher λ values may improve the testing effectiveness of λLCP (for example, by finding faults faster), but may reduce the testing efficiency (by incurring additional prioritization costs). Conversely, choosing lower λ values may improve the efficiency, but may also reduce the effectiveness. In this paper, we propose a new family of λLCP techniques, Repeated Small-strength Level-combination Coverage-based Prioritization (RSLCP), that repeatedly achieves the full combination coverage at lower strengths. RSLCP maintains λLCP’s advantages of being static and black box, but avoids the challenge of prioritization strength selection. We performed an empirical study involving five different versions of each of five C programs. Compared with λLCP, and Incremental strength LCP (ILCP), our results show that RSLCP could provide a good trade-off between testing effectiveness and efficiency. Our results also show that RSLCP is more effective and efficient than two popular techniques of Similarity-based Prioritization (SP). In addition, the results of empirical studies also show that RSLCP can remain robust over multiple system releases.

OpenDaylight is an open-source Software-Defined Network controller that has made programmable networks possible. It has got together both industry and the OpenDaylight community in delivering programmable and interoperable networks to service providers, enterprises, universities and organizations. It is a continuously evolving project with many different releases, where every new release requires extensive testing, to ensure quality software. Effective regression testing is required to ensure the existing features are not affected. Also a project of this scale will have many test cases, hence an effective selection method is required to ensure a reduced set of test cases for regression testing. The current paper proposes to adopt Combinatorial testing for generating test suite which has reduced number of test cases and better test coverage. Upon successful execution of the combinatorial test suite, the faults identified needs to be localized to help debug them effectively and in less time. This can be achieved through an efficient fault localization technique.KeywordsCombinatorial testingFault localizationOpenDaylightSoftware Defined Networks

Combinatorial testing is a prevailing approach to finding out faults in the extremely complex systems of train operation for railway. While combinatorial testing is applied to Chinese Train Control Systems (CTCS) for revealing faults, some potential faults tend to be arduous to trigger due to the interaction of multiple test cases. For the sake of accurate and efficient fault localization of CTCS under the circumstances of masking effects, a novel method of locating the faults of CTCS using the relationship matrix with frequent terms has been proposed in the paper. To begin with, failed test cases from CTCS were made full use of to calculate the frequentness of failure‐inducing elements for ordered tables. Then, the relationship matrix with frequent terms for CTCS was established to work out the suspiciousness of failure‐inducing combinations by the following relation of failure‐inducing elements in the above tables. Furthermore, an iterative approach to locating failure‐inducing combinations was adopted to obtain the full list of rankings of failure‐inducing combinations, among which top combinations, especially MFS (Minimal Failure‐causing Schema), were prone to discover the faults of CTCS. Finally, the effectiveness and accuracy of the proposed method for CTCS was confirmed on the simulation platform of CTCS for Beijing‐Shenyang High‐speed Rail. Empirical studies suggest that compared to the method of fault forest, the average suspiciousness in the proposed method is 12.17% higher than that and the number of additional test cases are reduced by 31.7% on average, which can produce more precise and efficient results of identifying the faults of CTCS in case of masking effects. The research achievements can provide some references for the safety evaluation and system optimization of CTCS for high‐speed rail. © 2022 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.

Usage of combinatorial testing is wide spreading as an effective technique to reveal unintended feature interaction inside a given system. To this aim, test cases are constructed by combining tuples of assignments of the different input parameters, based on some effective combinatorial strategy. The most commonly used strategy is two-way (pairwise) coverage, requiring all combinations of valid assignments for all possible pairs of input parameters to be covered by at least one test case. In this paper a new heuristic strategy developed for the construction of pairwise covering test suites is presented, featuring a new approach to support expressive constraining over the input domain. Moreover, it allows the inclusion or exclusion of ad-hoc combinations of parameter bindings to let the user customize the test suite outcome. Our approach is tightly integrated with formal logic, since it uses test predicates to formalize combinatorial testing as a logic problem, and applies an external model checker tool to solve it. The proposed approach is supported by a prototype tool implementation, and early results of experimental assessment are also presented.

Due to the complexity of the nuclear industrial distributed control system (DCS), input-domain testing techniques, including random testing and combinatorial testing, are usually utilized to test the control logics in nuclear industrial DCS. To improve the fault detection efficiency of random testing, the adaptive random testing technique selects a test case that significantly differs from all existing test cases. Similarly, to improve the fault detection efficiency of combinatorial testing, the greedy combinatorial testing technique adopts a greedy strategy to generate test cases that cover more uncovered tuple-combinations of parametric values. In this paper, we designed an experiment to compare the fault detection efficiency between adaptive random testing technique and greedy combinatorial testing technique for control logics of nuclear industrial DCS. Through the analysis of the fault detection ratios, the f-measure values, and the values of average percent of faults detected (APFD) on two experimental subjects, including the commonly used benchmarks in the field of Boolean-specification testing as well as a group of Boolean expressions extracted from the control logics in nuclear industrial DCS, the experimental results give us the following conclusions: (1) If the test suites' sizes are relatively small, the fault detection efficiencies of the two techniques are very close though there is a slight advantage in adaptive random testing; (2) With the gradual increase of test suites' sizes, the fault detection efficiency of greedy combinatorial testing is beyond adaptive random testing gradually. Such a result can help us select the appropriate testing techniques in the testing of the control logics in nuclear industry DCS.

Software reliability growth model with change-point and environmental function