Faulty Interaction Identification via Constraint Solving and Optimization

The Target Speed Monitor (TSM), which performs the safety critical function of the Chinese Train Control System Level 3 (CTCS-3), is characterized by large parameter input field and fault modes triggered by complicated combinations of parameters. Although combinatorial testing has been successfully applying in testing TSM, redundant test cases are generated because of covering the parameter combinations without interaction, which leads to the problem of high test execution costs. In this paper, an automatic reduction methodology based on decision tree equivalence and test suite output labels is presented. With previous combinatorial test suite generation experiment results of CTCS-3 TSM function, after data preprocessing, some data set have been constructed by test suites and corresponding output labels and they can be used to infer decision trees. After that, the reduction algorithm was used to reduce the data set to realize the test suite reduction. Related experimental results show that this method can achieve a reduction rate of up to 90%, and at the same time, the low-level combination coverage and the fault detection capability of reduced test suites are almost the same as that before the reduction.

Software testing and retesting occurs continuously during the software development lifecycle to detect errors as early as possible. As the software evolves the size of test suites also grows. When the no of test cases generated are more, obviously size of the test suite will also be more. So the testing time is to be minimized by reducing the execution time of the algorithm used for test data generation and also by introducing minimization procedure for test suite reduction. Due to limited resources and timing constraints for testing, test suite minimization techniques are needed to eliminate redundant test cases as possible. By considering multiple objectives rather than the coverage alone, the test cases are being generated which satisfies the testing requirements. Most of the existing techniques are code-based. In this article we present an approach by modifying an existing heuristic for test suite minimization. Genetic algorithm has been used for random test data generation and the output of GA is given to the minimization procedure for reducing the total no of generated test cases, collectively named as Hybrid Algorithm (HA). The results are satisfactory and show significant improvements in reducing test suite size with minimum execution time. Experiments have been done for simple to medium complexity java programs taken from SIR and execution time is reduced to 5,685ms for a test set. The results are compared with existing method Mutant Gene Algorithm and size of test suite is minimized upto 13.6% using Hybrid Algorithm.

Software testing is a fundamental step in the software development lifecycle. Its purpose is to evaluate the quality of software applications. Regression testing is an important testing methodology in software testing. The purpose of regression testing is to validate the software after each change of its code. This involves adding new test cases to the test suite and running the test suite as the software changes, making the test suite larger. The cost and time of the project are affected by the test suite size. The challenge is to run regression testing with a smaller number of test cases and larger amount of software coverage. Minimization of the test suite with maximization of the software coverage is an NP-complete problem. Search-based software engineering is an important topic in software engineering, which addresses software engineering optimization problems to find the optimal/approximate solution of the given problem. This study investigated an approach to reducing the regression testing effort and saving time. It also solved the regression testing optimization problem by achieving the maximum test suite coverage with the minimum test suite size. Several experiments were conducted to obtain the optimal solutions for the regression testing problem. We propose an optimization methodology that combines a genetic algorithm and a greedy algorithm to optimize regression testing by respectively maximizing the software test coverage and minimizing the test suite size. The proposed methodology can conveniently deliver fault-free, fully covered, and powerful programs for mission-critical functions. It can be applied to test a real-time system that has high requirements for reliability, security, and safety.

Combinatorial testing is a practical software testing approach that has been widely used in practice. Most research and applications of such approach focus on N-way combinatorial testing that provides a minimum coverage of all N-way interactions among factor. However, the strengths of different interactions may not be a fixed integer N, but a variable. Therefore, variable strength combinatorial testing approach is necessary in applications. Existing variable strength combinatorial testing, which allows some interactions have a higher strength than others, has a limitation that such higher-strength interactions must be disjoint. To avoid such a limitation, an improved variable strength combinatorial testing approach, which makes a more sufficient consideration on actual interaction relationship, is proposed in this article. Furthermore, two greedy heuristic algorithms are also proposed to generate combinatorial test suite. Compared to some existing algorithms and tools, the proposed algorithms have advantages on both the execution effectiveness and the optimality of generated test suite. Experimental results can prove such advantages.

Code coverage criteria define test objectives and provide actionable stopping conditions for creating test cases. They are also widely accepted as indicators of a test suite’s fault detection effectiveness. Several factors influence the relationship between code coverage and test suite effectiveness. However, previous empirical studies investigating this relationship tend to fail to control some of these factors, resulting in contradictory results. Through a systematic review of 417 previous studies investigating the effectiveness of test suites from various venues, such as journals, conferences, workshops, and book chapters, we have identified several factors that can impact experimental studies’ results and even threaten their validity. Some of these factors are well known, such as the test suite size (the number of test cases in the test suite), while some others are relatively unknown, such as the variety of execution traces (how different are test cases within a test suite in terms of structural coverage level). The list of factors we describe should be of interest to researchers and practitioners alike.

This study examines the implementation of Test Case Prioritization (TCP) using the Greedy Algorithm (GA) to enhance regression testing efficiency within a financial technology company's software development cycle. With testing durations increasing significantly, this study aims to address inefficiencies by applying the Greedy Algorithm to optimize test suite size and fault detection. The research methodology involves applying the Greedy Algorithm during the Regression Testing phase, comparing the prioritized suite with the original suite using metrics such as Average Percentage Fault Detection (APFD) and Test Suite Size Reduction (TSSR). Results show that the Greedy Algorithm achieved substantial improvements in both test suite size and fault detection effectiveness across different projects. For Project A, the test suite was reduced from 51-22 test cases, achieving a TSSR of 56.8%, with an APFD increase of 206.21%, rising from 0.0853-0.2613. Project B demonstrated even greater optimization, reducing the test suite from 36-8 test cases, resulting in a TSSR of 77.8% and an APFD improvement of 83.92%, rising from 0.3194-0.5875. These outcomes underscore the algorithm’s effectiveness in eliminating redundant test cases, accelerating testing, and enhancing fault detection thereby supporting the company's goal of faster release cycles without compromising quality.

Software testing is a critical part of software development. As new test cases are generated over time due to software modifications, test suite sizes may grow significantly. Because of time and resource constraints for testing, test suite minimization techniques are needed to remove those test cases from a suite that, due to code modifications over time, have become redundant with respect to the coverage of testing requirements for which they were generated. Prior work has shown that test suite minimization with respect to a given testing criterion can significantly diminish the fault detection effectiveness (FDE) of suites. We present a new approach for test suite reduction that attempts to use additional coverage information of test cases to selectively keep some additional test cases in the reduced suites that are redundant with respect to the testing criteria used for suite minimization, with the goal of improving the FDE retention of the reduced suites. We implemented our approach by modifying an existing heuristic for test suite minimization. Our experiments show that our approach can significantly improve the FDE of reduced test suites without severely affecting the extent of suite size reduction

Testing is an important activity to detect faults before software deployment. We focus on black-box combinatorial testing, where fault detection is one of the main objectives. In this paper, we argue that input model abstraction notably impacts the fault detection capability of a combinatorial test suite. First, we present experiments from previous work that support this argument. We then perform new experiments on a more diverse set of programs. These experiments use mutation testing to estimate fault detection capability, but we also include structural coverage measures in the new experiments. Finally, we elaborate on two possible improvements to obtain an optimal input abstraction strategy for not just continuous but all input domains. Both experiments suggest that input abstraction affects the fault detection capability. We claim that the improvements will produce a better input abstraction with which we can achieve better fault detection capability without increasing the test suite size.

Testers often represent systems under test in input parameter models. These contain parameters with associated values. Combinations of parameter values, with one value for each parameter, are potential test cases. In most models, some values of two or more parameters cannot be combined. Testers must then detect and avoid or remove these conflicts. This paper proposes two new methods for automatically handling such conflicts and compares these with two existing methods, based on the sizes of the final conflict-free test suites. A test suite reduction method, usable with three of the four investigated methods is also included in the study, resulting in seven studied conflict handling methods. In the experiment, the number and types of conflicts, as well as the size of the input parameter model and the coverage criterion used, are varied. All in all, 3854 test suites with a total of 929,158 test cases were generated. Two methods stand out as tractable and complementary. The best method (called the avoid methods) with respect to test suite size is to avoid selection of test cases with conflicts. However, this method cannot always be used. The second best method (called the replace method), removing conflicts from the final test suite, is completely general.

In this paper, we present a test data generation algorithm of combinatorial testing based on Differential Evolution, and introduce a selection and substitution based on the degree of unfinished interaction, in order to optimize the test case selected in further. An IPO-like strategy for generating the test suite is used to reduce the dimension for differential evolutionary computation in the optimization process and improve the efficiency of the algorithm. By preliminary experiments, the main parameters in DE algorithm are chosen. And in further experiments, we implement the algorithm in some typical instances to verify its effectiveness. Compare to other well-known algorithms, the final empirical results show the competitiveness of our algorithm in test suite size and running time.

Graphical user interfaces (GUIs) are used as front ends to most of today's software applications. The event-driven nature of GUIs presents new challenges for testing. One important challenge is test suite reduction. Conventional reduction techniques/tools based on static analysis are not easily applicable due to the increased use of multilanguage GUI implementations, callbacks for event handlers, virtual function calls, reflection, and multithreading. Moreover, many existing techniques ignore code in libraries and fail to consider the context in which event handlers execute. Consequently, they yield GUI test suites with seriously impaired fault-detection abilities. This paper presents a reduction technique based on the call-stack coverage criterion. Call stacks may be collected for any executing program with very little overhead. Empirical studies in this paper compare reduction based on call-stack coverage to reduction based on line, method, and event coverage, including variations that control for the size and optional consideration of library methods. These studies show that call-stack-based reduction provides unique trade-offs between the reduction in test suite size and the loss of fault detection effectiveness, which may be valuable in practice. Additionally, an analysis of the relationship between coverage requirements and fault-revealing test cases is presented.

Software testing is one of important phase in software development. The capabilities of t-way testing to cater bugs due to interactions while reducing the test suite size compare to exhaustive testing has been proven in past decades. However, the execution’s time of the t-way strategy also should be given attention as it could increase the productivity of the testing phase. Thus, this paper proposed a tune version of ant colony optimization algorithm (TACO). TACO is metaheuristic strategy where it adopts ant colony optimization in generating test suites. As further improvement, TACO also integrated with fuzzy logic to dynamically select amount of ant in the algorithm. TACO able to supports uniform strength t-way testing. Experiment result shows that TACO produce a remarkable result of test suite size and execution’s time compared to other strategy for uniform strength t-way testing.

Regression testing is the process of testing a modified system using the old test suite. As the test suite size is large, system retesting consumes large amount of time and computing resources. This issue of retesting of software systems can be handled using a good test case prioritization technique. A prioritization technique schedules the test cases for execution so that the test cases with higher priority executed before lower priority. The objective of test case prioritization is to detect fault as early as possible so that the debuggers can begin their work earlier. In this paper we propose a new prioritization technique to prioritize the test cases to perform regression testing for Component Based Software System (CBSS). The components and the state changes for a component based software systems are being represented by UML state chart diagrams which are then converted into Component Interaction Graph (CIG) to describe the interrelation among components. Our prioritization algorithm takes this CIG as input along with the old test cases and generates a prioritized test suit taking into account total number of state changes and total number of database access, both direct and indirect, encountered due to each test case. Our algorithm is found to be very effective in maximizing the objective function and minimizing the cost of system retesting when applied to few JAVA projects.

OpenDaylight is an open-source Software-Defined Network controller that has made programmable networks possible. It has got together both industry and the OpenDaylight community in delivering programmable and interoperable networks to service providers, enterprises, universities and organizations. It is a continuously evolving project with many different releases, where every new release requires extensive testing, to ensure quality software. Effective regression testing is required to ensure the existing features are not affected. Also a project of this scale will have many test cases, hence an effective selection method is required to ensure a reduced set of test cases for regression testing. The current paper proposes to adopt Combinatorial testing for generating test suite which has reduced number of test cases and better test coverage. Upon successful execution of the combinatorial test suite, the faults identified needs to be localized to help debug them effectively and in less time. This can be achieved through an efficient fault localization technique.KeywordsCombinatorial testingFault localizationOpenDaylightSoftware Defined Networks

Usage of combinatorial testing is wide spreading as an effective technique to reveal unintended feature interaction inside a given system. To this aim, test cases are constructed by combining tuples of assignments of the different input parameters, based on some effective combinatorial strategy. The most commonly used strategy is two-way (pairwise) coverage, requiring all combinations of valid assignments for all possible pairs of input parameters to be covered by at least one test case. In this paper a new heuristic strategy developed for the construction of pairwise covering test suites is presented, featuring a new approach to support expressive constraining over the input domain. Moreover, it allows the inclusion or exclusion of ad-hoc combinations of parameter bindings to let the user customize the test suite outcome. Our approach is tightly integrated with formal logic, since it uses test predicates to formalize combinatorial testing as a logic problem, and applies an external model checker tool to solve it. The proposed approach is supported by a prototype tool implementation, and early results of experimental assessment are also presented.