An evaluation of change-based coverage criteria

Software Testing is an important aspect of the real time software development process. Software testing always assures the quality of software product. As associated with software testing, there are few very important issues where there is a need to pay attention on it in the process of software development test. These issues are generation of effective test case and test suite as well as optimization of test case and suite while doing testing of software product. The important issue is that testing time of the test case and test suite. It is very much important that after development of software product effective testing should be performed. So to overcome these issues of optimization, we have proposed new approach for test suite optimization using genetic algorithm (GA). Genetic algorithm is evolutionary in nature so it is often used for optimization of problem by researcher. In this paper, our aim is to study various selections methods like tournament selection, rank selection and roulette wheel selection and then we apply this genetic algorithm (GA) on various programs which will generate optimized test suite with parameters like fitness value of test case, test suite and take minimum amount of time for execution after certain preset generation. In this paper our main objectives as per the experimental investigation, we show that tournament selection works very fine as compared to other methods with respect fitness selection of test case and test suites, testing time of test case and test suites as well as number of requirements.

The Java-UML Lightweight Enumerator (JULE) tool implements a vitally important aspect of the framework for software tool certification -test suite generation.The framework uses UML models as the test inputs for the bounded exhaustive-testing approach.Within a size bound for the metamodel types, JULE enumerates only the set of nonisomorphic models in the form of relational structures.These models are classified into two sets -demonstration and counterexample -using Binary Decision Diagrams (BDDs).The power of JULE lies in its model enumeration and its use of a high-performance grid infrastructure.Hence, JULE efficiently generates a very small test suite while increasing the bound on the input size to the extent that is practical for certification purpose.

Testers often represent systems under test in input parameter models. These contain parameters with associated values. Combinations of parameter values, with one value for each parameter, are potential test cases. In most models, some values of two or more parameters cannot be combined. Testers must then detect and avoid or remove these conflicts. This paper proposes two new methods for automatically handling such conflicts and compares these with two existing methods, based on the sizes of the final conflict-free test suites. A test suite reduction method, usable with three of the four investigated methods is also included in the study, resulting in seven studied conflict handling methods. In the experiment, the number and types of conflicts, as well as the size of the input parameter model and the coverage criterion used, are varied. All in all, 3854 test suites with a total of 929,158 test cases were generated. Two methods stand out as tractable and complementary. The best method (called the avoid methods) with respect to test suite size is to avoid selection of test cases with conflicts. However, this method cannot always be used. The second best method (called the replace method), removing conflicts from the final test suite, is completely general.

Test suites play an important role in developing reliable software applications. Generally, the behaviour of software applications is verified by executing test suites to find defects. The quality of a test suite needs to be evaluated and enriched (if needed) especially for testing critical systems, such as plane-navigation system. This paper presents a novel method for comparing concrete and executable test suites using equivalence classes. This comparison identifies gaps in test suites with respect to each other. These gaps indicate potential weaknesses in the test suites. Furthermore, this method provides a mechanism to enrich the test suites using these gaps. In this method, we devise equivalence classes, and associate each test case to an equivalence class. We, then, simulate the comparison of test suites by comparing sets of equivalence classes. The method compares test suites in a platform independent manner. The test suites, which are compared, are smaller than the original test suites because the redundant test cases are removed from the test suites, which makes it efficient. We exercise our method over three case studies to demonstrate its viability and effectiveness. The first case study illustrates the application of the method and evaluates its effectiveness using a mutation analysis. The second case study evaluates its effectiveness using mutation and coverage analyses. The final case study evaluates it on a real case study, which is Lucene search engine.

The term grammar-based software describes software whose input can be specified by a context-free grammar. This grammar may occur explicitly in the software, in the form of an input specification to a parser generator, or implicitly, in the form of a hand-written parser. Grammar-based software includes not only programming language compilers, but also tools for program analysis, reverse engineering, software metrics and documentation generation. Hence, ensuring their completeness and correctness is a vital prerequisite for their use. In this paper we propose a strategy for the construction of test suites for grammar based software, and illustrate this strategy using the ISO C + + grammar. We use the concept of grammar-rule coverage as a pivot for the reduction of an implementation-based test suite, and demonstrate a significant decrease in the size of this suite. The effectiveness of this reduced test suite is compared to the original test suite with respect to code coverage and more importantly, fault detection. This work greatly expands upon previous work in this area and utilises large scale mutation testing to compare the effectiveness of grammar-rule coverage to that of statement coverage as a reduction criterion for test suites of grammar-based software. This work finds that when grammar rule coverage is used as the sole criterion for reducing test suites of grammar based software, the fault detection capability of that reduced test suite is greatly diminished when compared to other coverage criteria such as statement coverage.

A number of coverage criteria have been proposed for testing classes and class clusters modeled with state machines. Previous research has revealed their limitations in terms of their capability to detect faults. As these criteria can be considered to execute the control flow structure of the state machine, we are investigating how data flow information can be used to improve them in the context of UML state machines. More specifically, we investigate how such data flow analysis can be used to further refine the selection of a cost‐effective test suite among alternative, adequate test suites for a given state machine criterion. This paper presents a comprehensive methodology to perform data flow analysis of UML state machines—with a specific focus on identifying the data flow from OCL guard conditions and operation contracts—and applies it to a widely referenced coverage criterion, the round‐trip path (transition tree) criterion. It reports on two case studies whose results show that data flow information can be used to select the best transition tree, in terms of cost effectiveness, when more than one satisfies the transition tree criterion. The results also suggest that different trees are complementary in terms of the data flow that they exercise, thus, leading to the detection of intersecting but distinct subsets of faults. Copyright © 2009 John Wiley & Sons, Ltd.

In Combinatorial Interaction Testing, models specify a set of parameters with associated domains, and some constraints over the parameters. Test generation tools produce, starting from these models, test suites for achieving some given coverage criteria. The validation of both the models and the produced test suites is a worthwhile activity. Validating the models permits to early discover possible defects in them, to feed the test generations tools with good inputs and, possibly, to improve the quality of the testing process. Validating the produced test suites, instead, permits to check if the test generation tools are correct and to judge their quality. This paper proposes to validate the models by checking that the constraints are consistent, that there is no constraint implied by the other constraints, and that the parameters and their values are really necessary. The proposed test suite validation, instead, consists in checking that the tests respect the type definitions and the constraints, that all the test requirements are covered, and that all the tests in the test suite are valid and necessary. For every error we propose a possible technique able to identify the potential causes and to suggest fixes for those problems. Experiments show that the targeted defects are widespread both in benchmark and real-life models.

Code coverage criteria define test objectives and provide actionable stopping conditions for creating test cases. They are also widely accepted as indicators of a test suite’s fault detection effectiveness. Several factors influence the relationship between code coverage and test suite effectiveness. However, previous empirical studies investigating this relationship tend to fail to control some of these factors, resulting in contradictory results. Through a systematic review of 417 previous studies investigating the effectiveness of test suites from various venues, such as journals, conferences, workshops, and book chapters, we have identified several factors that can impact experimental studies’ results and even threaten their validity. Some of these factors are well known, such as the test suite size (the number of test cases in the test suite), while some others are relatively unknown, such as the variety of execution traces (how different are test cases within a test suite in terms of structural coverage level). The list of factors we describe should be of interest to researchers and practitioners alike.

A common application of search-based software testing is to generate test cases for all goals defined by a coverage criterion (e.g., statements, branches, mutants). Rather than generating one test case at a time for each of these goals individually, whole test suite generation optimizes entire test suites towards satisfying all goals at the same time. There is evidence that the overall coverage achieved with this approach is superior to that of targeting individual coverage goals. Nevertheless, there remains some uncertainty on whether the whole test suite approach might be inferior to a more focused search in the case of particularly difficult coverage goals. In this paper, we perform an in-depth analysis to study if this is the case. An empirical study on 100 Java classes reveals that indeed there are some testing goals that are easier to cover with the traditional approach. However, their number is not only very small in comparison with those which are only covered by the whole test suite approach, but also those coverage goals appear in small classes for which both approaches already obtain high coverage.

On the order of test goals in specification-based testing

Spectral fault localization is an automatic fault-localization technique to expedite debugging, which uses risk evaluation formula to rank the risk of fault existence in each program entity after collecting testing information. To assess the potential usefulness of a test suite and improves the accuracy for spectral fault localization, methods of assessing and optimizing test suite are proposed in this paper, which. Firstly, Average Ranking Cost of the test suite quality and two kinds of constrains are defined; and test suite quality assessment method based on these definitions is given. Secondly, a new test suite optimization method based on greedy algorithm is proposed. Finally, two widely used program databases (SIR and Defects4j) and 8 SFL techniques are applied to verify the effectiveness of our method; and the fault localization cost before and after optimizing test suites of test objects are analyzed using effect size. The largest effect size reaches 0.5398 and Each SFL technology has different degrees of improvement in the rankings of faulty statements in different programs by optimizing test suite.

Supply chain efficiency is a major challenge in today’s business environment, where efficient resource allocation and coordination of activities are essential for competitive advantage. Traditional efficiency strategies often struggle for resources for the complex and dynamic network. In response, bio-inspired metaheuristic algorithms have emerged as powerful tools to solve these optimization problems. Referring to the random search nature of metaheuristic algorithms and emphasizing that no metaheuristic algorithm is the best optimizer for all optimization applications, the No Free Lunch (NFL) theorem encourages researchers to design newer algorithms to be able to provide more effective solutions to optimization problems. Motivated by the NFL theorem, the innovation and novelty of this paper is in designing a new meta-heuristic algorithm called Bobcat Optimization Algorithm (BOA) that imitates the natural behavior of bobcats in the wild. The basic inspiration of BOA is derived from the hunting strategy of bobcats during the attack towards the prey and the chase process between them. The theory of BOA is stated and then mathematically modeled in two phases (i) exploration based on the simulation of the bobcat’s position change while moving towards the prey and (ii) exploitation based on simulating the bobcat’s position change during the chase process to catch the prey. The performance of BOA is evaluated in optimization to handle the CEC 2017 test suite for problem dimensions equal to 10, 30, 50, and 100, as well as to address CEC 2020. The optimization results show that BOA has a high ability in exploration, exploitation, and balance them during the search process in order to achieve a suitable solution for optimization problems. The results obtained from BOA are compared with the performance of twelve well-known metaheuristic algorithms. The findings show that BOA has been successful in handling the CEC 2017 test suite in 89.65, 79.31, 93.10, and 89.65% of the functions for the problem dimension equal to 10, 30, 50, and 100, respectively. Also, the findings show that in order to handle the CEC 2020 test suite, BOA has been successful in 100% of the functions of this test suite. The statistical analysis confirms that BOA has a significant statistical superiority in the competition with the compared algorithms. Also, in order to analyze the efficiency of BOA in dealing with real world applications, twenty-two constrained optimization problems from CEC 2011 test suite and four engineering design problems have been selected. The findings show that BOA has been successful in 90.90% of CEC2011 test suite optimization problems and in 100% of engineering design problems. In addition, the efficiency of BOA to handle SCM applications has been challenged to solve ten case studies in the field of sustainable lot size optimization. The findings show that BOA has successfully provided superior performance in 100% of the case studies compared to competitor algorithms.

Software testing is a critical part of software development. As new test cases are generated over time due to software modifications, test suite sizes may grow significantly. Because of time and resource constraints for testing, test suite minimization techniques are needed to remove those test cases from a suite that, due to code modifications over time, have become redundant with respect to the coverage of testing requirements for which they were generated. Prior work has shown that test suite minimization with respect to a given testing criterion can significantly diminish the fault detection effectiveness (FDE) of suites. We present a new approach for test suite reduction that attempts to use additional coverage information of test cases to selectively keep some additional test cases in the reduced suites that are redundant with respect to the testing criteria used for suite minimization, with the goal of improving the FDE retention of the reduced suites. We implemented our approach by modifying an existing heuristic for test suite minimization. Our experiments show that our approach can significantly improve the FDE of reduced test suites without severely affecting the extent of suite size reduction

In many cases, Software Product Line Testing (SPLT) targets only the selection of test cases which cover product features or feature interactions. However, higher testing efficiency can be achieved through the selection of test cases with improved fault-revealing capabilities. By associating each test case a priority-value representing (or aggregating) different criteria, such as importance (in terms of fault discovered in previous test campaigns), duration or cost, it becomes possible to select a feature-covering test suite with improved capabilities. A crucial objective in SPLT then becomes to identify a test suite that optimizes reaching a specific goal (lower test duration or cost), while preserving full feature coverage.

A Case Study in Matching Test and Proof Coverage