Intelligent evaluation of test suites for developing efficient and reliable software

Software Testing is an important aspect of the real time software development process. Software testing always assures the quality of software product. As associated with software testing, there are few very important issues where there is a need to pay attention on it in the process of software development test. These issues are generation of effective test case and test suite as well as optimization of test case and suite while doing testing of software product. The important issue is that testing time of the test case and test suite. It is very much important that after development of software product effective testing should be performed. So to overcome these issues of optimization, we have proposed new approach for test suite optimization using genetic algorithm (GA). Genetic algorithm is evolutionary in nature so it is often used for optimization of problem by researcher. In this paper, our aim is to study various selections methods like tournament selection, rank selection and roulette wheel selection and then we apply this genetic algorithm (GA) on various programs which will generate optimized test suite with parameters like fitness value of test case, test suite and take minimum amount of time for execution after certain preset generation. In this paper our main objectives as per the experimental investigation, we show that tournament selection works very fine as compared to other methods with respect fitness selection of test case and test suites, testing time of test case and test suites as well as number of requirements.

Software testing is one of the important stages of software development. In software development, developers always depend on testing to reveal bugs. In the maintenance stage test suite size grow because of integration of new technique. Addition of new technique force to create new test case which increase the size of test suite. In regression testing new test case may be added to the test suite during the whole testing process. These additions of test cases create possibility of presence of redundant test cases. Due to limitation of time and resource, reduction techniques should be used to identify and remove them. Research shows that a subset of the test case in a suit may still satisfy all the test objectives which is called as representative set. Redundant test case increase the execution cost of the test suite, in spite of NP-completeness of the problem there are few good reduction techniques have been available. In this paper a new approach for test case reduction is proposed. This algorithm use genetic algorithm technique iteratively with varying chromosome length to reduce test case in a test suit by finding representative set of test cases that are fulfill the testing criteria.

Constructing test cases from counterexamples generated by a model checker is an important method to perform test automation. In fact, one counterexample may cover multiple test goals, which leads to unnecessary calls to the model checker, and redundant test cases in test suite such that affect the testi ng performance. A method to test suite reduction based on satisfiability is proposed. The kripke model is translated in conjunction with test goals (trap properties) into CNFs. And then test goal t o generate counterexample is selected according to the hardness of the corresponding CNF, after that, model checking the selected test goal to generate counterexample. The generated counterexample is translated in conjunction with those uncovered test goals i nto CNFs. If the corresponding CNF is unsatisfiable then the test goal is pick ed out from the set of test goals. Meanwhile, the new generated test case is winnowed by test suite to reduce the redundancy before it is added into the test suite. Experimental results show that t he method proposed in this paper is effective for reducing the model checker calls and the length of the test suite. At the same time, the coverage and error detection capability of the test suite are not decl ined.

Software developers use testing to gain and maintain confidence in the correctness of a software system. Automated reduction and prioritization techniques attempt to decrease the time required to detect faults during test suite execution. This paper uses the Harrold Gupta Soffa, delayed greedy, traditional greedy, and 2-optimal greedy algorithms for both test suite reduction and prioritization. Even though reducing and reordering a test suite is primarily done to ensure that testing is cost-effective, these algorithms are normally configured to make greedy choices with coverage information alone. This paper extends these algorithms to greedily reduce and prioritize the tests by using both test cost (e.g., execution time) and the ratio of code coverage to test cost. An empirical study with eight real world case study applications shows that the ratio greedy choice metric aids a test suite reduction method in identifying a smaller and faster test suite. The results also suggest that incorporating test cost during prioritization allows for an average increase of 17% and a maximum improvement of 141% for a time sensitive evaluation metric called coverage effectiveness.

Testing is an important activity in engineering of industrial embedded software. In certain application domains (e.g., railway industry) engineering software is certified according to safety standards that require extensive software testing procedures to be applied for the development of reliable systems. Mutation analysis is a technique for creating faulty versions of a software for the purpose of examining the fault detection ability of a test suite. Mutation analysis has been used for evaluating existing test suites, but also for generating test suites that detect injected faults (i.e., mutation testing). To support developers in software testing, we propose a technique for producing test cases using an automated test generation approach that operates using mutation testing for software written in IEC 61131-3 language, a programming standard for safety-critical embedded software, commonly used for Programmable Logic Controllers (PLCs). This approach uses the Uppaal model checker and is based on a combined model that contains all the mutants and the original program. We applied this approach in a tool for testing industrial PLC programs and evaluated it in terms of cost and fault detection. For realistic validation we collected industrial experimental evidence on how mutation testing compares with manual testing as well as automated decision-coverage adequate test generation. In the evaluation, we used manually seeded faults provided by four industrial engineers. The results show that even if mutation-based test generation achieves better fault detection than automated decision coverage-based test generation, these mutation-adequate test suites are not better at detecting faults than manual test suites. However, the mutation-based test suites are significantly less costly to create, in terms of testing time, than manually created test suites. Our results suggest that the fault detection scores could be improved by considering some new and improved mutation operators (e.g., Feedback Loop Insertion Operator (FIO)) for PLC programs as well as higher-order mutations.

With the increasing adoption of embedded processors, the need of developing compilers for the embedded processors with timely manner is also growing. Retargeting has been adopted as a viable approach to constructing new compilers by modifying the back-end of an existing compiler. This paper proposes a test suite generation system for testing retargetable C compilers. The proposed system generates the test suite using the grammar coverage concept. Generally, the size of the test suite satisfying the grammar coverage of the source language is very large. Hence, the proposed system also provides the facility to reduce the size of the test suite. According to the experimental result, the reduced test suite can detect 75% of the compiler faults detected by the original test suite though the size of the reduced test suite is only 10% of that of the original test suite in average. This result indicates that the reduction technique proposed in this paper can be effectively used in the prior phase of the development procedure of the embedded compilers. Keywords:Compiler Testing, Retargetable C Compiler, Embedded System, Intermediate Code

Database Management System (DBMS) developers have implemented extensive test suites to test their DBMSs. For example, the SQLite test suites contain over 92 million lines of code. Despite these extensive efforts, test suites are not systematically reused across DBMSs, leading to wasted effort. Integration is challenging, as test suites use various test case formats and rely on unstandardized test runner features. We present a unified test suite, SQuaLity, in which we integrated test cases from three widely-used DBMSs, SQLite, PostgreSQL, and DuckDB. In addition, we present an empirical study to determine the potential of reusing these systems' test suites. Our results indicate that reusing test suites is challenging: First, test formats and test runner commands vary widely; for example, SQLite has 4 test runner commands, while MySQL has 112 commands with additional features, to, for example, execute file operations or interact with a shell. Second, while some test suites contain mostly standard-compliant statements (e.g., 99% in SQLite), other test suites mostly test non-standardized functionality (e.g., 31% of statements in the PostgreSQL test suite are nonstandardized). Third, test reuse is complicated by various explicit and implicit dependencies, such as the need to set variables and configurations, certain test cases requiring extensions not present by default, and query results depending on specific clients. Despite the above findings, we have identified 3 crashes, 3 hangs, and multiple compatibility issues across four different DBMSs by executing test suites across DBMSs, indicating the benefits of reuse. Overall, this work represents the first step towards test-case reuse in the context of DBMSs, and we hope that it will inspire follow-up work on this important topic.

Redundant test cases in newly generated test suites often remain undetected until execution and waste scarce project resources. In model-based testing, the testing process starts early on in the developmental phases and enables early fault detection. The redundancy in the test suites generated from models can be detected earlier as well and removed prior to its execution. The article presents a novel model-based test suite optimization technique involving UML activity diagrams by formulating the test suite optimization problem as an Equality Knapsack Problem. The aim here is the development of a test suite optimization framework that could optimize the model-based test suites by removing the redundant test cases. An evolution-based algorithm is incorporated into the framework and is compared with the performances of two other algorithms. An empirical study is conducted with four synthetic and industrial scale Activity Diagram models and results are presented.

The Target Speed Monitor (TSM), which performs the safety critical function of the Chinese Train Control System Level 3 (CTCS-3), is characterized by large parameter input field and fault modes triggered by complicated combinations of parameters. Although combinatorial testing has been successfully applying in testing TSM, redundant test cases are generated because of covering the parameter combinations without interaction, which leads to the problem of high test execution costs. In this paper, an automatic reduction methodology based on decision tree equivalence and test suite output labels is presented. With previous combinatorial test suite generation experiment results of CTCS-3 TSM function, after data preprocessing, some data set have been constructed by test suites and corresponding output labels and they can be used to infer decision trees. After that, the reduction algorithm was used to reduce the data set to realize the test suite reduction. Related experimental results show that this method can achieve a reduction rate of up to 90%, and at the same time, the low-level combination coverage and the fault detection capability of reduced test suites are almost the same as that before the reduction.

Test minimization techniques aim at identifying and eliminating redundant test cases from test suites in order to reduce the total number of test cases to execute, thereby improving the efficiency of testing. In the context of software product line, we can save effort and cost in the selection and minimization of test cases for testing a specific product by modeling the product line. However, minimizing the test suite for a product requires addressing two potential issues: 1) the minimized test suite may not cover all test requirements compared with the original suite; 2) the minimized test suite may have less fault revealing capability than the original suite. In this paper, we apply weight-based Genetic Algorithms (GAs) to minimize the test suite for testing a product, while preserving fault detection capability and testing coverage of the original test suite. The challenge behind is to define an appropriate fitness function, which is able to preserve the coverage of complex testing criteria (e.g., Combinatorial Interaction Testing criterion). Based on the defined fitness function, we have empirically evaluated three different weight-based GAs on an industrial case study provided by Cisco Systems, Inc. Norway. We also presented our results of applying the three weight-based GAs on five existing case studies from the literature. Based on these case studies, we conclude that among the three weight-based GAs, Random-Weighted GA (RWGA) achieved significantly better performance than the other ones.

Data flow testing has recently attracted new interest in the context of testing object oriented systems, since data flow information is well suited to capture relations among the object states, and can thus provide useful information for testing method interactions. Unfortunately, classic data flow testing, which is based on static analysis of the source code, fails to identify many important data flow relations due to the dynamic nature of object oriented systems. In this paper, we propose a new technique to generate test cases for object oriented software. The technique exploits useful inter-procedural data flow information extracted dynamically from execution traces for object oriented systems. The technique is designed to enhance an initial test suite with test cases that exercise complex state based method interactions. The experimental results indicate that dynamic data flow testing can indeed generate test cases that exercise relevant behaviors otherwise missed by both the original test suite and by test suites that satisfy classic data flow criteria.

An empirical comparison of four Java-based regression test selection techniques

Guest Editorial: Special Section from the 11th International Conference on Quality Software (QSIC 2011)

A safe regression testing approach for safety critical systems

In development of software applications Graphical User Interface (GUI) plays an important role. It contains many test cases where some are useful and others are unusable test cases. Testing is used to recover bugs from the software. GUI needs to perform regular testing because it often requires frequent changes which depend on requirements of the users. This type of application requires regression testing which can be tested frequently when the modification occurs. The old version has a test suite and the modified version has a new suite. We focus on the part where the modified test suite is affecting the old test suite by making them unfeasible. The empirical study of these test cases is that to make the unfeasible test cases to feasible test case like reusing of the test paths by varying input values. We probably find where the input values are making the old test suite to reuse in the modified test suite. This approach can make more efficient by reducing the cost of regression testing and consumption of less time for the new modified application.