Automated code-based test selection for software product line regression testing

Regression testing as an essential activity in software development that has changed requirements. In practice, regression testing requires a lot of time so that an optimal strategy is needed. One approach that can be used to speed up execution time is the Regression Test Selection (RTS) approach. Currently, practitioners and academics have started to think about developing tools to optimize the process of implementing regression testing. Among them, STARTS and Ekstazi are the most popular regression testing tools among academics in running test case selection algorithms. This article discusses the comparison of the capabilities of the STARTS and Ekstazi features by using feature parameter evaluation. Both tools were tested with the same input data in the form of System Under Test (SUT) and test cases. The parameters used in the tool comparisons are platform technology, test case selection, functionality, usability and performance efficiency, the advantages, and disadvantages of the tool. he results of the trial show the differences and similarities between the features of STARTS and Ekstazi, so that it can be used by practitioners to take advantage of tools in the implementation of regression testing that suit their needs. In addition, experimental results show that the use of Ekstazi is more precise in sorting out important test cases and is more efficient, when compared to STARTS and regression testing with retest all.

Regression testing, which is performed after changes are made to a software system, can be used before release of new versions of the system. However, practitioners often have little time to perform this regression testing because of the quick-release cycles of such modified systems. Thus, they may use a random-testing approach or perform little regression testing. This lack of adequate regression testing can cause bugs in untested parts of the program to be exposed only during production or field usage. To improve the efficiency of the regression testing, and thus enable its use before release, techniques that select and run only those test cases that are related to the changes or prioritize the test cases based on criticality or perceived effectiveness have been presented. These technique typically use some representation of the software such as a system model or the source code to perform the test selection and prioritization. However, in practice, access to a system modelor the source code may not be possible. To provide regression test selection in practice, we have developed, and present in this paper, a novel approach to regression test selection that, instead of using a system model or the source code, uses the system requirements and their associated test cases, which are typically available to developers/testers. The approach uses the set of system requirements, usually in natural language or some informal notation, that represent what is to be tested about the system. The technique uses these requirements, along with the set of test cases and their criticality that are associated with the requirements, to select test cases for use in regression testing. In this paper, we also present a case study that shows the potential effectiveness of our technique

In many cases, Software Product Line Testing (SPLT) targets only the selection of test cases which cover product features or feature interactions. However, higher testing efficiency can be achieved through the selection of test cases with improved fault-revealing capabilities. By associating each test case a priority-value representing (or aggregating) different criteria, such as importance (in terms of fault discovered in previous test campaigns), duration or cost, it becomes possible to select a feature-covering test suite with improved capabilities. A crucial objective in SPLT then becomes to identify a test suite that optimizes reaching a specific goal (lower test duration or cost), while preserving full feature coverage.

Software testing (testing) is a crucial stage in software development. The success of the testing process will ensure the quality of the software. In the regression testing process, one issue is that not all test cases (retest all) in the test suite need to be executed. Retest all will consume massive resources, as well as a long time. Regression testing techniques seek to find ways to reduce test execution time. One of the regression testing techniques is test case selection, also known as regression test selection (RTS). This paper describes a study on babelRTS, an RTS algorithm, to see its effectiveness. Effectiveness is measured by comparing the execution time of the execution retest all and babelRTS. Experiments were carried out on five software under tests (SUT) that had some faults. Test cases are prepared by designing for each SUT. The results showed a reduction in time so that the effectiveness reached a maximum of 32%, and average of 23% .

Background: In today’s era, modifications in a software is a common requirement by customers. When changes are made to existing software, re-testing of all the test cases is required to ensure that the newly introduced changes do not have any unwanted effect on the behavior of the software. However, retesting of all the test cases would not only be time consuming but also expensive. Therefore, there is a need for a technique that reduces the number of tests to be performed. Regression testing is one of the ways to reduce the number of test cases. Selection technique is one such method which seeks to identify the test cases that are relevant to some set of recent changes. Objective: It is evident that most of the studies have used different selection techniques and have focused only on one parameter for achieving reduced test suite size without compromising the performance of regression testing. However, to the best of our knowledge, no study has taken two or more parameters of coverage, and/or execution time in a single testing. This paper presents a hybrid technique that combines both regression test selection using slicing technique and minimization of test cases using modified firefly algorithm with combination of parameters coverage and execution time in a single testing. Methods: A hybrid technique has been described that combines both selection and minimization. Selection of test cases is based upon slicing technique while minimization is done using firefly algorithm. Hybrid technique selects and minimizes the test suite using information on statement coverage and execution time. Results: The proposed technique gives 43.33% much superior result as compared to the other hybrid approach in terms of significantly reduced number of test cases. It shows that the resultant test cases were effective enough to cover 100% of the statements, for all the programs. The proposed technique was also tested on four different programs namely Quadratic, Triangle, Next day, Commission respectively for test suite selection and minimization which gave comparatively superior result in terms of reduction (%) in number of test cases required for testing. Conclusion: The combination of parameters used in slicing based approach, reduces the number of test cases making software testing an economical, feasible and time saving option without any fault in the source code. This proposed technique can be used by software practitioners/experts to reduce time, efforts and resources for selection and minimization of test cases.

The process of regression testing is generally done under critical time and resource constraints. In the existing approaches, Test Case Prioritization (TCP), Test Case Selection (TCS), and Test Suite Reduction (TSR) have been treated as different optimization problems. This is based on simple heuristics that do not use valuable information that can be extracted from the program under test. Moreover, they do not allow for making internally consistent budget-conscious decisions. This paper introduces a unified approach that combines various variants of Graph Neural Networks (GNNs) and Reinforcement Learning (RL) for the solutions of TCP, TCS, and TSR. In our proposal, the representation of regression artifacts such as test cases, code entities, and faults is used as nodes in the typed graph, while the edges are used to show the relations between these artifacts. The relation-aware GNN is used for generating test case embeddings, reflecting the distance between the test cases and the changed areas as well as the areas known to have issues in the past. The actor-critic RL agent uses these test case embeddings and the budget to decide whether to run, skip, or discard test cases. The performance of our proposal outstrips coverage-based heuristics, history-based ranking, a GA-based search, an enhanced QPSO, and two learning-based ablations, as validated by our experiments on four Java projects from Defects4J. In terms of various programs and budgets, using GNN–RL has led to an improvement in the Average Percentage of Faults Detected (APFD) and Cost-cognizant Average Percentage of Faults Detected (APFDc) by approximately 6–9 percentage points on lower budgets. In addition, it is possible to obtain a 42% suite reduction and a 48% cost reduction while still retaining 98% of all fault-revealing tests in the final suite. The results show that it is possible to obtain promising solutions for adaptive regression testing using budget-aware reinforcement learning and graph-based representation learning.

The unified modeling language (UML) designs come in a variety of different notations. UML designs can be quite large and interactions between various notations and the models they define can be difficult to assess. During the design phase, and between successive releases of systems, designs change. The impact of such changes and the resulting effect on behavior can be non-obvious and difficult to assess. This survey article explores techniques for such re-evaluation that can be classified as regression testing and suggests regression testing criteria for designs. These might vary depending on testing objectives and include both selective and regenerative regression testing approaches. The article provides a concise overview of regression testing approaches related to various UML design notations including use cases, class diagrams, sequence diagrams, activity diagrams, and statecharts, as well as combinations of these models. It discusses UML-related techniques involving cost and prioritization during selective regression testing. Finally, it evaluates these techniques with respect to inclusiveness, precision, efficiency, generality, accountability, and safety. An example is used throughout to illustrate how the techniques work. Copyright © 2009 John Wiley & Sons, Ltd. This survey article explores techniques for regression testing with UML designs and suggests regression testing criteria for designs. These might vary depending on testing objectives and include both selective and regenerative regression testing approaches. The article provides a concise overview of regression testing approaches related to various UML design notations including use cases, class diagrams, sequence diagrams, activity diagrams, and statecharts, as well as combinations of these models. Copyright © 2009 John Wiley & Sons, Ltd.

Software product line (SPL) testing consists of two separate but closely related test engineering activities: domain testing and application testing. Various software product line testing approaches have been developed over the last decade, and surveys have been conducted on them. However, thus far none of them deeply addressed the questions of what researches have been conducted in order to overcome the challenges posed by the two separate testing activities and their relationships. Thus, this paper surveys the current software product line testing approaches by defining a reference SPL testing processes and identifying, based on them, key research perspectives that are important in SPL testing. Through this survey, we identify the researches that addressed the challenges and also derive open research opportunities from each perspective.

Software product line (SPL) testing is a challenging task, due to the huge number of variants sharing common functionalities to be taken into account for efficient testing. By adopting the concept of regression testing, incremental SPL testing strategies cope with this challenge by exploiting the reuse potential of test artifacts between subsequent variants under test. In previous work, we proposed delta-oriented test case prioritization for incremental SPL integration testing, where differences between architecture test model variants allow for reasoning about the order of reusable test cases to be executed. However, the prioritization left two issues open, namely (1) changes to component behavior are ignored, which may also influence component interactions and, (2) the weighting and ordering of similar test cases result in an unintended clustering of test cases. In this paper, we extend the test case prioritization technique by (1) incorporating changes to component behavior allowing for a more fine-grained analysis and (2) defining a dissimilarity measure to avoid clustered test case orders. We prototyped our test case prioritization technique and evaluated its applicability and effectiveness by means of a case study from the automotive domain showing positive results.

In recent years, Automated Program Repair (APR) has been extensively studied in academia and even drawn wide attention from the industry. However, APR techniques can be extremely time consuming since (1) a large number of patches can be generated for a given bug, and (2) each patch needs to be executed on the original tests to ensure its correctness. In the literature, various techniques (e.g., based on learning, mining, and constraint solving) have been proposed/studied to reduce the number of patches. Intuitively, every patch can be treated as a software revision during regression testing; thus, traditional Regression Test Selection (RTS) techniques can be leveraged to only execute the tests affected by each patch (as the other tests would keep the same outcomes) to further reduce patch execution time. However, few APR systems actually adopt RTS and there is still a lack of systematic studies demonstrating the benefits of RTS and the impact of different RTS strategies on APR. To this end, this article presents the first extensive study of widely used RTS techniques at different levels (i.e., class/method/statement levels) for 12 state-of-the-art APR systems on over 2M patches. Our study reveals various practical guidelines for bridging the gap between APR and regression testing, including: (1) the number of patches widely used for measuring APR efficiency can incur skewed conclusions, and the use of inconsistent RTS configurations can further skew the conclusions; (2) all studied RTS techniques can substantially improve APR efficiency and should be considered in future APR work; (3) method- and statement-level RTS outperform class-level RTS substantially and should be preferred; (4) RTS techniques can substantially outperform state-of-the-art test prioritization techniques for APR, and combining them can further improve APR efficiency; and (5) traditional Regression Test Prioritization (RTP) widely studied in regression testing performs even better than APR-specific test prioritization when combined with most RTS techniques. Furthermore, we also present the detailed impact of different patch categories and patch validation strategies on our findings.

Abstract—Abstract Test Cases (ATCs) have been widely used in practice, including in combinatorial testing and in software product line testing. When constructing a set of ATCs, due to limited testing resources in practice (for example in regression testing), Test Case Prioritization (TCP) has been proposed to improve the testing quality, aiming at ordering test cases to increase the speed with which faults are detected. One intuitive and extensively studied TCP technique for ATCs is λ-wise Level-combination Coverage based Prioritization (λLCP), a static, black-box prioritization technique that only uses the ATC information to guide the prioritization process. A challenge facing λLCP, however, is the necessity for the selection of the fixed prioritization strength λ before testing — testers need to choose an appropriate λ value before testing begins. Choosing higher λ values may improve the testing effectiveness of λLCP (for example, by finding faults faster), but may reduce the testing efficiency (by incurring additional prioritization costs). Conversely, choosing lower λ values may improve the efficiency, but may also reduce the effectiveness. In this paper, we propose a new family of λLCP techniques, Repeated Small-strength Level-combination Coverage-based Prioritization (RSLCP), that repeatedly achieves the full combination coverage at lower strengths. RSLCP maintains λLCP’s advantages of being static and black box, but avoids the challenge of prioritization strength selection. We performed an empirical study involving five different versions of each of five C programs. Compared with λLCP, and Incremental strength LCP (ILCP), our results show that RSLCP could provide a good trade-off between testing effectiveness and efficiency. Our results also show that RSLCP is more effective and efficient than two popular techniques of Similarity-based Prioritization (SP). In addition, the results of empirical studies also show that RSLCP can remain robust over multiple system releases.

Regression testing of the software during its maintenance phase, requires test case prioritization and selection due to the dearth of the allotted time. The resources and the time in this phase are very limited, thus testers tend to use regression testing methods such as test case prioritization and selection. The current study evaluates the effectiveness of testing with two major goals: (1) Least running time and (2) Maximum fault coverage possible. Ant Colony Optimization (ACO) is a well-known soft computing technique that draws its inspiration from nature and has been widely researched, implemented, analyzed, and validated for regression test prioritization and selection. Many versions of ACO approaches have been prolifically applied to find solutions to many non-polynomial time-solvable problems. Hence, an attempt has been made to enhance the performance of the existing ACO_TCSP algorithm without affecting its time complexity. There have been efforts to enhance the exploration space of various paths in each iteration and with elite exploitation, reducing the total number of iterations required to converge to an optimal path. Counterbalancing enhanced exploration with intelligent exploitation implies that the run time is not adversely affected, the same has also been empirically validated. The enhanced algorithm has been compared with the existing ACO algorithm and with the traditional approaches. The approach has also been validated on four benchmark programs to empirically evaluate the proposed Enhanced ACO_TCSP algorithm. The experiment revealed the increased cost-effectiveness and correctness of the algorithm. The same has also been validated using the statistical test (independent t-test). The results obtained by evaluating the proposed approach against other reference techniques using Average Percentage of Faults Detected (APFD) metrics indicate a near-optimal solution. The multiple objectives of the highest fault coverage and least running time were fruitfully attained using the Enhanced ACO_TCSP approach without compromising the complexity of the algorithm.

Regression testing is very important process in software maintenance. Unfortunately, it is costly and time consuming to allow for the re-execution of all test cases during regression testing. The challenge in regression testing is the selection of best test cases from the existing test suite.This paper presents an evolutionary regression test case prioritization for object-oriented software based on extended system dependence graph model of the affected program using genetic algorithm. The approach is based on optimization of selected test case from dependency analysis of the source codes. The goal is to identify changes in a method's body due to data dependence, control dependence and dependent due to object relation such as inheritance and polymorphism, select the test cases based on affected statements and ordered them based on their fitness by using GA.The number of affected statements determined how fit a test case is good for regression testing. A case study is reported to provide evidence of the feasibility of the approach and its benefits in increasing the rate of fault detection and reduction in regression testing effort compared with retest-all. It was shown that our approach needs 30% of the test cases to cover all the faults, while 80% is needed to cover all the faults using retest-all, which is time consuming and costly.

Software product family or line is a software engineering paradigm that systematizes reuse. In software product line engineering, two phases are distinguished: domain engineering which is in charge of developing a common infrastructure and assets and application engineering which makes use of those assets to generate the products. One of the key aspects of product lines is variability and its management. However, the main focus has been on functional variability and quality variability in software product lines has not received so much attention by researchers. In a product line different members of the line may require different levels of a quality requirement, for instance they could differ in terms of their availability, security, reliability, etc. Due to this variability, quality evaluation in software product lines is much more complicated that in single-systems. One alternative is to evaluate all the products of a line but it is very expensive and ways of reducing evaluation efforts are necessary. In this direction, the paper presents a method for facilitating cost-effective quality evaluation of a product line taking into consideration variability on quality attributes.

Software Product Lines (SPLs) are not intended to create one application, but a number of them: a product family. In contrast to one-off development, SPLs are based on the idea that the distinct products of the family share a significant amount of assets. This forces a change in how software is developed. Likewise, software testing should mimic its code counterpart: product testing should also be produced out of a common set of assets. Specifically, this paper addresses how model-driven testing, used for one-off development, can be moved to an SPL setting. We focus on feature-oriented software development as the SPL realization technique. UML sequence diagrams are used to represent the common and feature scenarios. This models are transformed through model transformations to obtain test cases that conform to the UML Testing Profile.