Understanding and Improving Regression Test Selection in Continuous Integration

In recent years, Automated Program Repair (APR) has been extensively studied in academia and even drawn wide attention from the industry. However, APR techniques can be extremely time consuming since (1) a large number of patches can be generated for a given bug, and (2) each patch needs to be executed on the original tests to ensure its correctness. In the literature, various techniques (e.g., based on learning, mining, and constraint solving) have been proposed/studied to reduce the number of patches. Intuitively, every patch can be treated as a software revision during regression testing; thus, traditional Regression Test Selection (RTS) techniques can be leveraged to only execute the tests affected by each patch (as the other tests would keep the same outcomes) to further reduce patch execution time. However, few APR systems actually adopt RTS and there is still a lack of systematic studies demonstrating the benefits of RTS and the impact of different RTS strategies on APR. To this end, this article presents the first extensive study of widely used RTS techniques at different levels (i.e., class/method/statement levels) for 12 state-of-the-art APR systems on over 2M patches. Our study reveals various practical guidelines for bridging the gap between APR and regression testing, including: (1) the number of patches widely used for measuring APR efficiency can incur skewed conclusions, and the use of inconsistent RTS configurations can further skew the conclusions; (2) all studied RTS techniques can substantially improve APR efficiency and should be considered in future APR work; (3) method- and statement-level RTS outperform class-level RTS substantially and should be preferred; (4) RTS techniques can substantially outperform state-of-the-art test prioritization techniques for APR, and combining them can further improve APR efficiency; and (5) traditional Regression Test Prioritization (RTP) widely studied in regression testing performs even better than APR-specific test prioritization when combined with most RTS techniques. Furthermore, we also present the detailed impact of different patch categories and patch validation strategies on our findings.

An empirical comparison of four Java-based regression test selection techniques

Regression testing is an important but expensive activity in software development. Among various types of tests, web service tests are usually one of the most expensive (due to network communications) but widely adopted types of tests in commercial software development. Regression test selection (RTS) aims to reduce the number of tests which need to be retested by only running tests that are affected by code changes. Although a large number of RTS techniques have been proposed in the past few decades, these techniques have not been adopted on large-scale web service testing. This is because most existing RTS techniques either require direct code dependency between tests and code under test or cannot be applied on large scale systems with enough efficiency. In this paper, we present a novel RTS technique, TestSage, that performs RTS for web service tests on large scale commercial software. With a small overhead, TestSage is able to collect fine grained (function level) dependency between test and service under test that do not directly depend on each other. TestSage has also been successfully applied to large complex systems with over a million functions. We conducted experiments of TestSage on a large scale backend service at Google. Experimental results show that TestSage reduces 34% of testing time when running all AEC (Analysis, Execution and Collection) phases, 50% of testing time while running without collection phase. TestSage has been integrated with internal testing framework at Google and runs day-to-day at the company.

Multi-objective regression test selection in practice: An empirical study in the defense software industry

Software products are often configured with commercial-off-the-shelf (COTS) components. When new releases of these components are made available for integration and testing, source code is usually not provided. Various regression test selection processes have been developed and have been shown to be cost effective. However, the majority of these test selection techniques rely on access to source code for change identification. Based on our prior work, we are studying the solution to regression testing COTS-based applications that incorporate components of dynamic link library (DLL) files. We evolved the Integrated - Black-box Approach for Component Change Identification (I-BACCI) process that selects regression tests for applications based upon static binary code analysis to Version 4 to support DLL components. A feasibility case study was conducted at ABB on products written in C/C++ to determine the effectiveness of the I-BACCI process. The results of the case study indicate this process can reduce the required number of regression tests by as much as 100% if our analysis indicates the changes to the component are not called by the glue code of the application using the COTS component. Similar to other regression test selection techniques, when there are many changes in the new component I-BACCI suggests a retest-all regression test strategy.

Regression test selection is to select a subset of existing tests to run, so as to identify the possible faults in the modified program. A promising regression test selection technique needs to be safe, that is, to select tests from the original test suite that can expose faults in the modified program under controlled regression testing. Existing safe regression test selection techniques however may rerun some tests that do not expose faults. To address this problem, we present in this paper a new regression test selection technique based on program dependence graphs of a program and its modified version. Comparing with previous techniques, our technique can eliminate some unnecessary tests to rerun. We show the validity and feasibility of our approach through a running example.

The unified modeling language (UML) designs come in a variety of different notations. UML designs can be quite large and interactions between various notations and the models they define can be difficult to assess. During the design phase, and between successive releases of systems, designs change. The impact of such changes and the resulting effect on behavior can be non-obvious and difficult to assess. This survey article explores techniques for such re-evaluation that can be classified as regression testing and suggests regression testing criteria for designs. These might vary depending on testing objectives and include both selective and regenerative regression testing approaches. The article provides a concise overview of regression testing approaches related to various UML design notations including use cases, class diagrams, sequence diagrams, activity diagrams, and statecharts, as well as combinations of these models. It discusses UML-related techniques involving cost and prioritization during selective regression testing. Finally, it evaluates these techniques with respect to inclusiveness, precision, efficiency, generality, accountability, and safety. An example is used throughout to illustrate how the techniques work. Copyright © 2009 John Wiley & Sons, Ltd. This survey article explores techniques for regression testing with UML designs and suggests regression testing criteria for designs. These might vary depending on testing objectives and include both selective and regenerative regression testing approaches. The article provides a concise overview of regression testing approaches related to various UML design notations including use cases, class diagrams, sequence diagrams, activity diagrams, and statecharts, as well as combinations of these models. Copyright © 2009 John Wiley & Sons, Ltd.

Genetic Improvement (GI) uses search-based optimisation algorithms to automatically improve software with respect to both functional and non-functional properties. Our previous work showed that Regression Test Selection (RTS) can help speed up the use of GI and enhance the overall results while not affecting the software system’s validity. This article expands upon our investigation by answering further questions about safety and applying a GI algorithm based on Local Search (LS) in addition to the previously explored Genetic Programming (GP) approach. Further, we extend the number of subjects to 12 by analysing five larger real-world open-source programs. We empirically compare two state-of-the-art RTS techniques combined with GP and LS for these 12 programs. The results show that both RTS techniques are safe to use and can reduce the cost of GI by up to 80% and by 31% on average across programs. We also observe that both search-based algorithms impact the effectiveness gains of GI differently, and that various RTS strategies achieve differing gains in terms of efficiency. These results serve as further evidence that RTS must be used as a core component of the GI search process to maximise its effectiveness and efficiency.

Regression testing aims at showing that code has not been adversely affected by modification activities during maintenance. Regression test selection techniques reuse tests from an existing test suite to test a modified program. By reusing such a test suite to retest modified programs, maintainers or testers can reduce the required testing effort. This paper presents a regression test selection technique for AspectJ programs. The technique is based on various types of control flow graphs that can be used to select from the original test suite test cases that execute changed code for the new version of the AspectJ program. The code-base technique operates on the control flow graphs of AspectJ programs. The technique can be applied to modified individual aspects or classes as well as the whole program that uses modified aspects or classes.

Regression testing is an expensive maintenance process directed at validating modified software. Regression test selection techniques attempt to reduce the cost of regression testing by selecting tests from a program's existing test suite. Many regression test selection techniques have been proposed. Although there have been some analytical and empirical evaluations of individual techniques, to our knowledge only one comparative study, focusing on one aspect of two of these techniques, has been performed. We conducted an experiment to examine the relative costs and benefits of several regression test selection techniques. The experiment examined five techniques for reusing tests, focusing on their relative abilities to reduce regression testing effort and uncover faults in modified programs. Our results highlight several differences between the techniques, and expose essential tradeoffs that should be considered when choosing a technique for practical application.

Regression testing is a necessary but expensive maintenance activity aimed at showing that code has not been adversely affected by changes. Regression test selection techniques reuse tests from an existing test suite to test a modified program. Many regression test selection techniques have been proposed, however, it is difficult to compare and evaluate these techniques because they have different goals. This paper outlines the issues relevant to regression test selection techniques, and uses these issues as the basis for a framework within which to evaluate the techniques. The paper illustrates the application of the framework by using it to evaluate existing regression test selection techniques. The evaluation reveals the strengths and weaknesses of existing techniques, and highlights some problems that future work in this area should address.

In an end-to-end, regression testing framework employing a safe regression test selection (RTS) technique that uses control-flow graphs (CFGs) to model Web service interactions, service providers must share their CFGs to participate. However, CFGs are sensitive implementation details and service providers are unwilling to expose them especially across autonomous systems. In order to encourage participation in the proposed framework, several privacy-preserving techniques are employed designed to protect the sensitive information contained within CFGs while still maintaining the overall effectiveness of the approach. The privacy-preserving techniques protect the information contained within CFGs by sanitizing individual nodes and altering the shape of the CFG. A case study will be presented to help illuminate the framework and provide a measure of the overall effectiveness of the approach.

Regression testing - running tests after code modifications - is widely practiced in industry, including at Samsung. Regression Test Selection (RTS) optimizes regression testing by skipping tests that are not affected by recent code changes. Recent work has developed robust RTS tools, which mostly target managed languages, e.g., Java and C#, and thus are not applicable to large C projects, e.g., TizenRT, a lightweight RTOS-based platform. We present Selfection, an RTS tool for projects written in C; we discuss the key challenges to develop Selfection and our design decisions. Selfection uses the objdump and readelf tools to statically build a dependency graph of functions from binaries and detect modified code elements. We integrated Selfection in TizenRT and evaluated its benefits if tests are run in an emulator and on a supported hardware platform (ARTIK 053). We used the latest 150 revisions of TizenRT available on GitHub. We measured the benefits of Selfection as the reduction in the number of tests and reduction in test execution time over running all tests at each revision (i.e., RetestAll). Our results show that Selfection can reduce, on average, the number of tests to 4.95% and end-to-end execution time to 7.04% when tests are executed in the emulator, and to 5.74% and 26.82% when tests are executed on the actual hardware. Our results also show that the time taken to maintain the dependency graph and detect modified functions is negligible.

Regression testing, which is performed after changes are made to a software system, can be used before release of new versions of the system. However, practitioners often have little time to perform this regression testing because of the quick-release cycles of such modified systems. Thus, they may use a random-testing approach or perform little regression testing. This lack of adequate regression testing can cause bugs in untested parts of the program to be exposed only during production or field usage. To improve the efficiency of the regression testing, and thus enable its use before release, techniques that select and run only those test cases that are related to the changes or prioritize the test cases based on criticality or perceived effectiveness have been presented. These technique typically use some representation of the software such as a system model or the source code to perform the test selection and prioritization. However, in practice, access to a system modelor the source code may not be possible. To provide regression test selection in practice, we have developed, and present in this paper, a novel approach to regression test selection that, instead of using a system model or the source code, uses the system requirements and their associated test cases, which are typically available to developers/testers. The approach uses the set of system requirements, usually in natural language or some informal notation, that represent what is to be tested about the system. The technique uses these requirements, along with the set of test cases and their criticality that are associated with the requirements, to select test cases for use in regression testing. In this paper, we also present a case study that shows the potential effectiveness of our technique

Selective regression testing—assumptions and fault detecting ability