Regression Test Selection Tool for Python in Continuous Integration Process

Developers rely on regression testing in their continuous integration (CI) environment to find changes that introduce regression faults. While regression testing is widely practiced, it can be costly. Regression test selection (RTS) reduces the cost of regression testing by not running the tests that are unaffected by the changes. Industry has adopted module-level RTS for their CI environment, while researchers have proposed class-level RTS. In this paper, we compare module-and class-level RTS techniques in a cloud-based CI environment, Travis. We also develop and evaluate a hybrid RTS technique that combines aspects of the module-and class-level RTS techniques. We evaluate all the techniques on real Travis builds. We find that the RTS techniques do save testing time compared to running all tests (RetestAll), but the percentage of time for a full build using RTS (76.0%) is not as low as found in previous work, due to the extra overhead in a cloud-based CI environment. Moreover, we inspect test failures from RetestAll builds, and although we find that RTS techniques can miss to select failed tests, these test failures are almost all flaky test failures. As such, RTS techniques provide additional value in helping developers avoid wasting time debugging failures not related to the recent code changes. Overall, our results show that RTS can be beneficial for the developers in the CI environment, and RTS not only saves time but also avoids misleading developers by flaky test failures.

International audience

Today's advanced agile practices such as Continuous Integration and Test-Driven Development support a wide range of software development activities to facilitate the rapid delivery of high-quality software. However, the reuse of pre-existing, third-party software components is not one of them. Software reuse is still primarily perceived as a time-consuming, unsystematic and ultimately, discontinuous activity even though it aims to deliver the same basic benefits as continuous software engineering - namely, a reduction in the time and effort taken to deliver quality software. However, the increasingly central role of testing in continuous software engineering offers a way of addressing this problem by exploiting the new generation of test-driven search engines that can harvest components based on tests. This search technology not only exploits artifacts that have already been created as part of the continuous testing process to harvest components, it returns results that have a high likelihood of being fit for purpose and thus of being worth reusing. In this paper, we propose to augment continuous software engineering with the rapid, continuous reuse of software code units by integrating the test-driven mining of software artifact repositories into the continuous integration process. More specifically, we propose to use tests written as part of the Test-First Development approach to perform test-driven searches for matching functionality while developers are working on their normal development activities. We discuss the idea of rapid, continuous code reuse based on recent advances in our test-driven search platform and elaborate on scenarios for its application in the future.

Multi-objective regression test selection in practice: An empirical study in the defense software industry

When encountering multiple and complicated models, multisource structured and unstructured data, complex requirements analysis, the platform design and integration of hydroinformatics systems become a challenge. To properly solve these problems, we describe a distributed software framework and it’s continuous integration process in hydroinformatics systems. This distributed framework mainly consists of server cluster for models, distributed database, GIS (Geographic Information System) servers, master node and clients. Based on it, a GIS - based decision support system for joint regulating of water quantity and water quality of group lakes in Wuhan China is established.

Regression testing as an essential activity in software development that has changed requirements. In practice, regression testing requires a lot of time so that an optimal strategy is needed. One approach that can be used to speed up execution time is the Regression Test Selection (RTS) approach. Currently, practitioners and academics have started to think about developing tools to optimize the process of implementing regression testing. Among them, STARTS and Ekstazi are the most popular regression testing tools among academics in running test case selection algorithms. This article discusses the comparison of the capabilities of the STARTS and Ekstazi features by using feature parameter evaluation. Both tools were tested with the same input data in the form of System Under Test (SUT) and test cases. The parameters used in the tool comparisons are platform technology, test case selection, functionality, usability and performance efficiency, the advantages, and disadvantages of the tool. he results of the trial show the differences and similarities between the features of STARTS and Ekstazi, so that it can be used by practitioners to take advantage of tools in the implementation of regression testing that suit their needs. In addition, experimental results show that the use of Ekstazi is more precise in sorting out important test cases and is more efficient, when compared to STARTS and regression testing with retest all.

This paper covers the challenges in software validation, especially in the context of complex, embedded systems. Problems related to limited validation environment resources (DUT - Devices Under Test, workload and traffic generators, measurement setups, test probes), stability and complexity of systems will be highlighted. Authors will describe validation strategy which responds to quality assurance and risk management requirements applied for real products provided to the market. Presented technologies covers solutions based on Continuous Integration and Continuous Delivery process of modern, complex, embedded systems. Authors present both known and new solutions and approaches to known challenges, also suggest possible further development directions to improve efficiency and optimize product development cost.

Regression test selection (i.e., selecting a subset of a given regression test suite) is a problem that has been studied intensely over the last decade. However, with the increasing popularity of developer tests as the driver of the test process, more fine-grained solutions are in order. In this paper we investigate how method-level changes in the base-code can serve as a reliable indicator for identifying which tests need to be rerun. We validate the approach on two cases - PMD and Cruise Control - using mutation testing as a means to compare the selected subset against a retest all approach. Our results show that we are able to reach a sizable reduction of the complete test suite, yet with a comparable number of mutants killed by the reduced test suite.

A decentralized variant of continuous integration can be defined in terms of two fundamental rules: (1) Developers’ access to add contributions to the development version at any time, and (2) developers’ obligation to integrate their own contributions properly. Decentralized, continuous integration may adapt well to organizations where developers work relatively independently, as in many open source projects. The approach raises the issue of how these organizations can exercise central control, as attaining the benefits of continuous integration requires that contributions are useful and satisfy the project’s definition of successful integration. We have investigated the use of continuous integration in FreeBSD and Mozilla. Our account of quality assurance activities in the two open source projects distinguishes between Mintzberg’s three complementary forms of central control: Standardization and control of work output, work processes, and worker skills. Our study indicates that two major challenges face projects using decentralized, continuous integration: (1) To balance the access to add contributions against the need to stabilize and mature the software prior to a release, and (2) to consider the developers’ limited time and resources when interpreting their obligation to integrate their changes properly.

Faults in the automated continuous integration (CI) process can seriously impact the development of industrial code. To reduce manual intervention in automated CI processes, we want to understand better the CI systems' failure distribution to improve efficiency, reliability, and maintainability. This paper investigates failures in CI in four large industrial projects. We gather 11 731 builds over six months, identifying 1 414 failing builds. We also identify the distribution of different types of build failures in each of the four CI projects. Our results show that compilation is the most significant individual cause of failure with 47 %, followed by testing at 36 %. The checkout step with associated checks also incurs a non-negligible portion of failures with 12 %. Furthermore, we identify 14 distinct types of failures in the testing step. We conclude that configuration problems are a significant issue, as pipeline scripting and dependency errors make up a large number of failures.

Allure Reports, an open-source framework, significantly enhance the visualization and comprehensibility of test results in software testing. This paper explores the integration of Allure Reports with GitHub Workflows to automate the generation, publication, and management of detailed and interactive test reports. The primary focus is on improving communication, debugging processes, and overall testing efficiency through auto- mated, real-time reporting solutions. Key features of Allure Re- ports include interactive visualizations, comprehensive overviews, detailed test case information, customizable reports, and seamless integration with Continuous Integration/Continuous Deployment (CI/CD) pipelines. Additionally, the paper discusses the role of GitHub Pages and GitHub Artifacts in making test results accessible and supporting continuous integration and delivery processes. The implementation of this integration resulted in enhanced transparency, collaboration, and efficiency within development teams, as well as improved test result visibility and debugging capabilities. Empirical studies support our findings, highlighting improvements in task efficiency and reduced debugging time through the use of automated reporting and continuous integration practices [1- 3].

As part of agile methodologies seen in the past few years, IT organizations have continuously adopted new practices in their software delivery life-cycle to improve both efficiency and effectiveness of development teams. Two of these practices are continuous integration and continuous deployment, which are part of the DevOps cycle which has helped organizations build software effectively and efficiently. These practices must be considered for new technologies such as smart contracts, where security concerns and bugs might cost more once deployed than traditional software. This paper states the importance of using a proper DevOps routine and how it is possible to apply this practice to a smart contract build. Specifically, this paper introduces a framework to implement DevOps for smart contracts development by describing multiple DevOps tools and their applicability to smart contract development.

We propose a new approach to solving important practical problems of complex debugging, joint testing, and analysis of the execution time of software module versions in a heterogeneous distributed computing environment that integrating Grid and cloud computing. These problems arise in the process of supporting the continuous integration of modules of distributed applied software packages. The study focuses on the packages that are used to conduct large-scale computational experiments. The scientific novelty of the proposed approach is to combine the methodology for creating the packages with modern software development practices based on its continuous integration using knowledge about the specifics of the problems being solved. Our contribution is multifold. We expanded the capabilities of continuous integration tools by developing new additional tools for the markup and transformation of data from poorly structured sources and predicting modules execution time. In addition, we developed a technological scheme of the joint applying our developed tools and external systems for continuous integration. Therefore, we provide a more large range of capabilities of continuous integration in relation to the processes of creating and using the packages in comparison with the well-known tools. The fundamental basis of their functioning is a new conceptual model of the packages. This model supports the specification, planning, and execution of software continuous integration processes taking into account the specific subject data and problems being solved. Applying the developed tools in practice leads to a decrease in the number of errors and failures of applied software in the development and use of the packages. In turn, such decrease significantly reduces the time for large-scale computational experiments and increases the efficiency of using resources of the environment. The results of practical experiments on the use of system prototype for continuous integration of applied software show their high efficiency.

Background: Modern Code Reviews (MCR) are frequently adopted when assuring code and design quality in continuous integration and deployment projects. Although tiresome, they serve a secondary purpose of learning about the software product. Aim: Our objective is to design and evaluate a support tool to help software developers focus on the most important code fragments to review and provide them with suggestions on what should be reviewed in this code. Method: We used design science research to develop and evaluate a tool for automating code reviews by providing recommendations for code reviewers. The tool is based on Transformer-based machine learning models for natural language processing, applied to both programming language code (patch content) and the review comments. We evaluate both the ability of the language model to match similar lines and the ability to correctly indicate the nature of the potential problems encoded in a set of categories. We evaluated the tool on two open-source projects and one industry project. Results: The proposed tool was able to correctly annotate (only true positives) 35%–41% and partially correctly annotate 76%–84% of code fragments to be reviewed with labels corresponding to different aspects of code the reviewer should focus on. Conclusion: By comparing our study to similar solutions, we conclude that indicating lines to be reviewed and suggesting the nature of the potential problems in the code allows us to achieve higher accuracy than suggesting entire changes in the code considered in other studies. Also, we have found that the differences depend more on the consistency of commenting rather than on the ability of the model to find similar lines.

Continuous delivery of cloud systems requires constant running of jobs (build processes, tests, etc.). One issue that plagues this continuous integration (CI) process are intermittent failures - non-deterministic, false alarms that do not result from a bug in the software or job specification, but rather from issues in the underlying infrastructure. At Mozilla, such intermittent failures are called oranges as a reference to the color of the build status indicator. As such intermittent failures disrupt CI and lead to failures, they erode the developers' trust in the jobs. We present a novel approach that automatically classifies failing jobs to determine whether job execution failures arise from an actual software bug or were caused by flakiness in the job (e.g., test) or the underlying infrastructure. For this purpose, we train classification models using job telemetry data to diagnose failure patterns involving features such as runtime, cpu load, operating system version, or specific platform with high precision. In an evaluation on a set of Mozilla CI jobs, our approach achieves precision scores of 73%, on average, across all data sets with some test suites achieving precision scores good enough for fully automated classification (i.e., precision scores of up to 100%), and recall scores of 82% on average (up to 94%).