Practical Flaky Test Prediction using Common Code Evolution and Test History Data

Feature models are frequently used to capture the knowledge about configurable software systems and product lines. However, feature modeling of large-scale systems is challenging as many models are needed for diverse purposes. For instance, feature models can be used to reflect the perspectives of product management, technical solution architecture, or product configuration. Furthermore, models are required at different levels of granularity. Although numerous approaches and tools are available, it remains hard to define the purpose, scope, and granularity of feature models. In this paper we thus present experiences of developing feature models for two large-scale industrial automation software systems. Specifically, we extended an existing feature modeling tool to support models for different purposes and at multiple levels. We report results on the characteristics and modularity of the feature models, including metrics about model dependencies. We further discuss lessons learned during the modeling process.

Flaky tests are test cases that can pass or fail without code changes. They often waste the time of software developers and obstruct the use of continuous integration. Previous work has presented several automated techniques for detecting flaky tests, though many involve repeated test executions and a lot of source code instrumentation and thus may be both intrusive and expensive. While this motivates researchers to evaluate machine learning models for detecting flaky tests, prior work on the features used to encode a test case is limited. Without further study of this topic, machine learning models cannot perform to their full potential in this domain. Previous studies also exclude a specific, yet prevalent and problematic, category of flaky tests: order-dependent (OD) flaky tests. This means that prior research only addresses part of the challenge of detecting flaky tests with machine learning. Closing this knowledge gap, this paper presents a new feature set for encoding tests, called Flake16. Using 54 distinct pipelines of data preprocessing, data balancing, and machine learning models for detecting both non-order-dependent (NOD) and OD flaky tests, this paper compares Flake16 to another well-established feature set. To assess the new feature set's effectiveness, this paper's experiments use the test suites of 26 Python projects, consisting of over 67,000 tests. Along with identifying the most impactful metrics for using machine learning to detect both types of flaky test, the empirical study shows how Flake16 is better than prior work, including (1) a 13% increase in overall F1 score when detecting NOD flaky tests and (2) a 17% increase in overall F1 score when detecting OD flaky tests.

As software becomes more important and ubiquitous, high quality software also becomes crucial. We depend on software developers who write the software to also maintain and improve its quality. When developers make changes to software, they rely on continuous integration [6] and regression testing [15] to check that changes do not break existing functionality. Continuous integration (CI) automates the process of building and testing software after every change. The process of running tests on the code after every change is known as regression testing. The goal of regression testing is to allow developers to detect and fix faults early on, ideally the moment the faults are introduced. Regression testing is widely used in both industry and open source, but regression testing suffers from two main challenges: (1) regression testing is costly, and (2) regression test suites often contain flaky tests.

Anomaly detection plays a critical role in large-scale industrial systems, where real-time streaming data is generated at high volumes. This research journal presents an in-depth study on developing an efficient anomaly detection algorithm specifically designed for such industrial systems. The goal is to identify abnormal patterns and deviations from normal behavior, enabling proactive maintenance, improved operational efficiency, and reduced downtime. The proposed algorithm leverages machine learning and stream processing techniques to handle the challenges associated with real- time streaming data analysis. The research covers algorithm design, implementation, evaluation, and performance analysis using large-scale datasets from industrial domains. Keywords: anomaly detection, real-time streaming data, large-scale industrial systems, machine learning, stream processing

SUMMARY The test process for evolving software systems takes on a different measurement aspect than that of new systems. Existing systems are generally being modified on a continuing basis as a normal part of the software maintenance activity. This process of product modifications is fault prone because faults are introduced in the code as it is being modified. From a statistical perspective, regression testing should be focused on those areas that are most likely to contain the introduced faults. Based on that premise, we have developed an evolutionary fault index that works as a fault surrogate varying in the same manner as faults. However, the knowledge as to the location of probable faults is not enough to assess the capabilities of a regression test suite. It is necessary to understand what the software is doing under each test. For that purpose, test execution profiles are gathered. Test execution profiles and the evolutionary fault indexes are combined in one methodology to provide an assessment of the overall regression testing activity and the suitability of each individual test. The methodology is illustrated with data from a 300 KLOC embedded system and its corresponding regression test suite. Copyright © 2000 John Wiley & Sons, Ltd.

In this article, a correlation-aided neural network (CANN) framework is proposed for distributed process monitoring of large-scale industrial automation systems. First, to model the nonlinear data relationship of multiple subsystems, a feature space is learned for each subsystem by means of neural network-based nonlinear projection. Second, different from conventional distributed monitoring methods, a distributed learning method and a novel objective function are proposed, where the influence of uncertainties is decreased by considering the correlation of variables among multiple subsystems. Third, a process monitoring method is proposed based on the CANN framework. The impact of the fault is analyzed, and the optimal monitoring performance is achieved by considering the correlation of all subsystems. To further reduce communication overhead among multiple subsystems, a tradeoff between monitoring performance and communication overhead is made based on gradients of the CANN model. Theoretical analysis demonstrates the superiority of the proposed method. The case studies on a multizone heating, ventilation, and air-conditioning system and the Tennessee Eastman benchmark process are given to evaluate and compare the effectiveness of the proposed method.

Object-Relational Mapping (ORM) provides developers a conceptual abstraction for mapping the application code to the underlying databases. ORM is widely used in industry due to its convenience; permitting developers to focus on developing the business logic without worrying too much about the database access details. However, developers often write ORM code without considering the impact of such code on database performance, leading to cause transactions with timeouts or hangs in large-scale systems. Unfortunately, there is little support to help developers automatically detect suboptimal database accesses. In this paper, we propose an automated framework to detect ORM performance anti-patterns. Our framework automatically flags performance anti-patterns in the source code. Furthermore, as there could be hundreds or even thousands of instances of anti-patterns, our framework provides sup- port to prioritize performance bug fixes based on a statistically rigorous performance assessment. We have successfully evaluated our framework on two open source and one large-scale industrial systems. Our case studies show that our framework can detect new and known real-world performance bugs and that fixing the detected performance anti- patterns can improve the system response time by up to 98%.

Dealing with clones is a common problem in large-scale software projects. While most of the research in this area is focused on detecting, and investigating the reasons of clones, little research has been done on the use of clone technology in large-scale industrial software. In this paper we present an in-depth case study of cloning in Command and Control Software Framework (CCSF) developed by HAVELSAN, a large software company in the Turkish defense industry. We proposed a practical classification schema for clones based on their removal strategies. A detailed analysis on categorization of clones, and lessons learned in clone management of large-scale industrial software systems are presented.

Architecture-based software reliability analysis methods shall help software architects to identify critical software components and to quantify their influence on the system reliability. Although researchers have proposed more than 20 methods in this area, empirical case studies applying these methods on large-scale industrial systems are rare. The costs and benefits of these methods remain unknown. On this behalf, we have applied the Cheung method on the software architecture of an industrial control system from ABB consisting of more than 100 components organized in nine subsystems with more than three million lines of code. We used the Littlewood/Verrall model to estimate subsystems failure rates and logging data to derive subsystem transition probabilities. We constructed a discrete time Markov chain as an architectural model and conducted a sensitivity analysis. This paper summarizes our experiences and lessons learned. We found that architecture-based software reliability analysis is still difficult to apply and that more effective data collection techniques are required.

With the increasing importance of complex software systems in the software industry, the need for using agent technologies to develop large-scale commercial and industrial software systems is growing rapidly. Such systems are complex, and there is a pressing need for system modeling techniques to support reliable, maintainable and extensible design. G-nets are a type of Petri net defined to support the modeling of a system as a set of independent and loosely-coupled modules. In this paper, we first introduce an extension of G-nets - the agent-based G-net - as a generic model for agent design. Then, to progress from an agent-based design model to an agent-oriented model, new mechanisms to support inheritance modeling are introduced. To illustrate our formal modeling technique for multi-agent systems, an example of an agent family in electronic commerce is provided.

Agents are becoming one of the most important topics in distributed and autonomous decentralized systems, and there are increasing attempts to use agent technologies to develop large-scale commercial and industrial software systems. The complexity of such systems suggests a pressing need for system modeling techniques to support reliable, maintainable, and extensible design. G-nets are a type of Petri net defined to support system modeling in terms of a set of independent and loosely-coupled modules. In this paper, we customize the basic G-net model to define a so-called "agent-based G-net" that can serve as a generic model for agent design. Then, to progress from an agent-based design model to an agent-oriented model, new mechanisms to support inheritance modeling are introduced. To illustrate our formal modeling technique for multiagent systems, an example of an agent family in electronic commerce is provided. Finally, we demonstrate how we can use model checking to verify some key behavioral properties of our agent model. This is facilitated by the use of an existing Petri net tool.

We evaluate the benefits of applying aspect-oriented software development techniques in the context of a large-scale industrial embedded software system implementing a number of crosscutting concerns. Additionally, we assess the feasibility of automatically extracting these crosscutting concerns from the source code. In order to achieve this, we present an approach for reverse engineering aspects from an ordinary application automatically. This approach incorporates both a concern verification and an aspect construction phase. Our results show that such automated support is feasible, and can lead to significant improvements in source code quality.

In large-scale industrial recommendation systems, model checkpoints are instrumental in maintaining training goodput and numerical correctness during system failures and job preemptions. The increasing prevalence of multi-terabyte models has rendered frequent regular model checkpoints impractical, resulting in substantial lost progress when recovering from failures. As model sizes continue to grow, researchers and practitioners are compelled to investigate more efficient and scalable solutions. This paper presents DECK, a novel approach to delta model checkpointing designed for real-world industrial systems. Specifically, DECK focuses on extracting delta states with near-zero overhead, staging and streaming delta checkpoints without interrupting the training process, and merging delta checkpoints in an optimal and decoupled manner. Experimental results demonstrate that DECK achieves a 12-fold increase in checkpoint frequency while maintaining negligible impact on training throughput, thereby attaining state-of-the-art (SOTA) production performance.

To assure high software quality for large-scale industrial software systems, traditional approaches of software quality assurance, such as software testing and performance engineering, have been widely used within Alibaba, the world's largest retailer, and one of the largest Internet companies in the world. However, there still exists a high demand for software quality assessment to achieve high sustainability of business growth and engineering culture in Alibaba. To address this issue, we develop an industrial solution for software quality assessment by following the GQM paradigm in an industrial setting. Moreover, we integrate multiple assessment methods into our solution, ranging from metric selection to rating aggregation. Our solution has been implemented, deployed, and adopted at Alibaba: (1) used by Alibaba's Business Platform Unit to continually monitor the quality for 60+ core software systems; (2) used by Alibaba's R&D Efficiency Unit to support group-wide quality-aware code search and automatic code inspection. This paper presents our proposed industrial solution, including its techniques and industrial adoption, along with the lessons learned during the development and deployment of our solution.

When developing large-scale industrial software systems engineers need to instantiate, configure, and deploy diverse reusable components. The number of component instances required depends on customer requirements only known during configuration and is typically unknown when modeling the systems’ variability. Also, the hierarchy of dynamically created component instances leads to complex dependencies between configuration decisions. Dealing with component multiplicity and hierarchy thus requires an approach capable of expressing the dependencies among dynamically instantiated components and configuration decisions. Furthermore, users need tool support for navigating the complex decision space during configuration. In this experience paper we report on applying a decision-oriented modeling approach for defining component variability, multiplicity, and hierarchy. We further present a configuration tool that guides end users through the complex decision space. We report applications of the approach to industrial software systems and describe patterns and lessons learned.