The Effects of Computational Resources on Flaky Tests

Flaky tests are test cases that can pass or fail without code changes. They often waste the time of software developers and obstruct the use of continuous integration. Previous work has presented several automated techniques for detecting flaky tests, though many involve repeated test executions and a lot of source code instrumentation and thus may be both intrusive and expensive. While this motivates researchers to evaluate machine learning models for detecting flaky tests, prior work on the features used to encode a test case is limited. Without further study of this topic, machine learning models cannot perform to their full potential in this domain. Previous studies also exclude a specific, yet prevalent and problematic, category of flaky tests: order-dependent (OD) flaky tests. This means that prior research only addresses part of the challenge of detecting flaky tests with machine learning. Closing this knowledge gap, this paper presents a new feature set for encoding tests, called Flake16. Using 54 distinct pipelines of data preprocessing, data balancing, and machine learning models for detecting both non-order-dependent (NOD) and OD flaky tests, this paper compares Flake16 to another well-established feature set. To assess the new feature set's effectiveness, this paper's experiments use the test suites of 26 Python projects, consisting of over 67,000 tests. Along with identifying the most impactful metrics for using machine learning to detect both types of flaky test, the empirical study shows how Flake16 is better than prior work, including (1) a 13% increase in overall F1 score when detecting NOD flaky tests and (2) a 17% increase in overall F1 score when detecting OD flaky tests.

Flaky tests are tests that can non-deterministically pass or fail for the same code version. These tests undermine regression testing efficiency, because developers cannot easily identify whether a test fails due to their recent changes or due to flakiness. Ideally, one would detect flaky tests right when flakiness is introduced, so that developers can then immediately remove the flakiness. Some software organizations, e.g., Mozilla and Netflix, run some tools—detectors—to detect flaky tests as soon as possible. However, detecting flaky tests is costly due to their inherent non-determinism, so even state-of-the-art detectors are often impractical to be used on all tests for each project change. To combat the high cost of applying detectors, these organizations typically run a detector solely on newly added or directly modified tests, i.e., not on unmodified tests or when other changes occur (including changes to the test suite, the code under test, and library dependencies). However, it is unclear how many flaky tests can be detected or missed by applying detectors in only these limited circumstances. To better understand this problem, we conduct a large-scale longitudinal study of flaky tests to determine when flaky tests become flaky and what changes cause them to become flaky. We apply two state-of-theart detectors to 55 Java projects, identifying a total of 245 flaky tests that can be compiled and run in the code version where each test was added. We find that 75% of flaky tests (184 out of 245) are flaky when added, indicating substantial potential value for developers to run detectors specifically on newly added tests. However, running detectors solely on newly added tests would still miss detecting 25% of flaky tests. The percentage of flaky tests that can be detected does increase to 85% when detectors are run on newly added or directly modified tests. The remaining 15% of flaky tests become flaky due to other changes and can be detected only when detectors are always applied to all tests. Our study is the first to empirically evaluate when tests become flaky and to recommend guidelines for applying detectors in the future.

Flaky tests are defined as tests that manifest non-deterministic behaviour by passing and failing intermittently for the same version of the code. These tests cripple continuous integration with false alerts that waste developers' time and break their trust in regression testing. To mitigate the effects of flakiness, both researchers and industrial experts proposed strategies and tools to detect and isolate flaky tests. However, flaky tests are rarely fixed as developers struggle to localise and understand their causes. Additionally, developers working with large codebases often need to know the sources of non-determinism to preserve code quality, i.e., avoid introducing technical debt linked with non-deterministic behaviour, and to avoid introducing new flaky tests. To aid with these tasks, we propose re-targeting Fault Localisation techniques to the flaky component localisation problem, i.e., pinpointing program classes that cause the non-deterministic behaviour of flaky tests. In particular, we employ Spectrum-Based Fault Localisation (SBFL), a coverage-based fault localisation technique commonly adopted for its simplicity and effectiveness. We also utilise other data sources, such as change history and static code metrics, to further improve the localisation. Our results show that augmenting SBFL with change and code metrics ranks flaky classes in the top-1 and top-5 suggestions, in 26% and 47% of the cases. Overall, we successfully reduced the average number of classes inspected to locate the first flaky class to 19% of the total number of classes covered by flaky tests. Our results also show that localisation methods are effective in major flakiness categories, such as concurrency and asynchronous waits, indicating their general ability to identify flaky components.

The purpose of this case study is to evaluate how and which performance testing tools which can be used in continuous integration (CI) environments. By doing so, developers can see the effects of changes immediately and react against performance problems of their applications. This will help companies to eliminate performance issues which the media is reporting about more often every day. CI provides the reference platform for executing the performance tests and the performance testing tools provide metrics like response time and percentage of errors. These metrics can be combined through CI plugins. The results of this combination can be visualized in form of graphs and tables. Through this case study, we give a short market overview of current CI servers and performance testing tools. In respect of the requirements by adesso AG, we will only evaluate performance testing tools, which can be integrated into the Atlassian Bamboo or Jenkins CI. We evaluated six performance testing tools of which four were integratable into the CI servers. Based on the results of our evaluation we will give a recommendation.

Regression testing provides important pass or fail signals that developers use to make decisions after code changes. However, flaky tests, which pass or fail even when the code has not changed, can mislead developers. A common kind of flaky tests are order-dependent tests, which pass or fail depending on the order in which the tests are run. Fixing order-dependent tests is often tedious and time-consuming. We propose iFixFlakies, a framework for automatically fixing order-dependent tests. The key insight in iFixFlakies is that test suites often already have tests, which we call helpers, whose logic resets or sets the states for order-dependent tests to pass. iFixFlakies searches a test suite for helpers that make the order-dependent tests pass and then recommends patches for the order-dependent tests using code from these helpers. Our evaluation on 110 truly orderdependent tests from a public dataset shows that 58 of them have helpers, and iFixFlakies can fix all 58. We opened pull requests for 56 order-dependent tests (2 of 58 were already fixed), and developers have already accepted pull requests for 21 of them, with all the remaining ones still pending.

Test flakiness refers to the non-deterministic behavior of software tests. A flaky test may pass or fail without changes to the test case code or the code under test. This leads to decreased developer productivity, expensive and excessive retrying of tests, bugs masked or missed by persistent flakiness, and loss of developer trust in test suites. Despite the problem receiving increasing attention from both practitioners and researchers over the past decade, no workshop dedicated to flakiness had been established.

As software becomes more important and ubiquitous, high quality software also becomes crucial. We depend on software developers who write the software to also maintain and improve its quality. When developers make changes to software, they rely on continuous integration [6] and regression testing [15] to check that changes do not break existing functionality. Continuous integration (CI) automates the process of building and testing software after every change. The process of running tests on the code after every change is known as regression testing. The goal of regression testing is to allow developers to detect and fix faults early on, ideally the moment the faults are introduced. Regression testing is widely used in both industry and open source, but regression testing suffers from two main challenges: (1) regression testing is costly, and (2) regression test suites often contain flaky tests.

Federated learning (FL) enables privacy-preserving distributed machine learning by training models on edge client devices using their local data without revealing their raw data. In edge environments, various applications require different neural network models, making it crucial to perform joint training of multiple models on edge devices, known as multi-task FL. While existing multi-task FL approaches enhance resource utilization on edge devices through adaptive resource configuration or client selection, optimizing either of these aspects alone may lead to suboptimality. Therefore, in this paper, we explore a joint client selection and resource configuration method called JCSRC for multi-task FL, aiming to maximize energy efficiency in environments with limited computation and communication resources and heterogeneous client devices. Firstly, we formalize this problem as a mixed-integer nonlinear programming problem considering all these characteristics and prove its NP-hardness. To address this problem, we first design a multi-agent reinforcement learning (MARL)-based client selection method that selects appropriate clients for each task to train their models. The MARL method makes client selection decisions based on the clients’ data quality, energy efficiency, communication, and computation capacity to ensure fast convergence and energy efficiency. Then, we design a particle swarm optimization (PSO)-based resource configuration scheme that configures appropriate computation and bandwidth resources for each task on each client. The PSO scheme makes resource configuration decisions based on theoretically derived optimal CPU frequency and bandwidth to achieve high energy efficiency. Finally, we carry out extensive simulations and testbed-based experiments to validate our proposed JCSRC. The results demonstrate that, in comparison to state-of-the-art solutions, JCSRC can save energy consumption by up to 59% to achieve the target accuracy.

Non-deterministically behaving test cases cause developers to lose trust in their regression test suites and to eventually ignore failures. Detecting flaky tests is therefore a crucial task in maintaining code quality, as it builds the necessary foundation for any form of systematic response to flakiness, such as test quarantining or automated debugging. Previous research has proposed various methods to detect flakiness, but when trying to deploy these in an industrial context, their reliance on instrumentation, test reruns, or language-specific artifacts was inhibitive. In this paper, we therefore investigate the prediction of flaky tests without such requirements on the underlying programming language, CI, build or test execution framework. Instead, we rely only on the most commonly available artifacts, namely the tests' outcomes and durations, as well as basic information about the code evolution to build predictive models capable of detecting flakiness. Furthermore, our approach does not require additional reruns, since it gathers this data from existing test executions. We trained several established classifiers on the suggested features and evaluated their performance on a large-scale industrial software system, from which we collected a data set of 100 flaky and 100 non-flaky test- and code-histories. The best model was able to achieve an F1-score of 95.5% using only 3 features: the tests' flip rates, the number of changes to source files in the last 54 days, as well as the number of changed files in the most recent pull request.

When a change introduces a bug into a large software repository, there is often a delay between when the change is committed and when bug is detected. This is true even when the bug causes an existing test to fail! These delays are caused by resource constraints which prevent the organization from running all of the tests on every change. Due to the delay, a Continuous Integration system needs to locate buggy commits. Locating them is complicated by flaky tests that pass and fail non-deterministically. The flaky tests introduce noise into the CI system requiring costly reruns to determine if a failure was caused by a bad code change or caused by non-deterministic test behavior. This paper presents an algorithm, Flake Aware Culprit Finding, that locates buggy commits more accurately than a traditional bisection search. The algorithm is based on Bayesian inference and noisy binary search, utilizing prior information about which changes are most likely to contain the bug. A large scale empirical study was conducted at Google on 13,000+ test breakages. The study evaluates the accuracy and cost of the new algorithm versus a traditional deflaked bisection search.

Developers rely on regression testing in their continuous integration (CI) environment to find changes that introduce regression faults. While regression testing is widely practiced, it can be costly. Regression test selection (RTS) reduces the cost of regression testing by not running the tests that are unaffected by the changes. Industry has adopted module-level RTS for their CI environment, while researchers have proposed class-level RTS. In this paper, we compare module-and class-level RTS techniques in a cloud-based CI environment, Travis. We also develop and evaluate a hybrid RTS technique that combines aspects of the module-and class-level RTS techniques. We evaluate all the techniques on real Travis builds. We find that the RTS techniques do save testing time compared to running all tests (RetestAll), but the percentage of time for a full build using RTS (76.0%) is not as low as found in previous work, due to the extra overhead in a cloud-based CI environment. Moreover, we inspect test failures from RetestAll builds, and although we find that RTS techniques can miss to select failed tests, these test failures are almost all flaky test failures. As such, RTS techniques provide additional value in helping developers avoid wasting time debugging failures not related to the recent code changes. Overall, our results show that RTS can be beneficial for the developers in the CI environment, and RTS not only saves time but also avoids misleading developers by flaky test failures.

Fog computing has attracted considerable attention to meet the location awareness and real-time response requirements of various applications. However, compared to cloud computing, fog devices have relatively limited power supplies, computation resources, and communication resources, raising design challenges to meet real-time response requirements. To balance response time and energy consumption for multiple fog devices running multiple applications, this work presents an energy-efficient offloading decision mechanism and an offloading dispatcher to dispatch applications to the corresponding device by effectively managing the computation and communication resources of all devices in a fog computing environment. The offloading tasks are composed of several subtasks with an end-to-end deadline. To meet the response time requirements of such applications, a run-time scheduler with an end-to-end latency schedulability consideration is also presented. Evaluation results show considerable energy savings using this framework and a real platform study provides validation.

Software testing is vital for strengthening the security of embedded systems by identifying and rectifying code errors, flaws and vulnerabilities. This is particularly significant when addressing vulnerabilities associated with side-channel attacks, given that they introduce a distinctive class of vulnerabilities, primarily subject to manual testing procedures. Manual testing remains prevalent despite advances in automation, posing challenges, particularly for complex environments. This research aims to automate embedded software testing on hardware in a modular and scalable manner, addressing the limitations of manual testing. We present a system designed to automate testing, including Side-Channel Analysis (SCA), in Continuous Integration (CI) environments, emphasizing accessibility and collaboration through open-source tools. Our evaluation setup based on GitLab, Jenkins and the ChipWhisperer framework shows that automating and integrating Side-Channel Analysis (SCA) in Continuous Integration (CI) environments is possible in an efficient way.

Determining the mechanism of action (MOA) for natural products remains a significant bottleneck in drug discovery, particularly for researchers with limited computational resources or small compound libraries. Traditional approaches require screening large numbers of annotated compounds alongside unknowns, which is cost-prohibitive, or depend on complex machine learning models that need substantial computational resources and large datasets.Here, we present a dissertation chapter excerpt: MOAST (Mechanism of Action Similarity Tool), a BLAST-inspired computational workflow that addresses these limitations by providing rapid MOA hypotheses for newly screened compounds. This chapter investigates two complementary approaches: a kernel density estimation (KDE) method providing statistical significance measures and E-values for MOA class membership, and a CatBoost machine learning classifier for multi-class prediction with ranked outputs.Using cytological profiling data from HeLa and A549 cell lines, MOAST achieved 22% accuracy for the top 5 predictions among ~ 300 MOA classes, with the CatBoost classifier reaching 10% balanced accuracy—significantly better than the ~ 3% reported in literature. The tool suggests a 0.8 prediction probability threshold for trustworthy results and demonstrates robust performance across multiple feature reduction strategies.MOAST provides a practical, accessible solution that bridges traditional phenotypic screening and modern computational approaches, making MOA determination feasible for researchers with limited resources while maintaining statistical rigor and interpretability.

The IoT network allows IoT devices to communicate with other devices, applications, and services by exploiting existing network infrastructure. Recently, a promising paradigm, MEC, emerging for alleviating high latency data services in cloud computing framework plays an important role in the IoT network. Network performance and intelligence can be improved by integrating cognitive and cooperative mechanisms in the MEC framework. However, the QoS of computation-intensive tasks may degrade because of the limited available computational resources in MEC servers. Moreover, the characteristics of resources belonging to MEC servers and cloud servers are commonly different. In order to optimize the strategy of resource assignment, the tasks of assigning the limited computational resources in MEC servers and resolving the high latency problem in cloud servers have attracted growing interest from researchers. In this article, we propose a joint optimization paradigm for task-driven resource assignment based on evolutionary computation considering the power consumption and computation/communication delay simultaneously. The MEC framework consists of MEC servers, mobile devices, and cloud servers, and offloads the computational resources to the edge of end users. Additionally, we introduce and analyze three typical task-driven cases, which are the server-determined condition, server-flexible condition, and server-uncertain condition, respectively. Finally, we present the existing technical challenges and discuss the open research issues.