From software failure to explanation

SummaryPure spectrum‐based fault localization (SBFL) is a well‐studied statistical debugging technique that only takes a set of test cases (some failing and some passing) and their code coverage as input and produces a ranked list of suspicious program elements to help the developer identify the location of a bug that causes a failed test case. Studies show that pure SBFL techniques produce good ranked lists for small programs. However, our previous study based on the iBugs benchmark that uses the AspectJ repository shows that, for realistic programs, the accuracy of the ranked list is not suitable for human developers. In this paper, we confirm this based on a combined empirical evaluation with the iBugs and the Defects4J benchmark. Our experiments show that, on average, at most ∼40%, ∼80%, and ∼90% of the bugs can be localized reliably within the first 10, 100, and 1000 ranked lines, respectively, in the Defects4J benchmark. To reliably localize 90% of the bugs with the best performing SBFL metric D∗, ∼450 lines have to be inspected by the developer. For human developers, this remains unsuitable, although the results improve compared with the results for the AspectJ benchmark. Based on this study, we can clearly see the need to go beyond pure SBFL and take other information, such as information from the bug report or from version history of the code lines, into consideration.

Adaptive Random Testing (ART) has been proposed as an enhancement to random testing, based on assumptions on how failing test cases are distributed in the input domain. The main assumption is that failing test cases are usually grouped into contiguous regions. Many papers have been published in which ART has been described as an effective alternative to random testing when using the average number of test case executions needed to find a failure (F-measure). But all the work in the literature is based either on simulations or case studies with unreasonably high failure rates. In this paper, we report on the largest empirical analysis of ART in the literature, in which 3727 mutated programs and nearly ten trillion test cases were used. Results show that ART is highly inefficient even on trivial problems when accounting for distance calculations among test cases, to an extent that probably prevents its practical use in most situations. For example, on the infamous Triangle Classification program, random testing finds failures in few milliseconds whereas ART execution time is prohibitive. Even when assuming a small, fixed size test set and looking at the probability of failure (P-measure), ART only fares slightly better than random testing, which is not sufficient to make it applicable in realistic conditions. We provide precise explanations of this phenomenon based on rigorous empirical analyses. For the simpler case of single-dimension input domains, we also perform formal analyses to support our claim that ART is of little use in most situations, unless drastic enhancements are developed. Such analyses help us explain some of the empirical results and identify the components of ART that need to be improved to make it a viable option in practice.

In risk-driven testing, test cases are generated and/or prioritized based on different risk measures. For example, the most basic risk measure would analyze the history of the software and assigns higher risk to the test cases that used to detect bugs in the past. However, in practice, a test case may not be exactly the same as a previously failed test, but quite similar. In this study, we define a new risk measure that assigns a risk factor to a test case, if it is similar to a failing test case from history. The similarity is defined based on the execution traces of the test cases, where we define each test case as a sequence of method calls. We have evaluated our new risk measure by comparing it to a traditional risk measure (where the risk measure would be increased only if the very same test case, not a similar one, failed in the past). The results of our study, in the context of test case prioritization, on two open source projects show that our new risk measure is by far more effective in identifying failing test cases compared to the traditional risk measure.

Fuzzers, or random testing tools, are powerful tools for finding bugs. A major problem with using fuzzersis that they often trigger many bugs that are already known. The fuzzer taming problem addresses this issue by ordering bug-triggering random test cases generated by a fuzzer such that test cases exposing diverse bugs are found early in the ranking. Previous work on fuzzer taming first reduces each test case into a minimal failure-inducing test case using delta debugging, then finds the ordering by applying the Furthest Point First algorithm over the reduced test cases. During the delta debugging process, a sequence of failing test cases is generated (the "delta debugging trail"). We hypothesize that these additional failing test cases also contain relevant information about the bug and could be useful for fuzzertaming. In this paper, we propose to use these additional failing test cases generated during delta debugging to help tame fuzzers. Our experiments show that this allows for more diverse bugs to be found early in the furthest point first ranking.

Mutation-oriented test data augmentation for GUI software fault localization

Researchers have applied cluster analysis onto execution profiles induced by test cases in order to solve problems in the area of software testing and analysis. The employed clustering techniques varied, and no study was conducted to rate these techniques in terms of their effectiveness in this specific domain. This work aims at doing so experimentally. Specifically, given test sets comprising passing and failing test cases, we measure the performance of each technique at isolating the failing test cases from the passing cases. The study included one technique from each of the six main families of clustering algorithms. Our results suggested the following ranking of the evaluated techniques from best to worst: DBSCAN, K-Means, Agglomerative-AGNES and WaveCluster, Fuzzy-FCM, and K-Subspace.

Software refactoring is to improve readability, maintainability and expansibility of software by adjusting its internal structure, whereas the external behaviors of software are not changed. Although software refactoring should not change the external behaviors of software systems, they might make a regression test case obsolete (with syntax and runtime errors) or fail. People have investigated which refactorings had an influence on regression test case. However, how test cases are influenced by refactorings and what kind of errors might be introduced remain unknown. To this end, in this paper, we proposed an approach to analyze refactorings' impact on regression test cases. On one hand, we analyzed why regression test cases failed. On the other hand, we analyzed the influence of refactorings on software interfaces. Based on the analysis, we built up a mapping between refactorings and test case failure. Such a mapping can be used to guide test case repair automation where test cases are made obsolete by refactorings. The approach was evaluated on five open-source applications. Evaluation results suggest that the precision of the approach is greater than 80%.

Chapter 13 - Isolating Failure Causes

In order to improve the effectiveness of fault localization, various test suite reduction techniques have been proposed. However, excessive or improper reduction of test cases may lose some testing information, thus causing a negative impact on fault localization. In this paper, we propose a new similarity-based test suite reduction approach to improving spectrum-based fault localization. Firstly, this approach extracts the high suspicious statements in the faulty program and removes the coincidental passed test cases for test suite selection. Then, it selects similar passed test cases for each different failed test case from the new passed test set based on the similar proportion of their execution traces, and determines the final composition of each similar test set based on the contribution of failed test cases to fault localization. By using the execution information of each similar test set with a spectrum-based fault localization approach, the ranks of statements can be obtained. Finally, synthesizing all ranks of statements in each rank list, we obtain the final rank list of statements. Several experiments show that our approach can help reduce the debugging effort in terms of the percentage of statements needed to be inspected when locating faults in both single-fault and multi-fault programs. Moreover, the results demonstrate that the value of similar proportion have nontrivial influence on the effectiveness of fault localization.

Combinatorial testing is a prevailing approach to finding out faults in the extremely complex systems of train operation for railway. While combinatorial testing is applied to Chinese Train Control Systems (CTCS) for revealing faults, some potential faults tend to be arduous to trigger due to the interaction of multiple test cases. For the sake of accurate and efficient fault localization of CTCS under the circumstances of masking effects, a novel method of locating the faults of CTCS using the relationship matrix with frequent terms has been proposed in the paper. To begin with, failed test cases from CTCS were made full use of to calculate the frequentness of failure‐inducing elements for ordered tables. Then, the relationship matrix with frequent terms for CTCS was established to work out the suspiciousness of failure‐inducing combinations by the following relation of failure‐inducing elements in the above tables. Furthermore, an iterative approach to locating failure‐inducing combinations was adopted to obtain the full list of rankings of failure‐inducing combinations, among which top combinations, especially MFS (Minimal Failure‐causing Schema), were prone to discover the faults of CTCS. Finally, the effectiveness and accuracy of the proposed method for CTCS was confirmed on the simulation platform of CTCS for Beijing‐Shenyang High‐speed Rail. Empirical studies suggest that compared to the method of fault forest, the average suspiciousness in the proposed method is 12.17% higher than that and the number of additional test cases are reduced by 31.7% on average, which can produce more precise and efficient results of identifying the faults of CTCS in case of masking effects. The research achievements can provide some references for the safety evaluation and system optimization of CTCS for high‐speed rail. © 2022 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.

Software fault localization technology aims to identify suspicious statements that cause software failures, which is crucial for ensuring software quality. Spectrum-based software fault location (SBFL) technology calculates the suspiciousness of each statement by analyzing the correlation between statement coverage information and execution results in test cases. SBFL has attracted increasing attention from scholars due to its high efficiency and scalability. However, existing SBFL studies have shown that a large number of statements share the same suspiciousness, which hinders software debuggers from quickly identifying the location of faulty statements. To address this challenge, we propose an SBFL model based on weighted association rule mining and complex networks: FL-WARMCN. The algorithm first uses Jaccard to measure the distance between passing and failing test cases, and applies it as the weight of passing test cases. Next, FL-WARMCN calculates the initial suspiciousness of each statement based on the program spectrum data. Then, the FL-WARMCN model utilizes a weighted association rule mining algorithm to obtain the correlation relationships between statements and models the network based on this. In the network, the suspiciousness of statements is used as node weights, and the correlation between statements is used as edge weights. We chose the eigenvector centrality that takes into account the degree centrality of statements and the importance of neighboring statements to calculate the importance of each statement, and used it as a weight to incorporate into the weighted suspiciousness calculation of the statement. Finally, we applied the FL-WARMCN model for experimental validation on the Defects4J dataset. The results showed that the model was significantly superior to other baselines. In addition, we analyzed the impact of different node and edge weights on model performance.

In the software multi-fault scenario, the failed test cases of different fault sources are classified by clustering, and then the program statements in each cluster are ranked with suspicion by the fault location technology based on program spectrum. Software multi-fault location technology decouples failed test cases by clustering their code coverage information. At present, there are many clustering strategies for software failed test cases. However, due to the characteristics of high dimension, complex structure and variable shape of the execution information of failed test cases, it is difficult for a single clustering algorithm to accurately identify the cluster structure, so that it cannot be decoupled accurately. In this paper, a variety of clustering algorithms and different parameters are selected to obtain multiple cluster members, according to the clustering members to get the similarity matrix, and then hierarchical clustering is used to obtain the final clustering results, to achieve the integration between different clustering algorithms, so that the algorithm can identify the complex cluster structure, and then improve the efficiency of fault location.

In random testing, it is often desirable to produce a "quick test" -- an extremely inexpensive test suite that can serve as a frequently applied regression and allow the benefits of random testing to be obtained even in very slow or over-subscribed test environments. Delta debugging is an algorithm that, given a failing test case, produces a smaller test case that also fails, and typically executes much more quickly. Delta debugging of random tests can produce effective regression suites for previously detected faults, but such suites often have little power for detecting new faults, and in some cases provide poor code coverage. This paper proposes extending delta debugging by simplifying tests with respect to code coverage, an instance of a generalization of delta debugging we call cause reduction. We show that test suites reduced in this fashion can provide very effective quick tests for real-world programs. For Mozilla's Spider Monkey JavaScript engine, the reduced suite is more effective for finding software faults, even if its reduced runtime is not considered. The effectiveness of a reduction-based quick test persists through major changes to the software under test.

Although slices provide a good basis for analyzing programs during debugging, they lack in their capabilities providing precise information regarding the most likely root causes of faults. Hence, a lot of work is left to the programmer during fault localization. In this paper, we present an approach that combines an advanced dynamic slicing method with constraint solving in order to reduce the number of delivered fault candidates. The approach is called Constraints Based Slicing (CONBAS). The idea behind CONBAS is to convert an execution trace of a failing test case into its constraint representation and to check if it is possible to find values for all variables in the execution trace so that there is no contradiction with the test case. For doing so, we make use of the correctness and incorrectness assumptions behind a diagnosis, the given failing test case. Beside the theoretical foundations and the algorithm, we present empirical results and discuss future research. The obtained empirical results indicate an improvement of about 28% for the single fault and 50% for the double-fault case compared to dynamic slicing approaches.

Bug benchmarks are used in development and evaluation of debugging approaches, e.g. fault localization and automated repair. Quantitative performance comparison of different debugging approaches is only possible when they have been evaluated on the same dataset or benchmark. However, benchmarks are often specialized towards usage for certain debugging approaches in their contained data, metrics, and artifacts. Such benchmarks cannot be easily used on debugging approaches outside their scope as such approach may rely on specific data such as bug reports or code metrics that are not included in the dataset. Furthermore, benchmarks vary in their size w.r.t. the number of subject programs and the size of the individual subject programs. For these reasons, we have performed a systematic literature review where we have identified 73 benchmarks that can be used to evaluate debugging approaches. We compare the different benchmarks w.r.t. their size and the provided information such as bug reports, contained test cases, and other code metrics. This comparison is intended to help researchers to quickly identify all suitable benchmarks for evaluating their specific debugging approaches. Furthermore, we discuss reoccurring issues and challenges in selection, acquisition, and usage of such bug benchmarks, i.e., data availability, data quality, duplicated content, data formats, reproducibility, and extensibility.Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board.