Improving fault localization via weighted execution graph and graph attention network

Spectrum-based fault localization (SBFL) techniques are widely studied and have been evaluated to be effective in locating faults. Recent studies also showed that developers from industry value automated SBFL techniques. However, their effectiveness is still limited by two main reasons. First, the test coverage information leveraged to construct the spectrum does not reflect the root cause directly. Second, SBFL suffers from the tie issue so that the buggy code entities can not be well differentiated from non-buggy ones. To address these challenges, we propose to leverage the information of version histories in fault localization based on the following two intuitions. First, version histories record how bugs are introduced to software projects and this information reflects the root cause of bugs directly. Second, the evolution histories of code can help differentiate those suspicious code entities ranked in tie by SBFL. Our intuitions are also inspired by the observations on debugging practices from large open source projects and industry. Based on the intuitions, we propose a novel technique HSFL (historical spectrum based fault localization). Specifically, HSFL identifies bug-inducing commits from the version history in the first step. It then constructs historical spectrum (denoted as Histrum) based on bug-inducing commits, which is another dimension of spectrum orthogonal to the coverage based spectrum used in SBFL. HSFL finally ranks the suspicious code elements based on our proposed Histrum and the conventional spectrum. HSFL outperforms the state-of-the-art SBFL techniques significantly on the Defects4J benchmark. Specifically, it locates and ranks the buggy statement at Top-1 for 77.8% more bugs as compared with SBFL, and 33.9% more bugs at Top-5. Besides, for the metrics MAP and MRR, HSFL achieves an average improvement of 28.3% and 40.8% over all bugs, respectively. Moreover, HSFL can also outperform other six families of fault localization techniques, and our proposed Histrum model can be integrated with different families of techniques and boost their performance.

Software testing is a popular practice to evaluate the software quality, and debugging is one of the most time-consuming tasks. In the last decades, spectrum-based fault localization (SBFL) techniques have been extensively studied and empirically shown effective in locating faults in a program. However, recent researches demonstrated that the accuracy of an SBFL technique may decrease when it is applied to a program containing code-omission faults. In this paper, we present a novel approach – MCFL. It models the behavior of code omission, embeds code-omission probes into programs to identify potential locations of missing code, captures spectra of program execution, and evaluates the suspiciousness of program entities being related to faults. Different from existing SBFL techniques, MCFL synthesizes a ranked list consisting of both suspicious statements and suspicious code-omission sites, which reflect the probability of a normal statement being faulty and the probability of missing code at specific positions in the program, respectively. We conducted a controlled experiment to compare the fault-localization accuracy of MCFL with those of four popular SBFL techniques. Six real-world projects from the dataset Defects4J are used as the experiment subjects. The experiment result showed that (i) MCFL outperforms the experimented SBFL techniques on most subjects, and on average has a 17.47% improvement; (ii) For more than 60% of the faults, MCFL successfully tells whether they are due to code omission.

Fault Localization Prioritization (FLP) aims at reordering existing test cases so that the location of detected faulty components can be identified earlier, using certain fault localization techniques. Although some researchers have proposed adaptive prioritization strategies with white-box code coverage information, such information may not always be available. In this paper, we address the FLP problem using black-box information derived from partitioning the input domain. Based on the well-known technique of Spectra-Based Fault Localization (SBFL), three test case prioritization strategies are designed following some basic SBFL heuristics. The implementation of these proposed strategies relies only on the partition information, and does not require any test case execution history. Experiments show that our strategies, when compared with pure random selection, result in a faster localization of faulty statements, reducing the number of test case executions required. Here, we analyze the characteristics and merits of the three proposed strategies.

Spectrum-based Fault Localization (SBFL) techniques use risk evaluation formulas to calculate each statement's likelihood of having a bug based on test results. SBFL can not only be used in statement level, but also can be used with other program entities such as branches, functions and so on. Most previous studies have been conducted under the assumption of a single bug. However, software always contains multi-bugs in practice. A natural idea of debugging is to isolate bugs and then use SBFL techniques to locate one bug for each group. In this paper, we conduct an empirical study on clustering for isolating bugs in fault localization. We analyze the effects of six fault localization techniques and two cluster algorithms. The main observations are: (1) ER5 (Wong1) achieves the best results of fault localization with clustering; (2) K-means outperforms hierarchical clustering for isolating bugs in fault localization.

Due to the complexity of software systems, bugs are inevitable. Software debugging is tedious and time consuming. To help developers perform this crucial task, a number of spectra-based fault localization techniques have been proposed. In general, spectra-based fault localization helps developers to find the location of a bug given its symptoms (e.g., program failures). A previous study by Parnin and Orso however implies that several assumptions made by existing work on spectra-based fault localization do not hold in practice, which hinders the practical usage of these tools. Moreover, a recent study by Xie et al. claims that spectra-based fault localization can potentially weaken programmers' abilities in fault detection.Unfortunately, these studies are performed either using only 2 bugs from small systems (Parnin and Orso's study) or synthetic bugs injected into toy programs (Xie et al.'s study), only involve students, and use dated spectra-based fault localization tools. Thus, the question whether spectra-based fault localization techniques can help professionals to improve their debugging efficiency in a reasonably large project is still insufficiently answered. In this paper, we perform a more realistic investigation of how professionals can use and benefit from spectra-based fault localization techniques. We perform a user study of spectra-based fault localization with a total of 16 real bugs from 4 reasonably large open-source projects, with 36 professionals, amounting to 80 recorded debugging hours. The 36 professionals are divided into 3 groups, i.e., those that use an accurate fault localization tool, use a mediocre fault localization tool, and do not use any fault localization tool. Our study finds that both the accurate and mediocre spectra-based fault localization tools can help professionals to save their debugging time, and the improvements are statistically significant and substantial. We also discuss implications of our findings to future directions of spectra-based fault localization.

Background: Prior studies lack the perspective of using developer's skills to augment the performance of automated program repair (APR). APR has a phase referred to as fault localization (FL), which automatically finds the faulty statement that causes faults. To achieve a well-performed FL phase, we study developers' FL skills, which allow developers to find faulty statements. We suppose that such FL skills can add additional information to fault localization to augment the accuracy of fault localization and reduce the execution cost of APR. Aims: We aim at revealing a criterion that distinguishes whether using the FL skill reduces the execution cost of the state-of-the-art APR, TBar, depending on the accuracy of the FL skill. Method: We conduct a simulation case study in the Defects4J dataset, which is the most popular dataset. We compare the numbers of candidate patches generated by TBar using the FL skill or using spectrum-based fault localization (SBFL). Results: Our case study revealed that, if developers localized the faulty statements before inspecting 40 % of the statements in the target program, the execution cost of TBar reduces for over half of the studied faults. The 40 % value is a requirement for developers using the FL skill to augment the performance of APR. Conclusion: If developers can localize the faulty statement before inspecting 40 % of the statements, integrating the FL skill with SBFL makes TBar faster compared to when SBFL is used.

Fault localization aims to support the debugging activities of human developers by highlighting the program elements that are suspected to be responsible for the observed failure. Spectrum Based Fault Localization (SBFL), an existing localization technique that only relies on the coverage and pass/fail results of executed test cases, has been widely studied but also criticized for the lack of precision and limited effort reduction. To overcome restrictions of techniques based purely on coverage, we extend SBFL with code and change metrics that have been studied in the context of defect prediction, such as size, age and code churn. Using suspiciousness values from existing SBFL formulas and these source code metrics as features, we apply two learn-to-rank techniques, Genetic Programming (GP) and linear rank Support Vector Machines (SVMs). We evaluate our approach with a ten-fold cross validation of method level fault localization, using 210 real world faults from the Defects4J repository. GP with additional source code metrics ranks the faulty method at the top for 106 faults, and within the top five for 173 faults. This is a significant improvement over the state-of-the-art SBFL formulas, the best of which can rank 49 and 127 faults at the top and within the top five, respectively.

Software testing is the key to ensuring software quality. Test case generation and software fault localization are two important research objects of testing. Spectrum-based fault localization (SBFL) method is a mainstream dynamic fault localization method, which inputs test cases into the program for execution and collects statement coverage information and execution results. By analysing this information, a fault report can be generated. The performance of SBFL is affected by test case quality. Therefore, this paper proposes a test case generation method based on improved genetic algorithm to assist SBFL. This method uses a small number of initial test cases to execute the program, then uses the XGBoost model to pre-process the spectrum information and generates a list of potential fault statements to guide the iterative process of the genetic algorithm. The experimental results show that the test cases generated using the improved genetic algorithm have certain advantages over the test cases generated by general genetic algorithm and the baseline test cases.

In Spectrum-Based Fault Localization, program elements such as statements or functions are ranked according to a suspiciousness score which can guide the programmer in finding the fault more efficiently. However, such a ranking does not include any additional information about the element under investigation. In this work, we propose to complement function-level spectrum based fault localization with function call chains - i.e., snapshots of the call stack occurring during execution - on which the fault localization is first performed, and then narrowed down to functions. Our experiments using defects from four Defects4J programs show that (i) 84% of the defective functions can be found in call chains with highest scores, (ii) the proposed approach improves Ochiai ranking of 1 to 6 positions on average, with a relative improvement of 45%, and (iii) the improvement is substantial when Ochiai produces bad rankings.

Many Spectrum-based Fault Localization (SBFL) techniques have been published over the years, but they are still rarely used in the industry. Traditional SBFL methods may be effective in certain situations but may fall short in others due to a lack of contextual information for the algorithm to function properly. I have made efforts to improve SBFL algorithms by investigating how SBFL aligns bug-fix patterns with high-ranked elements, creating an improved algorithm that uses call frequencies, examining how division by zero affects SBFL’s efficiency, and proposing ways to avoid it. I have also developed a user feedback-based SBFL tool called CharmFL, which is helpful for Python programmers. I consider all of these things as contexts that can be used to create a more context-aware SBFL approach. In this doctoral symposium, I present my research and outline my plans for the remainder of my Ph.D. program.

In order to improve the effectiveness of fault localization, various test suite reduction techniques have been proposed. However, excessive or improper reduction of test cases may lose some testing information, thus causing a negative impact on fault localization. In this paper, we propose a new similarity-based test suite reduction approach to improving spectrum-based fault localization. Firstly, this approach extracts the high suspicious statements in the faulty program and removes the coincidental passed test cases for test suite selection. Then, it selects similar passed test cases for each different failed test case from the new passed test set based on the similar proportion of their execution traces, and determines the final composition of each similar test set based on the contribution of failed test cases to fault localization. By using the execution information of each similar test set with a spectrum-based fault localization approach, the ranks of statements can be obtained. Finally, synthesizing all ranks of statements in each rank list, we obtain the final rank list of statements. Several experiments show that our approach can help reduce the debugging effort in terms of the percentage of statements needed to be inspected when locating faults in both single-fault and multi-fault programs. Moreover, the results demonstrate that the value of similar proportion have nontrivial influence on the effectiveness of fault localization.

Fault localization is known to be tedious and expensive. To help practitioners reduce the needed effort, researchers have proposed many automated fault localization techniques over recent years. Spectrum-based fault localization techniques utilize the coverage data of test cases to localize faults. Although evaluations of these techniques show promising results, they are not being applied in practice yet. With this paper, we would like to stimulate discussions on challenges of operationalizing these techniques from a data-centric point of view. We describe the challenges we have encountered in our collaboration with industry. We also describe future plans.

Learn-to-rank techniques have been successfully applied to fault localisation to produce ranking models that place faulty program elements at or near the top. Genetic Programming has been successfully used as a learning mechanism to produce highly effective ranking models for fault localisation. However, the inherent stochastic nature of GP forces its users to learn multiple ranking models and choose the best performing one for the actual use. This train-and-select approach means that the absolute majority of the computational resources that go into the evolution of ranking models are eventually wasted. We introduce Ensemble Model for Fault Localisation (EMF), which is a learn-to-rank fault localisation technique that utilises all trained models to improve the accuracy of localisation even further. EMF ranks program elements using a lightweight, voting-based ensemble of ranking models. We evaluate EMF using 389 real-world faults in Defects4J benchmark. EMF can place 30.1% more faults at the top when compared to the best performing individual model from the train-and-select approach. We also apply Genetic Algorithm (GA) to construct the best performing ensemble. Compared to naively using all ranking models, GA generated ensembles can localise further 9.2% more faults at the top on average.

Automated program repair (APR) tools apply fault localization (FL) techniques to identify the locations of likely faults to be repaired. The effectiveness, performance, and repair correctness of APR depends in part on the FL method used. If FL does not identify the location of a fault, the application of an APR tool will not be effective--it will fail to repair the fault. If FL assigns the actual faulty statement a low priority for repair, APR performance will be reduced by increasing the time required to find a potential repair. In addition, the correctness of a generated repair will be decreased since APR will modify fault-free statements that are assigned a higher priority for repair than an actual faulty statement. We conducted a controlled experiment to evaluate the impact of ten FL techniques on APR effectiveness, performance, and repair correctness using a brute force APR tool applied to faulty versions of the Siemens Suite and two other large programs: space and sed. All FL techniques were effective in identifying all faults; however, Wong3 and Ample1 were the least effective FL techniques since they assigned the lowest priority for repair in more than 26 % of the trials. We obtained the worst APR performance significantly when Ample1 was used since it generated a large number of variants in 29.11 % of the trials, and took the longest time to produce potential repairs. Jaccard FL improved repair correctness by generating more validated repairs---potential repairs that pass a set of regression tests, and generating potential repairs that failed fewer regression tests. Also Jaccard's performance is noteworthy in that it never generated a large number of variants during the repair process compared to the alternatives.

Spectrum-based fault localization aims to find the program statements that contain faults, according to the dynamic information of executing program. We propose a spectra-based fault localization method with test case reduction, which removes redundant test cases in fault localization. Experiments show that new method requires fewer test cases to obtain the better fault localization results.