Predicting Effectiveness of Generate-and-Validate Patch Generation Systems Using Random Forest

Automated Program Repair (APR) is becoming a hot topic in Software Engineering community with many approaches being proposed and experiments being performed over the years. The results obtained from different experiments can be used as practical guidance to advance APR techniques. However, researchers have generally ignored the biases with respect to the unexpected results generated by various APR techniques, in which case the repair process cannot be finished normally and is terminated with unexpected exceptions (referred to as the non-repairability factors). In this paper, we aim to thoroughly understand the reasons for such non-repairability factors of various APR techniques, thus to provide practical insights for diverse stakeholders to establish an unbiased evaluation of APR techniques. To achieve so, we performed a systematic study on the existing execution logs that are ended with unexpected exceptions collected from different APR studies. Specifically, we investigated different types of exceptions with their frequencies of occurrence, the behind reasons of such occurrences, as well as the impact of such exceptions on the repairability of APR techniques. Our experimental results reveal that: 1) non-repairability factors happen in 25.7% of our studied logs and are widespread among diverse combinations of APR tools with FL strategies; 2) Inherent defect of APR tools is the most common reason for the occurrence of the non-repairability factors; 3) the impact of the non-repairability factors on the performance of APR tools can be rather significant. Our empirical study indicates that it is of great importance to eliminate the biases from the non-repairability factors. We also highlight several implications for actions that we can take to eliminate such biases.

A metamorphic testing approach for supporting program repair without the need for a test oracle

In recent years, Automated Program Repair (APR) has been extensively studied in academia and even drawn wide attention from the industry. However, APR techniques can be extremely time consuming since (1) a large number of patches can be generated for a given bug, and (2) each patch needs to be executed on the original tests to ensure its correctness. In the literature, various techniques (e.g., based on learning, mining, and constraint solving) have been proposed/studied to reduce the number of patches. Intuitively, every patch can be treated as a software revision during regression testing; thus, traditional Regression Test Selection (RTS) techniques can be leveraged to only execute the tests affected by each patch (as the other tests would keep the same outcomes) to further reduce patch execution time. However, few APR systems actually adopt RTS and there is still a lack of systematic studies demonstrating the benefits of RTS and the impact of different RTS strategies on APR. To this end, this article presents the first extensive study of widely used RTS techniques at different levels (i.e., class/method/statement levels) for 12 state-of-the-art APR systems on over 2M patches. Our study reveals various practical guidelines for bridging the gap between APR and regression testing, including: (1) the number of patches widely used for measuring APR efficiency can incur skewed conclusions, and the use of inconsistent RTS configurations can further skew the conclusions; (2) all studied RTS techniques can substantially improve APR efficiency and should be considered in future APR work; (3) method- and statement-level RTS outperform class-level RTS substantially and should be preferred; (4) RTS techniques can substantially outperform state-of-the-art test prioritization techniques for APR, and combining them can further improve APR efficiency; and (5) traditional Regression Test Prioritization (RTP) widely studied in regression testing performs even better than APR-specific test prioritization when combined with most RTS techniques. Furthermore, we also present the detailed impact of different patch categories and patch validation strategies on our findings.

Automated program repair (APR) is a promising approach to automatically fixing software bugs. Most APR techniques use tests to drive the repair process; this makes them readily applicable to realistic code bases, but also brings the risk of generating spurious repairs that overfit the available tests. Some techniques addressed the overfitting problem by targeting code using contracts (such as pre- and postconditions), which provide additional information helpful to characterize the states of correct and faulty computations; unfortunately, mainstream programming languages do not normally include contract annotations, which severely limits the applicability of such contract-based techniques. This paper presents JAID, a novel APR technique for Java programs, which is capable of constructing detailed state abstractions---similar to those employed by contract-based techniques---that are derived from regular Java code without any special annotations. Grounding the repair generation and validation processes on rich state abstractions mitigates the overfitting problem, and helps extend APR’s applicability: in experiments with the DEFECTS4J benchmark, a prototype implementation of JAID produced genuinely correct repairs, equivalent to those written by programmers, for 25 bugs---improving over the state of the art of comparable Java APR techniques in the number and kinds of correct fixes.

Recent advances in automated program repair (APR) have widely caught the attention of industrial developers as a way of reducing debugging costs. While hundreds of studies have evaluated the effectiveness of APR on open-source software, industrial case studies on APR have been rarely reported; it is still unclear whether APR can work well for industrial software. This paper reports our experience applying a state-of-the-art APR technique, ELIXIR, to large industrial software consisting of 150+ Java projects and 13 years of development histories. It provides lessons learned and recommendations regarding obstacles to the industrial use of current APR: low recall (7.7%), lack of bug-exposing tests (90%), low success rate (10%), among others. We also report the preliminary results of our ongoing improvement of ELIXIR. With some simple enhancements, the success rate of repair has been greatly improved by up to 40%.

To realistically evaluate a software testing or debugging technique, it must be run on defects and tests that are characteristic of those a developer would encounter in practice. For example, to determine the utility of a fault localization or automated program repair technique, it could be run on real defects from a bug tracking system, using real tests that are committed to the version control repository along with the fixes. Although such a methodology uses real tests, it may not use tests that are characteristic of the information a developer or tool would have in practice. The tests that a developer commits after fixing a defect may encode more information than was available to the developer when initially diagnosing the defect. This paper compares, both quantitatively and qualitatively, the developer-provided tests committed along with fixes (as found in the version control repository) versus the user-provided tests extracted from bug reports (as found in the issue tracker). It provides evidence that developer-provided tests are more targeted toward the defect and encode more information than user-provided tests. For fault localization, developer-provided tests overestimate a technique’s ability to rank a defective statement in the list of the top-n most suspicious statements. For automated program repair, developer-provided tests overestimate a technique’s ability to (efficiently) generate correct patches—user-provided tests lead to fewer correct patches and increased repair time. This paper also provides suggestions for improving the design and evaluation of fault localization and automated program repair techniques.

Fault localization is a crucial step of automated program repair, because accurately identifying program locations that are most closely implicated with a fault greatly affects the effectiveness of the patching process. An ideal fault localization technique would provide precise information while requiring moderate computational resources—to best support an efficient search for correct fixes. In contrast, most automated program repair tools use standard fault localization techniques—which are not tightly integrated with the overall program repair process, and hence deliver only subpar efficiency. In this paper, we present <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">retrospective fault localization</i> : a novel fault localization technique geared to the requirements of automated program repair. A key idea of retrospective fault localization is to reuse the outcome of failed patch validation to support mutation-based dynamic analysis—providing accurate fault localization information without incurring onerous computational costs. We implemented retrospective fault localization in a tool called <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Restore</small> —based on the <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Jaid</small> Java program repair system. Experiments involving faults from the <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Defects4J</small> standard benchmark indicate that retrospective fault localization can boost automated program repair: <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Restore</small> efficiently explores a large fix space, delivering state-of-the-art effectiveness (41 <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Defects4J</small> bugs correctly fixed, 8 of which no other automated repair tool for Java can fix) while simultaneously boosting performance (speedup over 3 compared to <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Jaid</small> ). Retrospective fault localization is applicable to any automated program repair techniques that rely on fault localization and dynamic validation of patches.

Fault localization (FL) and automated program repair (APR) are two main tasks of automatic software debugging. Compared with traditional methods, deep learning-based approaches have been demonstrated to achieve better performance in FL and APR tasks. However, the existing deep learning-based FL methods ignore the deep semantic features or only consider simple code representations. And for APR tasks, existing template-based APR methods are weak in selecting the correct fix templates for more effective program repair, which are also not able to synthesize patches via the embedded end-to-end code modification knowledge obtained by training models on large-scale bug-fix code pairs. Moreover, in most of FL and APR methods, the model designs and training phases are performed separately, leading to ineffective sharing of updated parameters and extracted knowledge during the training process. This limitation hinders the further improvement in the performance of FL and APR tasks. To solve the above problems, we propose a novel approach called MTL-TRANSFER, which leverages a multi-task learning strategy to extract deep semantic features and transferred knowledge from different perspectives. First, we construct a large-scale open-source bug datasets and implement 11 multi-task learning models for bug detection and patch generation sub-tasks on 11 commonly used bug types, as well as one multi-classifier to learn the relevant semantics for the subsequent fix template selection task. Second, an MLP-based ranking model is leveraged to fuse spectrum-based, mutation-based and semantic-based features to generate a sorted list of suspicious statements. Third, we combine the patches generated by the neural patch generation sub-task from the multi-task learning strategy with the optimized fix template selecting order gained from the multi-classifier mentioned above. Finally, the more accurate FL results, the optimized fix template selecting order, and the expanded patch candidates are combined together to further enhance the overall performance of APR tasks. Our extensive experiments on widely-used benchmark Defects4J show that MTL-TRANSFER outperforms all baselines in FL and APR tasks, proving the effectiveness of our approach. Compared with our previously proposed FL method TRANSFER-FL (which is also the state-of-the-art statement-level FL method), MTL-TRANSFER increases the faults hit by 8/11/12 on Top-1/3/5 metrics (92/159/183 in total). And on APR tasks, the number of successfully repaired bugs of MTL-TRANSFER under the perfect localization setting reaches 75, which is 8 more than our previous APR method TRANSFER-PR. Furthermore, another experiment to simulate the actual repair scenarios shows that MTL-TRANSFER can successfully repair 15 and 9 more bugs (56 in total) compared with TBar and TRANSFER, which demonstrates the effectiveness of the combination of our optimized FL and APR components.

Automated Program Repair (APR) is an area of research focused on the automatic generation of bug-fixing patches. Current APR approaches present some limitations, namely overfitted patches and low maintainability of the generated code. Several works are tackling this problem by attempting to come up with algorithms producing higher quality fixes. In this experience paper, we explore an alternative. We believe that by using existing low-cost APR techniques, fast enough to provide real-time feedback, and encouraging the developer to work together with the APR inside the IDE, will allow them to immediately discard proposed fixes deemed inappropriate or prone to reduce maintainability. Most developers are familiar with real-time syntactic code suggestions, usually provided as code completion mechanisms. What we propose are semantic code suggestions, such as code fixes, which are seldom automatic and rarely real-time. To test our hypothesis, we implemented a Visual Studio Code extension (named pAPRika), which leverages unit tests as specifications and generates code variations to repair bugs in JavaScript. We conducted a preliminary empirical study with 16 participants in a crossover design. Our results provide evidence that, although incorporating APR in the IDE improves the speed of repairing faulty programs, some developers are too eager to accept patches, disregarding maintenance concerns.

Automated program repair (APR) is a technology that identifies and repairs bugs automatically. However, repairing multi-chunk bugs remains a long-standing and challenging problem because an APR technique must consider dependencies and then reduce the large patch space. In addition, little is known about how to combine individual candidate patches even though multi-chunk bugs require combinations. Therefore, we propose a novel APR technique called multi-code repair (MCRepair), which applies a buggy block, patch optimization, and CodeBERT to target multi-chunk bugs. A buggy block is a novel method that binds buggy chunks into a multi-buggy chunk and preprocesses the chunk with its buggy contexts for patch space reduction and dependency problems. Patch optimization is a novel strategy that effectively combines the generated candidate patches with patch space reduction. In addition, CodeBERT, a BERT for source code datasets, is fine-tuned to address the lack of datasets and out-of-vocabulary problems. We conducted several experiments to evaluate our approach on six project modules of Defects4J. In the experiments using Defects4J, MCRepair repaired 66 bugs, including 22 multi-chunk bugs. Moreover, it fixed 19 unique bugs, including nine multi-chunk bugs, and improved 45--266% performance than the baselines.

A method to identify overfitting program repair patches based on expression tree

Automatic Program Repair works by generating patches automatically to fix software bugs. This domain is very popular among the software engineering researchers as it helps the developers to mitigate tiresome debugging effort and increase productivity. This domain mostly focuses on how to develop better program repair techniques. However, a systematic literature review is needed to help the researchers for getting specific and scientific overview of this domain. This paper conducts a systematic literature review on 324 papers for analyzing the trendiness and associated factors of this domain. After that, this paper considers 16 papers to thoroughly review and states the evolution of automatic program repair. Hopefully, this study will be a valuable resource for the researchers of this domain.

CGenProg: Adaptation of cartesian genetic programming with migration and opposite guesses for automatic repair of software regression faults

Introduction: Controversy exists for ustekinumab concentrations needed in Crohn’s disease (CD). No data exist comparing ustekinumab concentrations and validated radiologic outcomes. We characterized these relationships and clarified concentrations needed. Methods: CD patients on maintenance ( > 16 weeks) ustekinumab with both ustekinumab concentrations and simplified magnetic resonance index of activity (sMaRIA) scoring were included. Ustekinumab concentrations were compared between those with and without (1) radiologic remission (sMaRIA < 2), (2) severe radiologic inflammation (sMaRIA < 3) and (3) fecal calprotectin (FCP) biomarker remission (FCP < 50 μg/g). Area under the receiver-operating characteristic (AUROC) curve determined optimal ustekinumab concentrations. Outcomes were compared between patients above and below identified ustekinumab thresholds. Results: Thirty-eight paired ustekinumab concentrations and MRE were included. Ustekinumab concentrations were higher with radiologic remission (11.4 μg/mL vs. 6.4 μg/mL, P=.005) and had good diagnostic accuracy for radiologic remission (AUROC 0.76, 95% CI 0.60 – 0.91) and for absence of severe inflammation (AUROC 0.71, 95% CI 0.55 – 0.88, optimal concentration 8.4 μg/mL). With ustekinumab ≥ 8.4 μg/mL, higher proportions had radiologic remission (63.2% vs. 21.1%, P=.01) and absence of severe inflammation (78.9% vs. 36.8%, P=.01) compared to patients with lower concentrations. Ustekinumab concentrations had good diagnostic accuracy (AUROC 0.73, 95% CI 0.52 – 0.94) for FCP biomarker remission (optimal concentration: 6.1 μg/mL). Patients with ustekinumab concentrations ≥ 6.1 μg/mL had higher proportions with biomarker remission (72.2% vs. 12.5% P < .01) compared to those with lower concentrations. (Figure) Conclusion: Ustekinumab concentrations are associated with radiologic and biomarker outcomes in CD. These data validate the need for higher ustekinumab concentrations.Figure 1.: (A) Receiver-operating characteristic curve analysis for sMaRIA score of < 2, absence of any inflammation, based on ustekinumab trough concentrations. The optimal concentration is 8.4 μg/mL as indicated by the best receiver-operating characteristic curve (area under curve, 0.76; sensitivity, 75%; specificity 68%). (B) Receiver-operating characteristic curve analysis for sMaRIA score of < 3, absence of severe inflammation, based on ustekinumab concentrations. The optimal serum concentration is 8.4 μg/mL as indicated by the best receiver-operating characteristic curve (area under curve, 0.71).

Automated debugging is an emerging research field that aims to automatically find and repair bugs. In this field, Fault Localization (FL) and Automated Program Repair (APR) gain the most research efforts. Most recently, researchers have adopted pre-trained Large Language Models (LLMs) to facilitate FL and APR and their results are promising. However, the LLMs they used either vanished (such as Codex) or outdated (such as early versions of GPT). In this paper, we evaluate the performance of recent commercial closed-source general-purpose LLMs on FL and APR, i.e., ChatGPT 3.5, ERNIE Bot 3.5, and IFlytek Spark 2.0. We select three popular LLMs and evaluate them on 120 real-world Java bugs from the benchmark Defects4J. For FL and APR, we designed three kinds of prompts for each, considering different kinds of information. The results show that these LLMs could successfully locate 53.3% and correctly fix 12.5% of these bugs.