Severe software defects can lead to significant issues and even result in substantial financial losses. As a result, automated code defect detection has garnered widespread attention. To fix these defects as soon as possible, J ust- I n- T ime D efect P rediction (JIT-DP) and J ust- I n- T ime D efect L ocalization (JIT-DL) have been investigated. Specifically, JIT-DP aims to predict commit-level defects when a code change is submitted, and JIT-DL aims to perform line-level defect localization before they lead to failures. Several methods have been developed for JIT-DP and JIT-DL, with those using C ode P re- T rained M odels (CPTMs) leading to the best results. However, most previous studies concentrate on JIT-DP, even though JIT-DL is arguably more crucial. Identifying which specific lines of a commit are defective facilitates the prediction of whether the commit is defective. Therefore, we propose a M ulti- T ask L earning method (JIT-MTL) designed to address both JIT-DP and JIT-DL. Specifically, we train a CPTM to simultaneously identify defective commits and lines. The line-level predictions are directly used for JIT-DL, while the commit-level predictions for the same commit are merged to determine the results for JIT-DP. To enhance the accuracy of line-level predictions, we provide the CPTM with additional commit-level information, including commit-level defect code representation and expert features. Experimental results on the JIT-Defect4J dataset show that JIT-MTL outperforms the state-of-the-art baselines, with improvements of 14% in F1 in JIT-DP, 53.9% in Top-10 Accuracy, and 36.4% in Top-5 Accuracy in JIT-DL.

Trusting software systems, particularly autonomous ones, is challenging. To address this, formal verification techniques can ensure these systems behave as expected. Runtime Verification (RV) is a leading, lightweight method for verifying system behaviour during execution. However, traditional RV assumes perfect information, meaning the monitoring component perceives everything accurately. This assumption often fails, especially with autonomous systems operating in real-world environments where sensors might be faulty. Additionally, traditional RV considers the monitor to be passive, lacking the capability to interpret the system’s information and thus unable to address incomplete data. In this work, we extend standard RV of Linear Temporal Logic properties to accommodate scenarios where the monitor has imperfect information and behaves rationally. We outline the necessary engineering steps to update the verification pipeline and demonstrate our implementation in a case study involving robotic systems.

This report presents a framework that enables the reproduction and extension of our empirical evaluations using Large Language Models (LLMs) for automated web-form-test generation. The framework includes HTML pruning , context construction , prompt design , LLM communication , and web-form-test insertion . It involves the construction of three types of prompts (from HTML) to guide the test generation: Raw HTML for Task Prompt (RH-P); LLM-Processed HTML for Task Prompt (LH-P); and Parser-Processed HTML for Task Prompt (PH-P). The framework provides an LLM communication module that standardizes interactions with provider Application Programming Interfaces (APIs). Our study utilized public-API models and demonstrated that PH-P consistently achieved a higher successfully-submitted rate (SSR) than RH-P and LH-P. To support the replication of our work, we have released the source code, a dataset subset, and the relevant scripts.

Self-Admitted Technical Debt (SATD), cases where developers intentionally acknowledge suboptimal solutions in code through comments, poses a significant challenge to software maintainability. Left unresolved, SATD can degrade code quality, increase maintenance costs, and hinder long-term software sustainability. Manually identifying and resolving SATD is time-consuming and error-prone, often competing with feature development and release deadlines. Automating SATD repayment is therefore essential to alleviate developer burden. While Large Language Models (LLMs) have shown promise in tasks like code generation and program repair, their potential in automated SATD repayment remains underexplored. In this paper, we identify three key technical challenges in training and evaluating LLMs for SATD repayment: (1) dataset representativeness and scalability, (2) removal of irrelevant SATD repayment samples, and (3) limitations of existing evaluation metrics, i.e., BLEU and CrystalBLEU. To address the first two dataset-related challenges, we adopt a language-independent SATD tracing tool and design a 10-step filtering pipeline to extract SATD repayments from repository commit histories, leveraging LLM-as-judge for relevance filtering. This results in two new large-scale SATD repayment datasets: 58,722 items for Python and 97,347 items for Java. To improve evaluation, we introduce two diff-based metrics, BLEU-diff and CrystalBLEU-diff, which measure code changes rather than whole code, thereby mitigating the impact of SATD-irrelevant source code. Additionally, we propose a new metric, Line-Level Exact Match on Diff (LEMOD), which is both interpretable and informative, providing fine-grained insights into SATD repayment quality. Using our new benchmarks and evaluation metrics, we evaluate two types of automated SATD repayment methods: fine-tuning smaller models (a few hundred million parameters) and prompt engineering with large-scale models, including four SOTA open-source LLMs and GPT-4o-mini. Our results reveal that while fine-tuned smaller models achieve Exact Match (EM) scores comparable to prompt-based approaches, they underperform on BLEU-based metrics and LEMOD. The best EM performance is achieved by Gemma-2-9B, correctly addressing 10.1% of Python SATDs and 8.1% of Java SATDs with a simple prompt. The only previous study in this domain focused solely on Java, achieving an EM score of 2.3%. When using BLEU-diff, CrystalBLEU-diff, and LEMOD metrics, Llama-3.1-70B-Instruct and GPT-4o-mini deliver the highest performance. While our three proposed metrics demonstrate strong correlations with EM (0.65~0.84), both BLEU and CrystalBLEU show almost no correlation with this intuitive metric (0.01~0.08), highlighting their largely unpredictable behavior when applied directly to the entire code rather than the diff. Our work contributes a robust benchmark, improved evaluation metrics, and a comprehensive evaluation of LLMs, advancing research on automated SATD repayment.

Copy-and-paste of code is a common practice in software development, especially with the availability of myriad open-source projects. Reusing existing code fragments may accelerate the development process and ensure better quality of software if high-quality code fragments are reused. Therefore, code cloning might not be avoided even though it can increase software maintenance cost as bugs can be propagated inadvertently by code cloning and go beyond the boundary of a system. Generally speaking, reusable code clones are highly popular, have fewer bugs, and can stand the test of time thus exist in the system for a long term. Current code reuse studies primarily leverage API usage and method clone structures to provide coding recommendations. However, the existing approaches focus on the functional utility of code snippets without considering the quality aspects, such as fault resiliency, when analyzing code reuse. Without clone quality in mind, the virtues of code cloning are severely diminished. To help developers determine if the code clones are reusable or not, we leverage Machine Learning classifiers and Large Language Models (LLMs) to automatically identify reusable code clones from three perspectives: clone prevalence (i.e., the number of clone siblings), clone fault-resiliency (i.e., the percentage of non-buggy commits versus buggy commits), and clone longevity (i.e., the clone genealogy length). Our approach achieves a median AUC of 0.73, with an F1-score of 0.89, based on experiments conducted on 60 open-source projects, consisting of 30 Java and 30 C projects, which collectively encompass 538,598 commits. The results show that CountFollowers (i.e., number of people following the contributors that wrote the clones), SimilarityClonePaths (i.e., the Jaccard similarity coefficient among clone paths), and CountContributors (i.e., number of distinct contributors that access the clone group) provide the most explanatory power that contributes to correctly classifying reusable code clones. Hence, practitioners can utilize our classifiers and insights from our findings to make more reliable use of code clones and prioritize the use of high-quality clones in their clone management activities.

WebAssembly (Wasm) is an emerging binary format that serves as a compilation target for over 40 programming languages. Wasm runtimes provide execution environments that enhance portability by abstracting away operating systems and hardware details. A key component in these runtimes is the WebAssembly System Interface (WASI), which manages interactions with operating systems, like file operations. Considering the critical role of Wasm runtimes, the community has aimed to detect their implementation bugs. However, no work has focused on WASI-specific bugs that can affect the original functionalities of running Wasm binaries and cause unexpected results. To fill the void, we present DrWASI , the first general-purpose differential testing framework for WASI implementations. Our approach uses a large language model to generate seeds and applies variant and environment mutation strategies to expand and enrich the test case corpus. We then perform differential testing across major Wasm runtimes. By leveraging dynamic and static information collected during and after the execution, DrWASI can identify bugs. Our evaluation shows that DrWASI uncovered 33 unique bugs, with all confirmed and 7 fixed by developers. This research represents a pioneering step in exploring a promising yet under-explored area of the Wasm ecosystem, providing valuable insights for stakeholders.

In software engineering, empirical studies on automated fault localization (FL) methods mainly focus on general software, and substantial progress has been made. However, the applicability and efficacy of these methods in specialized, domain-specific software like industrial software remains under-explored. Such specialized software is usually characterized by complex inputs and iterative computing paradigms, which could significantly influence the effectiveness of existing FL methods. To address this gap, this study takes a typical categorical of industrial software (i.e., computer-aided engineering (CAE) projects) as a case study, to investigate the feasibility and effectiveness of state-of-the-art FL methods within CAE projects. Through the reproduction of 76 real-world bugs from three widely used CAE projects (i.e., FDS, deal.II, and MFEM), we find that even the most precise FL methods require developers to examine on average 467.18 statements before finding bugs and can take 208.13 hours to execute. The complex inputs and long-term computation characteristics of CAE projects further increase the difficulty of FL. Moreover, FL on CAE also faces challenges, such as insufficient differentiation of coverage information and missing CAE-specific FL features. Based on our findings, we improve FL on CAE projects by proposing a set of CAE main module-based features, which improve the best-performed FL method in this study (i.e., DeepFL) by 35.93% and 45%, in terms of MAR and MFR , respectively.

In recent years, the practice of fuzzing Deep Learning (DL) APIs has received significant attention in the software engineering community. Many API-level DL fuzzers have been proposed to test individual DL APIs by generating malformed input. Although these fuzzers have been effective in detecting bugs and outperforming prior work, there remains a gap in benchmarking them against ground-truth, real-world bugs in DL libraries. Existing comparisons among these API-level DL fuzzers primarily focus on the bugs detected but do not offer a comprehensive, in-depth evaluation of the fuzzers’ effectiveness. In this work, we perform the first in-depth evaluation of state-of-the-art API-level DL fuzzers that generate tests for single DL APIs, focusing on their effectiveness against real-world bugs. We manually created an extensive benchmark dataset, including 517 real-world DL bugs collected from PyTorch and TensorFlow libraries that can be triggered by malformed inputs. We then apply seven state-of-the-art DL fuzzers— FreeFuzz , DeepRel , NablaFuzz , DocTer , ACETest , TitanFuzz , and FuzzGPT —to our benchmark dataset, following their respective instructions. Our results show that these fuzzers detect only 6.5% (34 out of 517) of the unique real-world bugs in the dataset. Our analysis identifies two dominant factors that impact the effectiveness of these fuzzers in detecting real-world bugs. These findings suggest opportunities for improving the performance of fuzzers in future work. Overall, this study extends previous work on DL fuzzers by providing an extensive evaluation and benchmarking platform for fuzzing DL libraries.

Software testing and defect prediction are widely recognized quality assurance activities that can contribute to the effectiveness and efficiency of Just-In-Time Software Quality Assurance (JIT-SQA) by automating processes and reducing software maintenance costs. However, most studies treat them separately, with limited exploration of how defect prediction can guide testing and how testing can support prediction. Existing methods are not tailored for JIT-SQA and often ignore change-specific characteristics such as concept drift and verification latency, leading to unstable defect prediction. Moreover, test annotations on recent code changes are rarely used to update models, missing an opportunity to improve accuracy. Beyond this, effectively leveraging test-annotated samples remains an open research question. On the one hand, these samples frequently suffer from one-sided label noise, as instances that are not identified as defective during testing may still contain defects. On the other hand, defect-inducing changes are relatively infrequent, leading to a significant class imbalance issue. Therefore, exploring how to utilize test annotations to mitigate class imbalance and enhance prediction robustness presents a critical challenge. To fill this research gap, we propose an automatic JIT-SQA framework that incorporates defect prediction and testing processes collaboratively. Specifically, we direct testing activities through defect prediction models and design a novel update mechanism that integrates waiting time with testing feedback to update the predictor in real time. Moreover, we further design a reinforcement learning-based dynamic selection mechanism to mitigate the instability in defect prediction performance over time. Our framework facilitates mutual guidance between automatic testing and defect prediction, leveraging shared knowledge within the JIT-SQA system. This integration holds significant potential for producing high-quality software within modern development timelines. Notably, this study is the first to investigate the integrated application of these two quality assurance activities within a unified framework for JIT-SQA throughout the software development process. We conducted experiments using both statically typed (Java) and dynamically typed (Python) languages by integrating various testing techniques. The results indicate that, after implementing the novel update and dynamic selection mechanisms, our framework significantly outperforms simple integrated frameworks of defect prediction and testing. Specifically, the Recall improves by 21.6%, 10.5%, and 6.3% compared with the testing-integrated versions of ODaSC, ECo-HumLa, and HumLa, respectively, while the F1 score improves by 13.2%, 7.0%, and 5.1%. At the same time, in terms of guiding testing activities, the total number of tests is reduced by 2.9%, 19.5%, and 24.8% compared with the same baselines. These findings demonstrate that our framework not only mitigates the adverse effects of concept drift, verification latency, and class imbalance in code evolution scenarios but also effectively guides testing to reduce resource consumption.

Many methods of TPL recommendation are developed to help app developers find available TPLs, but current methods encounter two common limitations when evaluating the match degree between the app and the candidate TPL. First, they mainly complete the evaluation task based on the TPL context, but ignore the function information of the app. In fact, since TPLs are primarily used to support the implementation of various app functions, this information is critical for making an informed decision. Second, they focus on the relationship between the candidate TPL and the entire TPL context. However, it is crucial to emphasize that a candidate TPL is worth considering as long as it can collaborate with certain TPLs in the TPL context. In this study, we propose a novel model called Atten-TPL that can evaluate the match degree between the app and the candidate TPL by mining the relationships among the candidate TPL, TPL context, and app functions. By using an attention mechanism, Atten-TPL can pay more attention to the TPLs in the TPL context that belong to the same task domain as the candidate TPL, thus mitigating the second limitation of current methods. The experiments indicate that Atten-TPL outperforms prevalent methods of recommendation.