How Far Have We Progressed in Identifying Self-admitted Technical Debts? A Comprehensive Empirical Study

Technical debt is a metaphor to reflect the tradeoff software engineers make between short-term benefits and long-term stability. Self-admitted technical debt (SATD), a variant of technical debt, has been proposed to identify debt that is intentionally introduced during software development, e.g., temporary fixes and workarounds. Previous studies have leveraged human-summarized patterns (which represent n-gram phrases that can be used to identify SATD) or text-mining techniques to detect SATD in source code comments. However, several characteristics of SATD features in code comments, such as vocabulary diversity, project uniqueness, length, and semantic variations, pose a big challenge to the accuracy of pattern or traditional text-mining-based SATD detection, especially for cross-project deployment. Furthermore, although traditional text-mining-based method outperforms pattern-based method in prediction accuracy, the text features it uses are less intuitive than human-summarized patterns, which makes the prediction results hard to explain. To improve the accuracy of SATD prediction, especially for cross-project prediction, we propose a Convolutional Neural Network-- (CNN) based approach for classifying code comments as SATD or non-SATD. To improve the explainability of our model’s prediction results, we exploit the computational structure of CNNs to identify key phrases and patterns in code comments that are most relevant to SATD. We have conducted an extensive set of experiments with 62,566 code comments from 10 open-source projects and a user study with 150 comments of another three projects. Our evaluation confirms the effectiveness of different aspects of our approach and its superior performance, generalizability, adaptability, and explainability over current state-of-the-art traditional text-mining-based methods for SATD classification.

Technical debt refers to decisions made during the design and development of software that postpone the resolution of technical problems or the enhancement of the software’s features to a later date. If not properly managed, technical debt can put long-term software quality and maintainability at risk. Self-admitted technical debt is defined as the addition of specific comments to source code as a result of conscious and deliberate decisions to accumulate technical debt. In this paper, we will look at the presence of self-admitted technical debt in open-source blockchain projects, which are characterized by the use of a relatively novel technology and the need to generate trust. The self-admitted technical debt was analyzed using NLP techniques for the classification of comments extracted from the source code of ten projects chosen based on capitalization and popularity. The analysis of self-admitted technical debt in blockchain projects was compared with the results of previous non-blockchain open-source project analyses. The findings show that self-admitted design technical debt outnumbers requirement technical debt in blockchain projects. The analysis discovered that some projects had a low percentage of self-admitted technical debt in the comments but a high percentage of source code files with debt. In addition, self-admitted technical debt is on average more prevalent in blockchain projects and more equally distributed than in reference Java projects.If not managed, the relatively high presence of detected technical debt in blockchain projects could represent a threat to the needed trust between the blockchain system and the users. Blockchain projects development teams could benefit from self-admitted technical debt detection for targeted technical debt management.

Context: Software development teams constantly opt for faster, lower quality solutions to solve current problems without planning for the future. This situation will have a negative long-term impact and is called technical debt. Similar to a financial debt, technical debts require interest payments and must be managed and detected so that the team can evaluate the best way to deal with them. One way to detect technical debts is through classification of source code comments. Developers often insert comments warning of the need to improve their own code in the future. This is known as Self-Admitted Technical Debt (SATD). Objective: Combine Word2vec for word embedding with a Long short-term memory (LSTM) neural network model to identify SATDs from comments in source code and compare with other studies and LSTM without word embedding. Method: We plan and execute an experimental process with model’s effectiveness data validation. Results: In general, the classification improves when all SATD types were grouped in a single label. In relation to other studies, the LSTM model with Word2vec achieved better recall and f-measure. The LSTM model without word embedding achieves greater recall, but perform worse in precision and f-measure. Conclusion: We found evidence that LSTM models combined with word embedding are promising for the development a more effective SATD classifier.

Technical Debt (TD) has been defined as being not quite right yet, and its presence is often self-admitted by developers through comments. The purpose of such comments is to keep track of TD and appropriately address it when possible. Building on a previous quantitative investigation by Maldonado et al. on the removal of self-admitted technical debt (SATD), in this paper we perform an in-depth quantitative and qualitative study of how SATD is addressed in five Java open source projects. On the one hand, we look at whether SATD is removed, and the extent to which the SATD removal is being documented. We found that that (i) between 20% and 50% of SATD comments are accidentally removed while entire classes or methods are dropped, (ii) 8% of the SATD removal is acknowledged in commit messages, and (iii) while most of the changes addressing SATD require complex source code changes, very often SATD is addressed by specific changes to method calls or conditionals. Our results can be used to better plan TD management or learn patterns for addressing certain kinds of TD and provide recommendations to developers.

Most previous research focuses on finding Self-Admitted Technical Debt (SATD) or detecting bugs alone, rather to addressing the concurrent identification of both issues. These study investigations solely identify and classify the SATD or faults, without identifying or categorising bugs based on SATD. Furthermore, the majority of current methodologies do not incorporate contemporary deep learning techniques. This work presents an innovative method utilising deep learning techniques to discover and classify Self-Admitted Technical Debt (SATD) and to find defects in software comments associated with SATD. The proposed approach detects this issue and classifies and enhances the understanding and localization of defects. The methodology involves developing a deep learning model using diverse data from repositories, including Apache, Mozilla Firefox, and Eclipse. The chosen data set comprises projects, designated SATD examples, and bug instances, facilitating thorough model training and evaluation. The methodology comprises data analysis, preprocessing, and model training utilising deep learning architectures such as LSTM, BI-LSTM, GRU, and BI-GRU, with Transformer models like BERT and GPT-3, in conjunction with machine learning methods. The performance evaluation criteria, such as precision, recall, accuracy, and F1 score, illustrate the efficacy of the suggested method. Comparative assessment with existing methodologies underscores notable improvements, while cross-validation ensures model resilience. All deep learning models achieved an accuracy and precision of 0.98, and transformer models achieved slightly higher metrics. The GPT-3 achieved an overall accuracy of 0.984. We see that using the transfer learning approach the transformer model (GPT-3) outperformed the other as it achieved an overall accuracy of 0.96 and F1-Score of 0.96, precision of 0.96, and recall of 0.96, and deep learning models (LSTM, GRU) also give significant performance, but their accuracy is slightly lower than baseline model (Naive Bayes). The research has significant implications for software engineering, providing a comprehensive method for software quality assessment and maintenance. It enhances software architecture technical debt (SATD) and knowledge of bugs, as well as prioritization and resource allocation for software maintenance and evolution. The research's ramifications go beyond academia; it has a direct impact on business procedures and makes it easier to create software systems that are reliable and long-lasting.

The technical debt (TD) in a software project refers to the adoption of an inadequate solution from its design to the source code. When developers admit the presence of technical debt in the source code, through comments or commit messages, it is called self-admitted technical debt (SATD). This aspect of TD has been the subject of numerous research studies, which have investigated its distribution, the impact on software quality, and removal. Therefore, this work focuses on the relationship between SATD and TD values. In particular, the study aims to compare the admitted technical debt with respect to its objective measure. In fact, the trends of TD values during SATD removals have been studied. This was done thanks to the use of an SATD dataset and their related removals in four open source projects. Instead, the SonarQube tool was used to measure TD values. Thanks to this work, it turned out that SATD removals in a few cases correspond to an effective reduction of TD values, while in numerous cases, the classes indicated are removed.

In software projects, technical debt takes place when a developer adopting a trivial solution containing quick and easy shortcuts to implement over a suitable solution that can take a longer time to solve a problem. This can cause major additional costs leading to negative impacts for software maintenance since those shortcuts might need to be reworked in the future. Detecting technical debt early can help a team cope with those risks. In this paper, we focus on Self-Admitted Technical Debt (SATD) that is a debt intentionally produced by developers. We propose an automated model to identify two most common types of self-admitted technical debt, requirement and design debt, from source code comments. We combine N-gram IDF and auto-sklearn machine learning to build the model. With the empirical evaluation on ten projects, our approach outperform the baseline method by improving the performance over 20% when identifying requirement self-admitted technical debt and achieving an average F1-score of 64% when identifying design self-admitted technical debt.

Self-Admitted Technical Debt (SATD) is primarily studied in Object-Oriented (OO) languages and traditionally commercial software. However, scientific software coded in dynamically-typed languages such as R differs in paradigm, and the source code comments’ semantics are different (i.e., more aligned with algorithms and statistics when compared to traditional software). Additionally, many Software Engineering topics are understudied in scientific software development, with SATD detection remaining a challenge for this domain. This gap adds complexity since prior works determined SATD in scientific software does not adjust to many of the keywords identified for OO SATD, possibly hindering its automated detection. Therefore, we investigated how classification models (traditional machine learning, deep neural networks, and deep neural Pre-Trained Language Models (PTMs)) automatically detect SATD in R packages. This study aims to study the capabilities of these models to classify different TD types in this domain and manually analyze the causes of each in a representative sample. Our results show that PTMs (i.e., RoBERTa) outperform other models and work well when the number of comments labelled as a particular SATD type has low occurrences. We also found that some SATD types are more challenging to detect. We manually identified sixteen causes, including eight new causes detected by our study. The most common cause was failure to remember, in agreement with previous studies. These findings will help the R package authors automatically identify SATD in their source code and improve their code quality. In the future, checklists for R developers can also be developed by scientific communities such as rOpenSci to guarantee a higher quality of packages before submission.

Self-Admitted Technical Debt (SATD), cases where developers intentionally acknowledge suboptimal solutions in code through comments, poses a significant challenge to software maintainability. Left unresolved, SATD can degrade code quality, increase maintenance costs, and hinder long-term software sustainability. Manually identifying and resolving SATD is time-consuming and error-prone, often competing with feature development and release deadlines. Automating SATD repayment is therefore essential to alleviate developer burden. While Large Language Models (LLMs) have shown promise in tasks like code generation and program repair, their potential in automated SATD repayment remains underexplored. In this paper, we identify three key technical challenges in training and evaluating LLMs for SATD repayment: (1) dataset representativeness and scalability, (2) removal of irrelevant SATD repayment samples, and (3) limitations of existing evaluation metrics, i.e., BLEU and CrystalBLEU. To address the first two dataset-related challenges, we adopt a language-independent SATD tracing tool and design a 10-step filtering pipeline to extract SATD repayments from repository commit histories, leveraging LLM-as-judge for relevance filtering. This results in two new large-scale SATD repayment datasets: 58,722 items for Python and 97,347 items for Java. To improve evaluation, we introduce two diff-based metrics, BLEU-diff and CrystalBLEU-diff, which measure code changes rather than whole code, thereby mitigating the impact of SATD-irrelevant source code. Additionally, we propose a new metric, Line-Level Exact Match on Diff (LEMOD), which is both interpretable and informative, providing fine-grained insights into SATD repayment quality. Using our new benchmarks and evaluation metrics, we evaluate two types of automated SATD repayment methods: fine-tuning smaller models (a few hundred million parameters) and prompt engineering with large-scale models, including four SOTA open-source LLMs and GPT-4o-mini. Our results reveal that while fine-tuned smaller models achieve Exact Match (EM) scores comparable to prompt-based approaches, they underperform on BLEU-based metrics and LEMOD. The best EM performance is achieved by Gemma-2-9B, correctly addressing 10.1% of Python SATDs and 8.1% of Java SATDs with a simple prompt. The only previous study in this domain focused solely on Java, achieving an EM score of 2.3%. When using BLEU-diff, CrystalBLEU-diff, and LEMOD metrics, Llama-3.1-70B-Instruct and GPT-4o-mini deliver the highest performance. While our three proposed metrics demonstrate strong correlations with EM (0.65~0.84), both BLEU and CrystalBLEU show almost no correlation with this intuitive metric (0.01~0.08), highlighting their largely unpredictable behavior when applied directly to the entire code rather than the diff. Our work contributes a robust benchmark, improved evaluation metrics, and a comprehensive evaluation of LLMs, advancing research on automated SATD repayment.

When developers use different keywords such as TODO and FIXME in source code comments to describe self-admitted technical debt (SATD), we refer it as Keyword-Labeled SATD (KL-SATD). We study KL-SATD from 33 software repositories with 13,588 KL-SATD comments. We find that the median percentage of KL-SATD comments among all comments is only 1,52%. We find that KL-SATD comment contents include words expressing code changes and uncertainty, such as remove, fix, maybe and probably. This makes them different compared to other comments. KL-SATD comment contents are similar to manually labeled SATD comments of prior work. Our machine learning classifier using logistic Lasso regression has good performance in detecting KL-SATD comments (AUC-ROC 0.88). Finally, we demonstrate that using machine learning we can identify comments that are currently missing but which should have a SATD keyword in them. Automating SATD identification of comments that lack SATD keywords can save time and effort by replacing manual identification of comments. Using KL-SATD offers a potential to bootstrap a complete SATD detector.

SCGRU: A general approach for identifying multiple classes of self-admitted technical debt with text generation oversampling

Developers of deep learning applications (shortened as application developers) commonly use deep learning frameworks in their projects. However, due to time pressure, market competition, and cost reduction, developers of deep learning frameworks (shortened as framework developers) often have to sacrifice software quality to satisfy a shorter completion time. This practice leads to technical debt in deep learning frameworks, which results in the increasing burden to both the application developers and the framework developers in future development.In this paper, we analyze the comments indicating technical debt (self-admitted technical debt) in 7 of the most popular open-source deep learning frameworks. Although framework developers are aware of such technical debt, typically the application developers are not. We find that: 1) there is a significant number of technical debt in all the studied deep learning frameworks. 2) there is design debt, defect debt, documentation debt, test debt, requirement debt, compatibility debt, and algorithm debt in deep learning frameworks. 3) the majority of the technical debt in deep learning framework is design debt (24.07% - 65.27%), followed by requirement debt (7.09% - 31.48%) and algorithm debt (5.62% - 20.67%). In some projects, compatibility debt accounts for more than 10%. These findings illustrate that technical debt is common in deep learning frameworks, and many types of technical debt also impact the deep learning applications. Based on our findings, we highlight future research directions and provide recommendations for practitioners.

<p>Self-Admitted Technical Debt (SATD) is a workaround for current gains and subsequent software quality in software comments. Some studies have been conducted using NLP-based techniques or CNN-based classifiers. However, there exists a class imbalance problem in different software projects since the software code comments with SATD features are significantly less than those without Non-SATD. Therefore, to design a classification model with the ability of dealing with this class imbalance problem is necessary for SATD detection. We propose an improved loss function based on information entropy. Our proposed function is studied in a variety of application scenarios. Empirical research on 10 JAVA software projects is conducted to show the competitiveness of our new approach. We find our proposed approach can perform significantly better than state-of-the-art baselines. </p> <p> </p>

Cross-disciplinary and cross-life-cycle-phase Technical Debt in automated Production Systems: two industrial case studies and a survey

Chapter 2 - Measuring and Monitoring Technical Debt