Cross-project defect prediction: a large scale experiment on data vs. domain vs. process

Cross‐project defect prediction (CPDP), where data from different software projects are used to predict defects, has been proposed as a way to provide data for software projects that lack historical data. Evaluations of CPDP models using the Nearest Neighbour (NN) Filter approach have shown promising results in recent studies. A key challenge with defect‐prediction datasets is class imbalance, that is, highly skewed datasets where non‐buggy modules dominate the buggy modules. In the past, data resampling approaches have been applied to within‐projects defect prediction models to help alleviate the negative effects of class imbalance in the datasets. To address the class imbalance issue in CPDP, the authors assess the impact of data resampling approaches on CPDP models after the NN Filter is applied. The impact on prediction performance of five oversampling approaches (MAHAKIL, SMOTE, Borderline‐SMOTE, Random Oversampling and ADASYN) and three undersampling approaches (Random Undersampling, Tomek Links and One‐sided selection) is investigated and results are compared to approaches without data resampling. The authors examined six defect prediction models on 34 datasets extracted from the PROMISE repository. The authors' results show that there is a significant positive effect of data resampling on CPDP performance, suggesting that software quality teams and researchers should consider applying data resampling approaches for improved recall ( pd ) and g ‐measure prediction performance. However, if the goal is to improve precision and reduce false alarm ( pf ) then data resampling approaches should be avoided.

There has been a great deal of interest in defect prediction: using prediction models trained on historical data to help focus quality-control resources in ongoing development. Since most new projects don't have historical data, there is interest in cross-project prediction: using data from one project to predict defects in another. Sadly, results in this area have largely been disheartening. Most experiments in cross-project defect prediction report poor performance, using the standard measures of precision, recall and F-score. We argue that these IR-based measures, while broadly applicable, are not as well suited for the quality-control settings in which defect prediction models are used. Specifically, these measures are taken at specific threshold settings (typically thresholds of the predicted probability of defectiveness returned by a logistic regression model). However, in practice, software quality control processes choose from a range of time-and-cost vs quality tradeoffs: how many files shall we test? how many shall we inspect? Thus, we argue that measures based on a variety of tradeoffs, viz., 5%, 10% or 20% of files tested/inspected would be more suitable. We study cross-project defect prediction from this perspective. We find that cross-project prediction performance is no worse than within-project performance, and substantially better than random prediction!

To help developers better allocate testing and debugging efforts, many software defect prediction techniques have been proposed in the literature. These techniques can be used to predict classes that are more likely to be buggy based on past history of buggy classes. These techniques work well as long as a sufficient amount of data is available to train a prediction model. However, there is rarely enough training data for new software projects. To deal with this problem, cross-project defect prediction, which transfers a prediction model trained using data from one project to another, has been proposed and is regarded as a new challenge for defect prediction. So far, only a few cross-project defect prediction techniques have been proposed. To advance the state-of-the-art, in this work, we investigate 7 composite algorithms, which integrate multiple machine learning classifiers, to improve cross-project defect prediction. To evaluate the performance of the composite algorithms, we perform experiments on 10 open source software systems from the PROMISE repository which contain a total of 5,305 instances labeled as defective or clean. We compare the composite algorithms with CODEP Logistic, which is the latest cross-project defect prediction algorithm proposed by Panichella et al., in terms of two standard evaluation metrics: cost effectiveness and F-measure. Our experiment results show that several algorithms outperform CODEP Logistic: Max performs the best in terms of F-measure and its average F-measure outperforms that of CODEP Logistic by 36.88%. Bagging J48 performs the best in terms of cost effectiveness and its average cost effectiveness outperforms that of CODEP Logistic by 15.34%.

How far does the predictive decision impact the software project? The cost, service time, and failure analysis from a cross-project defect prediction model

Currently, the research community is addressing the problem of defect prediction with the availability of project defect data. The availability of different project data leads to extend the research on cross projects. Cross-project defect prediction has now become an accepted area of software project management. In this paper, an empirical study is carried out to investigate the predictive performance of availability within project and cross-project defect prediction models. Furthermore, different categories of cross-project data are taken for training and testing to analyse various statistical models. In this paper data models are analysed and compared using various statistical performance measures. The findings during the empirical analysis of the data models state that gradient boosting predictor outperforms in the cross-project defect prediction scenario. Results also infer that cross-project defect prediction is comparable with project defect prediction and has statistical significance.

In the traditional software defect prediction methodology, the historical record (dataset) of the same project is partitioned into training and testing data. In a practical situation where the project to be predicted is new, traditional software defect prediction cannot be employed. An alternative method is cross-project defect prediction, where the historical record of one project (source) is used to predict the defect status of another project (target). The cross-project defect prediction method solves the limitations of the historical records in the traditional software defect prediction method. However, the performance of cross-project defect prediction is relatively low because of the distribution differences between the source and target projects. Furthermore, the software defect dataset used for cross-project defect prediction is characterized by high-dimensional features, some of which are irrelevant and contribute to low performance. To resolve these two issues, this study proposes a transformation and feature selection approach to reduce the distribution difference and high-dimensional features in cross-project defect prediction. A comparative experiment was conducted on publicly available datasets from the AEEEM. Analysis of the results obtained shows that the proposed approach in conjugation with random forest as the classification model outperformed the other four state-of-the-art cross-project defect prediction methods based on the commonly used performance evaluation metric F1_score.

Cross-project defect prediction (CPDP) is a practical solution that allows software defect prediction (SDP) to be used earlier in the software lifecycle. With the CPDP technique, the software defect predictor trained by labeled data of mature projects can be applied for the prediction task of a new project. Most previous CPDP approaches ignored the semantic information in the source code, and existing semantic-feature-based SDP methods do not take into account the data distribution divergence between projects. These limitations may weaken defect prediction performance. To solve these problems, we propose a novel approach, the transfer convolutional neural network (TCNN), to mine the transferable semantic (deep-learning (DL)-generated) features for CPDP tasks. Specifically, our approach first parses the source file into integer vectors as the network inputs. Next, to obtain the TCNN model, a matching layer is added into convolutional neural network where the hidden representations of the source and target project-specific data are embedded into a reproducing kernel Hilbert space for distribution matching. By simultaneously minimizing classification error and distribution divergence between projects, the constructed TCNN could extract the transferable DL-generated features. Finally, without losing the information contained in handcrafted features, we combine them with transferable DL-generated features to form the joint features for CPDP performing. Experiments based on 10 benchmark projects (with 90 pairs of CPDP tasks) showed that the proposed TCNN method is superior to the reference methods.

Although a long-running project has experienced many releases, removing defects from a product is still a challenge. Cross-version defect prediction (CVDP) regards project data of prior releases as a useful source for predicting fault-prone modules based on defect prediction techniques. Recent studies have explored cross-project defect prediction (CPDP) that uses the project data from outside a project for defect prediction. While CPDP techniques and CPDP data can be diverted to CVDP, its effectiveness has not been investigated. To investigate whether CPDP approaches and CPDP data are useful for CVDP. The investigation also compared the usage of prior release data. We chose a style of replication of a previous comparative study on CPDP approaches. Some CPDP approaches could improve the performance of CVDP. The use of the latest prior release was the best choice. If one has no CVDP data, the use of CPDP data for CVDP was found to be effective. 1) Some CPDP approaches could improve CVDP, 2), if one can access project data from the latest release, project data from older releases would not bring clear benefit, and 3) even if one has no CVDP data, appropriate CPDP approaches would be able to deliver quality prediction with CPDP data.

Cross-project defect prediction (CPDP) has recently become very popular in the field of software defect prediction. It was generally treated as a binary classification problem or a regression problem in most of previous studies. However, these existing CPDP methods may be not suitable for those software projects that have limited manpower and budget. To address the issue of priority estimation for buggy software entities, in this paper CPDP is formulated as a ranking problem. Inspired by the idea of the pointwise approach to learning to rank, we propose a ranking-oriented CPDP approach called ROCPDP. A case study conducted on the datasets collected from AEEEM and PROMISE shows that ROCPDP outperforms the eight baseline methods in two CPDP scenarios, namely one-to-one and many-to-one. Besides, in the many-to-one scenario ROCPDP is, by and large, comparable to the best baseline method performed in a specific within-project defect prediction scenario.

Software defect prediction is a vital area in software engineering that helps developers detect potential faults before software is deployed. Cross-Project Defect Prediction (CPDP) is particularly valuable, as it enables the use of defect data from one project to predict errors in another, making it beneficial in cases where project-specific defect data is insufficient. However, the effectiveness of CPDP largely depends on how well the machine learning models are trained, and a key factor influencing their performance is the size of the feature set used. This study focuses on evaluating the impact of feature set size on the performance of two widely used machine learning models, Random Forest (RF) and Support Vector Machine (SVM), in the context of CPDP. We used defect datasets from the AEEEM repository, which consists of multiple real-world software projects. An outlier detection technique was applied to select the number of features in the training and testing data, ensuring a systematic analysis of their impact on model performance. The F1-score was used as the primary evaluation metric, as it provides a balance between precision and recall, making it a reliable measure of defect prediction accuracy. Our findings suggest that the size of the feature set plays a crucial role in determining the effectiveness of both RF and SVM models. Too many features introduce noise, reducing predictive accuracy, while too few cause underfitting, leading to the missed detection of defect patterns. Identifying an optimal feature set size improves model performance, providing practical insights for enhancing CPDP. Optimizing feature selection can lead to more accurate predictions, thereby aiding software maintenance and enhancing overall software quality.

Cross-project defect prediction (CPDP) aims to identify defective software modules in a target project by using historical defect data from other source projects. Recently, CPDP has attracted much more research interest. However, existing CPDP models are parametric methods, which usually require intensive parameter selection and tuning to achieve better prediction performance. Moreover, software metrics (features) usually have strong correlation among themselves and this is detrimental to build prediction models. However, most CPDP methods don't consider to reduce correlated metrics, which may bring negative effect on the performance of CPDP. In the article, we proposed a new Correlation Metric Selection based Correlation Alignment (CMSCA) approach for CPDP to address the above concerns. Specifically, CMSCA is a non-parametric algorithm, which can perform knowledge transfer across projects without the need for parameter selection and tuning. It is simple but effective. Through an empirical investigation of 5 publicly-available defect datasets, experimental results demonstrate that the proposed CMSCA model outperforms or has comparable to the related CPDP models.

In Software Testing, there are typically two ways to predict defects in the software—within-project defect prediction (WPDP) and cross project defect prediction (CPDP). In this research, we are using a hybrid model for cross project defect prediction. It is a two-phase model consisting of ensemble learning (EL) and genetic algorithm (GA) phase. For our research, we used datasets from the PROMISE repository and created clusters after normalization using k-means clustering algorithm. This further helped us improve the accuracy of the model. Our dataset consists of 22 attributes and were labeled defective or not. Our results show that our hybrid model after implementing k-means clustering achieved an F1 score of 0.666. CPDP is a newer and faster approach for software defect prediction but is often error prone. This method can change the software industry as it will lead to improved software development and faster software delivery.KeywordsCross project defect predictionk-means clusteringEnsemble learningGenetic algorithmCompositional modelEnsemble model

Using classification methods to predict software defect is receiving a great deal of attention and most of the existing studies primarily conduct prediction under the within‐project setting. However, there usually had no or very limited labelled data to train an effective prediction model at an early phase of the software lifecycle. Thus, cross‐project defect prediction (CPDP) is proposed as an alternative solution, which is learning a defect predictor for a target project by using labelled data from a source project. Differing from previous CPDP methods that mainly apply instances selection and classifiers adjustment to improve the performance, in this study, the authors put forward a novel distribution–adaptation‐based CPDP approach, joint distribution matching (JDM). Specifically, JDM aims to minimise the joint distribution divergence between the source and target project to improve the CPDP performance. By constructing an adaptive weight vector for the instances of the source project, JDM can be effective and robust at reducing marginal distribution discrepancy and conditional distribution discrepancy simultaneously. Extensive experiments verify that JDM can outperform related distribution–adaptation‐based methods on 15 open‐source projects that are derived from two types of repositories.

Cross-project defect prediction (CPDP) aims to predict software defects in a target project domain by leveraging information from different source project domains, allowing testers to identify defective modules quickly. However, CPDP models often underperform due to different data distributions between source and target domains, class imbalances, and the presence of noisy and irrelevant instances in both source and target projects. Additionally, standard features often fail to capture sufficient semantic and contextual information from the source project, leading to poor prediction performance in the target project. To address these challenges, this research proposes Smote Correlation and Attention Gated recurrent unit based Long Short-Term Memory optimization (SCAG-LSTM), which first employs a novel hybrid technique that extends the synthetic minority over-sampling technique (SMOTE) with edited nearest neighbors (ENN) to rebalance class distributions and mitigate the issues caused by noisy and irrelevant instances in both source and target domains. Furthermore, correlation-based feature selection (CFS) with best-first search (BFS) is utilized to identify and select the most important features, aiming to reduce the differences in data distribution among projects. Additionally, SCAG-LSTM integrates bidirectional gated recurrent unit (Bi-GRU) and bidirectional long short-term memory (Bi-LSTM) networks to enhance the effectiveness of the long short-term memory (LSTM) model. These components efficiently capture semantic and contextual information as well as dependencies within the data, leading to more accurate predictions. Moreover, an attention mechanism is incorporated into the model to focus on key features, further improving prediction performance. Experiments are conducted on apache_lucene, equinox, eclipse_jdt_core, eclipse_pde_ui, and mylyn (AEEEM) and predictor models in software engineering (PROMISE) datasets and compared with active learning-based method (ALTRA), multi-source-based cross-project defect prediction method (MSCPDP), the two-phase feature importance amplification method (TFIA) on AEEEM and the two-phase transfer learning method (TPTL), domain adaptive kernel twin support vector machines method (DA-KTSVMO), and generative adversarial long-short term memory neural networks method (GB-CPDP) on PROMISE datasets. The results demonstrate that the proposed SCAG-LSTM model enhances the baseline models by 33.03%, 29.15% and 1.48% in terms of F1-measure and by 16.32%, 34.41% and 3.59% in terms of Area Under the Curve (AUC) on the AEEEM dataset, while on the PROMISE dataset it enhances the baseline models’ F1-measure by 42.60%, 32.00% and 25.10% and AUC by 34.90%, 27.80% and 12.96%. These findings suggest that the proposed model exhibits strong predictive performance.

Cross-Project Defect Prediction (CPDP) as a means to focus quality assurance of software projects was under heavy investigation in recent years. However, within the current state-of-the-art it is unclear which of the many proposals performs best due to a lack of replication of results and diverse experiment setups that utilize different performance metrics and are based on different underlying data. Within this article [2, 3], we provide a benchmark for CPDP. Our benchmark replicates 24 CPDP approaches proposed by researchers between 2008 and 2015. Through our benchmark, we answer the following research questions: • RQ1: Which CPDP approaches perform best in terms of F-measure, G-measure, AUC, and MCC? • RQ2: Does any CPDP approach consistently fulfill the performance criteria for successful predictions postulated by Zimmermann et al. [4], i.e., have at least 0.75 recall, 0.75 precision, and 0.75 accuracy? • RQ3: What is the impact of using only larger products (> 100 instances) with a certain balance (at least 5% defective instances and at least 5% non-defective instances) on the benchmark results? • RQ4: What is the impact of using a relatively small subset of a larger data set on the benchmark results? We identified 5 public data sets, which contain defect data about 86 software products that we used to answer these research question. The advantage of using multiple data sets was that we could increase the number of software products and, thereby, increase the external validity of our results. Moreover, we wanted to use multiple performance criteria for the evaluation of the CPDP approaches. Therefore, RQ1 ranks approaches not just using a single criterion, but using the four performance metrics AUC, F-measure, G-measure, and MCC. Existing approaches for the ranking of statistically different approaches neither account for software products from different data sets, nor multiple performance metrics. Therefore, we defined a new approach for the combination of separate rankings for the performance criteria and data sets, into one common ranking. Figure 1 depicts the results for RQ1. The results show that an approach proposed by Camargo Cruz and Ochimizu [1] performs best and even outperforms cross-validation. Moreover, our results show that only 6 of the 24 approaches outperform one of our baselines, i.e., using all data for training without any transfer learning. Regarding RQ2, we determined that predictions only seldomly achieve a high performance of 0.75 recall, precision, and accuracy. The best CPDP approaches only fulfill the criterion for 4 of the 86 products, i.e., 4.6% of the time. Thus, CPDP still has not reached a point where the performance of the results is sufficient for the application in practice. RQ3 and RQ4 were used to see if results are affected by subsetting data, as is often done for defect prediction experiments. For RQ3, i.e., using a large subset, we determined no difference between using all data and using the subset. For RQ4, i.e., using a small subset of of data, we found that there are statistically signifcant differences in reported performances of up to 5%. Thus, the use of small subsets should be avoided.