Does Deep Learning improve the performance of duplicate bug report detection? An empirical study

Duplicate bug report detection is one of the major problems in software triage systems like Bugzilla to deal with end user requests. User request contains some categorical and especially textual fields which need feature extraction for duplicate detection. Contextual and topical features are acquired using calculating cosine similarity between term frequency or inverse document frequency or BM25F technique from a pair of bug reports against some topics. This research proposes the individual Manhattan distance similarity approach instead of cosine distance similarity for every topic in contextual features to expand the feature dimension which can increase the accuracy of the duplicate bug report detection process. The four famous datasets of bug reports have used for evaluation of the proposed method including Android, Eclipse, Mozilla, and Open Office which the experimental results indicate performance improvement for four contextual features including general, cryptography, network, and Java topics.

Bug reporting is essentially an uncoordinated process. The same bugs could be repeatedly reported because users or testers are unaware of previously reported bugs. As a result, extra time could be spent on bug triaging and fixing. In order to reduce redundant effort, it is important to provide bug reporters with the ability to search for previously reported bugs. The search functions provided by the existing bug tracking systems are using relatively simple ranking functions, which often produce unsatisfactory results. In this paper, we adopt Ranking SVM, a Learning to Rank technique to construct a ranking model for effective bug report search. We also propose to use the knowledge of Wikipedia to discover the semantic relations among words and documents. Given a user query, the constructed ranking model can search for relevant bug reports in a bug tracking system. Unlike related works on duplicate bug report detection, our approach retrieves existing bug reports based on short user queries, before the complete bug report is submitted. We perform evaluations on more than 16,340 Eclipse and Mozilla bug reports. The evaluation results show that the proposed approach can achieve better search results than the existing search functions provided by Bugzilla and Lucene. We believe our work can help users and testers locate potential relevant bug reports more precisely.

Bug deduplication is a hot topic in software engineering information retrieval research, but it is often not deployed. Typically to de-duplicate bug reports developers rely upon the search capabilities of the bug report software they employ, such as Bugzilla, Jira, or Github Issues. These search capabilities range from simple SQL string search to IR-based word indexing methods employed by search engines. Yet too often these searches do very little to stop the creation of duplicate bug reports. Some bug trackers have more than 10\% of their bug reports marked as duplicate. Perhaps these bug tracker search engines are not enough? In this paper we propose a method of attempting to prevent duplicate bug reports before they start: continuous querying. That is as the bug reporter types in their bug report their text is used to query the bug database to find duplicate or related bug reports. This continuous querying allows the reporter to be alerted to duplicate bug reports as they report the bug, rather than formulating queries to find the duplicate bug report. Thus this work ushers in a new way of evaluating bug report deduplication techniques, as well as a new kind of bug deduplication task. We show that simple IR measures show some promise for addressing this problem but also that further research is needed to refine this novel process that is integrate-able into modern bug report systems.

Free/Open Source Software (FOSS) communities often use open bug reporting to allow users to participate by reporting bugs. This practice can lead to more duplicate reports, as users can be less rigorous about researching existing bug reports. This paper examines how FOSS projects deal with duplicate bug reports. We examined 12 FOSS projects: 4 small, 4 medium and 4 large, where size was determined by number of code contributors. First, we found that contrary to what has been reported from studies of individual large projects like Mozilla and Eclipse, duplicate bug reports are a problem for FOSS projects, especially medium-sized, which struggle with a large number of submissions without the resources of large projects. Second, we found that the focus of a project does not affect the number of duplicate bug reports. Our findings indicate a need for additional scaffolding and training for bug reporters.

Bug handling is an essential part of the software development process. Ideally, in a bug‐tracking system, bugs are reported, fixed, verified, and closed. In some cases, bugs have to be reopened mostly due to an incorrect fix. However, instead of reopening the existing bug report, users may submit a new report for a previously reported bug, which results in duplicate bug reports. Users might report duplicate bugs if they miss the previous bug report or if the previous bug is unresolved (i.e., in progress) and therefore cannot be reopened. These duplicate bug reports may cost extra maintenance efforts in triaging and bug fixing. There have been several studies on characterizing reopened bugs and duplicate bug reports; however, to the best of our knowledge, there has been no prior work on understanding the dynamics of their interaction, which ismissed reopenbugs. Our study is based on analyzing the difference between duplicate and nonduplicate bugs and further categorizing the duplicate bugs. In this regard, we categorize duplicate bugs according to the original bug's resolution status at the duplicate's creation time as Master‐Unresolved bugs and Master‐Resolved (Missed Reopen) bugs to distinguish their properties. We compare these two types of bugs in terms of their relationship to their master bug, bug surface time, bug fix time, bug's severity, and the number of users involved. We perform case studies using the Eclipse and Mozilla projects' bug repositories that include more than 165,500 and 394,000 bug reports, respectively. Additionally, we investigate the perceived importance, impact, and causes of duplicate bugs, as well as the difference between nonduplicate and duplicate bugs and its categories for practitioners in the software industry by conducting a survey.

Bug reporting is an uncoordinated process that is often the cause of redundant workload in triaging and fixing bugs due to many duplicated bug reports. Furthermore, quite often, same bugs are repeatedly reported as users or testers are unaware of whether they have been reported from the search query results. In order to reduce both the users and developers' efforts, the quality of search in a bug tracking system is crucial. However, all existing search functions in a bug tracking system produce results with undesired relevance and ranking. Hence, it is essential to provide an effective search function to any bug tracking system.Learning to rank (LTR) is a supervised machine learning technique that is used to construct a ranking model from training data. We propose a novel approach by using LTR to search for potentially related bug reports in a bug tracking system. Our method uses a set of proposed features of bug reports and queries. A preliminary evaluation shows that our approach can enhance the quality of searching for similar bug reports, therefore, relieving the burden of developers in dealing with duplicate bug reports.

For a large and complex software system, the project team could receive a large number of bug reports. Some bug reports could be duplicates as they essentially report the same problem. It is often tedious and costly to manually check if a newly reported bug is a duplicate of an already reported bug. In this paper, we propose BugSim, a method that can automatically retrieve duplicate bug reports given a new bug report. BugSim is based on learning to rank concepts. We identify textual and statistical features of bug reports and propose a similarity function for bug reports based on the features. We then construct a training set by assembling pairs of duplicate and non-duplicate bug reports. We train the weights of features by applying the stochastic gradient descent algorithm over the training set. For a new bug report, we retrieve candidate duplicate reports using the trained model. We evaluate BugSim using more than 45,100 real bug reports of twelve Eclipse projects. The evaluation results show that the proposed method is effective. On average, the recall rate for the top 10 retrieved reports is 76.11%. Furthermore, BugSim outperforms the previous state-of-art methods that are implemented using SVM and BM25Fext.

The traditional duplicate bug reports detection approaches are usually based on vector space model. However, the experimental result is rarely satisfying since this method cannot distinguish semantic correlation among bug reports which written by natural languages. Topic model, as a method to model underlying topics of texts, can solve the problem of document similarity calculation methods used in the information retrieving. It can find the semantic topics among the texts through massive training data, and obtain semantic relatedness among documents. Therefore, this paper proposes a novel duplication detection method based on topic model. Through selecting bug reports with execution information and combing with classified information of bugs, not only does this new method overcome the problem of high dimension, sparse data and loud noise, but also avoid the problem of synonymy and ambiguity in the natural languages. Comparing to the traditional SVM method, the recall rate and precision rate of our proposed approach have obviously increased, which indicates the effectiveness of this new method.

In software maintenance, bug reports play an important role in the correctness of software packages. Unfortunately, the duplicatebug report problem arises because there are too many duplicate bug reports in various software projects. Handling with duplicate bug reports is thus time-consuming and has high cost of software maintenance. Therefore, this research introduces a detection scheme based on the extended class centroid information (ECCI) to enhance thedetection performance. This method is extended from the previous one, which used only centroid method without considering the effects of both inner and inter class. Besides, this method also improved the previous use of normalized cosine in identifying the similarity between two bug reports by denormalized cosine. The effectiveness of ECCI is proved through the empirical study with three open-source projects: SVN, Argo UML and Apache. The experimental results show thatECCI outperforms other detection schemes by about 10% in all cases.

In order to accelerate the imaging process of MRI (Magnetic Resonance Imaging), CS-MRI (Compressed Sensing Magnetic Resonance Imaging) utilizes the prior information of MR images and reconstructs under-sampled MR images by solving an optimization problem through an iterative algorithm, and it is time consuming. With the emergence of deep learning technology, more and more optimization problems can be solved through well-designed networks and large amounts of training data. Therefore, DL-Based (Deep Learning-Based) methods have attracted much attention. However, most of the current work simply treats MRI reconstruction as an image-to-image task, so there is still a lot of room for improvement in how to better integrate MRI information. This paper proposes a new DL-based method, which reconstructs the MR images in both k-space and image domain using several sub-networks and fuses information among different sub-networks. We also used down-sampling and up-sampling pairs to reduce the computational complexity. Our proposed architecture obtained promising experimental results on Calgary-Campinas Brain MRI datasets.

The automated task of locating the potential buggy files in a software project given a bug report is called bug localization. Bug localization helps developers focus on crucial files. However, the existing automated bug localization approaches face a key challenge, called lexical mismatch. Specifically, the terms used in bug reports to describe a bug are different from the terms and code tokens used in source files. To address that, we present a novel approach that uses deep neural network (DNN) in combination with rVSM, an information retrieval (IR) technique. rVSM collects the feature on the textual similarity between bug reports and source files. DNN is used to learn to relate the terms in bug reports to potentially different code tokens and terms in source files. Our empirical evaluation on real-world bug reports in the open-source projects shows that DNN and IR complement well to each other to achieve higher bug localization accuracy than individual models. Importantly, our new model, DNNLOC, with a combination of the features built from DNN, rVSM, and project's bug-fixing history, achieves higher accuracy than the state-of-the-art IR and machine learning techniques. In half of the cases, it iscorrect with just a single suggested file. In 66% of the time, acorrect buggy file is in the list of three suggested files. With 5 suggested files, it is correct in almost 70% of the cases.

In this data package, you will find two folders: `artifacts` and `outputs` `artifacts` contains the videos we collected in our user study, the model files for the different models we evaluated, and the detailed results that we generated (See `Detailed Results` section for more information). The `videos` folder is broken down by user, where each user has a folder contain the apps they were given to create videos for. Each of the apps contain folders that denote the bug they generated a report for. Finally, instead these bug folders there is the actual video-based bug report as an mp4 file. The `user_assignment.csv` just contains the finalized assignments of users to corresponding bug reports. In the `models` folder, you will find the two models we evaluated (SIFT, SimCLR, and OCR+IR). In each folder you will find the corresponding trained codebook files that we generated for SIFT and SimCLR. These codebook files are pickle files that contain the binary representation of the trained codebooks. Additionally, in the SimCLR folder, you will find a checkpoint and pytorch model file that contains all the necessary information for reloading our trained SimCLR model. For the `OCR+IR` folder, you will find all of the code for the OCR+IR model as well as the intermediate output for this particular model, other models' outputs are stored in the `outputs` folder. The `outputs` folder contains all of the intermediate outputs of our code, except for OCR+IR. In the `results` folder, you will find all of the raw rankings and metrics for the SIFT and SimCLR model for all combinations of video-based bug reports per app. <strong>NOTE: SIFT is missing the 10k raw ranking and metrics, but will be provided in a future version. </strong>`evaluation_setting` contains a json file that contains all of the duplicate detection tasks we used for evaluating our models, i.e. `setting 2` (See paper for more details). `user_rankings_weighted_all` and `user_results_weighted_all` contain converted version of the raw rankings and metrics for the SIFT and SimCLR model to match `setting 2`. `extracted_text` contains the output of running the OCR model, i.e. the frames of the videos and the text from each frame. Lastly, `combined` contains the results of the combined tango approach. You can find a spreadsheet containing the results for all of the different configurations we tested at `tango_reproduction_package/artifacts/detailed_results.xlsx`. In this excel file, we have multiple sheets. `overall` shows the performance of the different model configurations averaged across all apps. `overall_comb` shows the combined performance of the visual and textual model configurations averaged across all apps. Additionally, `per-app` and `per-app-comb` has the performance of the single and combined model configurations per app, respectively. Lastly, we provide the overall performance in sheet `overall_user_study` and `overall_user_study_comb` of the single and combined model configurations on the settings (used only APOD app) given to the users for evaluating how much time and effort tango can save developers. All sheets show the performance in terms of mRR (`avg_rr`), the standard deviation of recipical rank, median (`med_rr`), and quartile 1 and 3 (`q#_rr`). The same is true for mAP (`avg_ap`). We also show the performance in terms of average rank including standard deviation, and quartiles. Lastly, we providing HIT@1-5, 7, and 10 (`h#`). Sheets that contain the `weight` column have information regarding how much weight is given to the visual and textual information. A value of `0.1` means that the textual information received a weight of `0.1` while the visual information was given a weight of `0.9`. For values containing two numbers, e.g. `0.1-0.0`, refers to the weighting scheme introduced in the paper for when there may be high overlap in vocabulary between duplicate and non-duplicates (See paper for more details). If an app does not have high overlap, then a weight of `0.1` is used for the textual information, else the textual information is not considered, i.e., weight of `0.0`. To use our data to reproduce our results please visit our github repository: https://github.com/ncoop57/tango

automatic duplicate bug reports detection is a famous problem in mining software repositories since 2004 for software triage systems e.g. Bugzilla. Textual features are the most important type of features in similarity and duplicate detection e.g. BM25F which indicate the common term frequency in two reports. Sometimes a common sequence can show more similarity in two texts, thus new features based on longest common sequence (LCS) of two bug reports proposed in this paper as new textual features for text similarity detection. Android, Eclipse, Mozilla, and Open Office dataset are used for evaluation of proposed features and the experimental results show LCS-based features are important and the accuracy, precision and recall of classifier prediction models improved 4.5, 2.5 and 2.5 percent respectively on average after using LCS and get up to 96, 98 and 97 percent respectively on average using different classifiers.

Many software comprehension tasks depend on how stakeholders textually describe their problems. These textual descriptions are leveraged by Text Retrieval (TR)-based solutions to more than 20 software engineering tasks, such as duplicate issue detection. The common assumption of such methods is that text describing the same issue in multiple places will have a common vocabulary. This paper presents an empirical study aimed at verifying this assumption and discusses the impact of the common vocabulary on duplicate issue detection. The study investigated 13K+ pairs of duplicate bug reports and Stack Overflow (SO) questions. We found that on average, more than 12.2% of the duplicate pairs do not have common terms. The other duplicate issue descriptions share, on average, 30% of their vocabulary. The good news is that these duplicates have significantly more terms in common than the non-duplicates. We also found that the difference between the lexical agreement of duplicate and non-duplicate pairs is a good predictor for the performance of TR-based duplicate detection.

Most bug assignment approaches utilize text classification and information retrieval techniques.These approaches use the textual contents of bug reports to build recommendation models.The textual contents of bug reports are usually of high dimension and noisy source of information.These approaches suffer from low accuracy and high computational needs.In this paper, we investigate whether using categorical fields of bug reports, such as component to which the bug belongs, are appropriate to represent bug reports instead of textual description.We build a classification model by utilizing the categorical features, as a representation, for the bug report.The experimental evaluation is conducted using three projects namely NetBeans, Freedesktop, and Firefox.We compared this approach with two machine learning based bug assignment approaches.The evaluation shows that using the textual contents of bug reports is important.In addition, it shows that the categorical features can improve the classification accuracy.