Identifying security bug reports via text mining: An industrial case study

In order that the general public is not vulnerable to hackers, security bug reports need to be handled by small groups of engineers before being widely discussed. But learning how to distinguish the security bug reports from other bug reports is challenging since they may occur rarely. Data mining methods that can find such scarce targets require extensive optimization effort. The goal of this research is to aid practitioners as they struggle to optimize methods that try to distinguish between rare security bug reports and other bug reports. Our proposed method, called SWIFT, is a dual optimizer that optimizes both learner and pre-processor options. Since this is a large space of options, SWIFT uses a technique called 𝜖-dominance that learns how to avoid operations that do not significantly improve performance. When compared to recent state-of-the-art results (from FARSEC which is published in TSE’18), we find that the SWIFT’s dual optimization of both pre-processor and learner is more useful than optimizing each of them individually. For example, in a study of security bug reports from the Chromium dataset, the median recalls of FARSEC and SWIFT were 15.7% and 77.4%, respectively. For another example, in experiments with data from the Ambari project, the median recalls improved from 21.5% to 85.7% (FARSEC to SWIFT). Overall, our approach can quickly optimize models that achieve better recalls than the prior state-of-the-art. These increases in recall are associated with moderate increases in false positive rates (from 8% to 24%, median). For future work, these results suggest that dual optimization is both practical and useful.

Bug reporting is essentially an uncoordinated process. The same bugs could be repeatedly reported because users or testers are unaware of previously reported bugs. As a result, extra time could be spent on bug triaging and fixing. In order to reduce redundant effort, it is important to provide bug reporters with the ability to search for previously reported bugs. The search functions provided by the existing bug tracking systems are using relatively simple ranking functions, which often produce unsatisfactory results. In this paper, we adopt Ranking SVM, a Learning to Rank technique to construct a ranking model for effective bug report search. We also propose to use the knowledge of Wikipedia to discover the semantic relations among words and documents. Given a user query, the constructed ranking model can search for relevant bug reports in a bug tracking system. Unlike related works on duplicate bug report detection, our approach retrieves existing bug reports based on short user queries, before the complete bug report is submitted. We perform evaluations on more than 16,340 Eclipse and Mozilla bug reports. The evaluation results show that the proposed approach can achieve better search results than the existing search functions provided by Bugzilla and Lucene. We believe our work can help users and testers locate potential relevant bug reports more precisely.

Bug reporting is an uncoordinated process that is often the cause of redundant workload in triaging and fixing bugs due to many duplicated bug reports. Furthermore, quite often, same bugs are repeatedly reported as users or testers are unaware of whether they have been reported from the search query results. In order to reduce both the users and developers' efforts, the quality of search in a bug tracking system is crucial. However, all existing search functions in a bug tracking system produce results with undesired relevance and ranking. Hence, it is essential to provide an effective search function to any bug tracking system.Learning to rank (LTR) is a supervised machine learning technique that is used to construct a ranking model from training data. We propose a novel approach by using LTR to search for potentially related bug reports in a bug tracking system. Our method uses a set of proposed features of bug reports and queries. A preliminary evaluation shows that our approach can enhance the quality of searching for similar bug reports, therefore, relieving the burden of developers in dealing with duplicate bug reports.

Bug fixing is a critical activity in the software development process. In issue tracking systems such as JIRA, each bug report is assigned a priority level to indicate the urgency and importance level of the bug. The priority may change during the bug fixing process, indicating that the urgency and importance level of the bug will change with the bug fixing. However, manually evaluating priority changes for bugs is a tedious process that heavily relies on the subjective judgment of developers and project managers, leading to incorrect priority changes and thus hindering timely bug fixes. Given the lack of research on bug priority change prediction, we propose a novel two-phase bug report priority change prediction method based on bug fixing evolution features and class imbalance handling strategy. Specifically, we divided the bug lifecycle into two phases: bug reporting and bug fixing, and constructed bug priority change prediction models for each phase. To evaluate the performance of our method, we conducted experiments on a bug dataset constructed from 32 non-trivial Apache projects. The experimental results show that our proposed bug fixing evolution features and the adopted class imbalance handling strategy can effectively improve the performance of prediction models. The F1-score of the prediction model constructed for the bug reporting phase reached 0.798, while the F1-weighted and F1-macro of the prediction model constructed for the bug fixing phase were 0.712 and 0.613, respectively. Furthermore, we explored the cross-project applicability of our prediction models and their performance at different priority levels. The findings indicate large variations in model performance across different projects, although the overall scores remain decent. Meanwhile, the predictive performance across various priority levels remained relatively consistently high.

To guarantee the security of the wireless network, effort must be paid to software security as software plays a more and more important role in the development of wireless network intelligentization. Identifying security bug reports (SBRs) from bug repository matters much for reducing security risk of wireless network. Cross-project SBR prediction, which uses a prediction model trained with labeled data from one project to predict another project, has been proposed to eliminate SBRs of software products. While reviewing the previous work focused on cross-project SBR prediction, we find the performance (e.g., Recall, F1-score) of cross-project SBR prediction is too low to the production application. This paper proposes a hybrid sampling approach based on text similarity and uncertainty-sampling. We conduct experiments on ten publicly available datasets. The results show our approach could significantly improve the performance of cross-project SBR prediction. On average, the performance of the classification model can be improved by 34%, 64%, 48%, and 11% in terms of Recall, Precision, F1-score, and AUC, respectively.

For a large and complex software system, the project team could receive a large number of bug reports. Some bug reports could be duplicates as they essentially report the same problem. It is often tedious and costly to manually check if a newly reported bug is a duplicate of an already reported bug. In this paper, we propose BugSim, a method that can automatically retrieve duplicate bug reports given a new bug report. BugSim is based on learning to rank concepts. We identify textual and statistical features of bug reports and propose a similarity function for bug reports based on the features. We then construct a training set by assembling pairs of duplicate and non-duplicate bug reports. We train the weights of features by applying the stochastic gradient descent algorithm over the training set. For a new bug report, we retrieve candidate duplicate reports using the trained model. We evaluate BugSim using more than 45,100 real bug reports of twelve Eclipse projects. The evaluation results show that the proposed method is effective. On average, the recall rate for the top 10 retrieved reports is 76.11%. Furthermore, BugSim outperforms the previous state-of-art methods that are implemented using SVM and BM25Fext.

Bug tracking systems manage bug reports for assuring the quality of software products. A bug report also referred as trouble, problem, ticket or defect contains several features for problem management and resolution purposes. Severity and priority are two essential features of a bug report that define the effect level and fixing order of the bug. Determining these features is challenging and depends heavily on human being, e.g., software developers or system operators, especially for assessing a large number of error and warning events occurring on software products or network services. This study proposes an approach of using random forest for assessing severity and priority for software bug reports automatically. This approach aims at constructing multiple decision trees based on the subsets of the existing bug dataset and features, and then selecting the best decision trees to assess the severity and priority of new bugs. The approach can be applied for detecting and forecasting faults in large, complex communication networks and distributed systems today. We have presented the applicability of random forest for bug report analysis and performed several experiments on software bug datasets obtained from open source bug tracking systems. Random forest yields an average accuracy score of 0.75 that can be sufficient for assisting system operators in determining these features. We have provided some analysis of the experimental results.

A large number of bug reports are submitted by users and developers in bug-tracking system every day. It is time-consuming for software maintainers to assign bug reports to appropriate developers for fixing manually. Many bug-triaging methods have been developed to automate this process. However, most previous studies mainly focused on analyzing textual content and failed to make full use of the structured information embedded in the bug-tracking system. In fact, this structured information, which plays an important role in bug triaging, reflects the process of bug tracking and the historical activities. To further improve the performance of automatic bug triaging, in this study, we propose a new representation learning model, PTITransE, for knowledge bases, which extends TransE via enhancing the embeddings with textual entity descriptions and is more suitable for bug triaging. Moreover, we make the first attempt to apply knowledge base and link prediction techniques to bug triaging. For each new bug report, the proposed framework can recommend top-k developers for fixing the bug report by using the learned embeddings of entities and relations. Evaluation is performed on three real-world projects, and the results indicate that our method outperforms baseline bug triaging approaches and can alleviate the cold-start problem in bug triaging.

It is important that information provided in bug reports is relevant and complete in order to help resolve bugs quickly. However, often such information trickles to developers after several iterations of communication between Developers and reporters. Poorly designed bug tracking systems are partly to blame for this exchange of information being stretched over time. Our paper addresses the concerns of bug tracking systems by proposing four broad directions for enhancements. As a proof-of-concept, we also demonstrate a prototype interactive bug tracking system that gathers relevant information from the user and identifies files that need to be fixed to resolve the bug.

For large open source projects (e.g., Eclipse, Mozilla), developers usually utilize bug reports to facilitate software maintenance tasks such as fixer assignment. However, there are a large portion of short reports in bug repositories. We find that 78.1% of bug reports only include less than 100 words in Eclipse and require bug fixers to spend more time on resolving them due to limited informative contents. To address this problem, in this paper, we propose a novel approach to enrich bug reports. Concretely, we design a sentence ranking algorithm based on a new textual similarity metric to select the proper contents for bug report enrichment. For the enriched bug reports, we conduct a user study to assess whether the additional sentences can provide further help to fixer assignment. Moreover, we assess whether the enriched versions can improve the performance of automated fixer recommendation. In particular, we perform three popular automated fixer recommendation approaches on the enriched bug reports of Eclipse, Mozilla, and GNU Compiler Collection (GCC). The experimental results show that enriched bug reports improve the average F-measure scores of the automated fixer recommendation approaches by up to 10% for DREX, 13.37% for DRETOM, and 8% for DevRec when top-10 bug fixers are recommended.

Users can use bug tracking systems (BTS) such as Bugzilla, JIRA, and GitHub to report defects they detect while using or testing the software applications. Bugzilla has specified the bug's information such as bug ID, description, product, classification, component, platform, operating system, bug status, resolution, priority, and severity. The severity assigned to the bug report is conducted manually. So, it takes some considerable time to assign the severity manually, and also it is a tedious task. In this research, we used a deep learning-based technique to develop the prediction model for severity. Our proposed approach is based on the convolutional neural network (CNN) for severity prediction of the bug reports. First, we pre-process the textual content in bug reports using natural language processing (NLP) approaches. Then we extract the features from the textual context (short description) using the Bag-of-Words feature extraction method. Finally, we train a CNN-based classifier to severity prediction based on its input. Then our result is compared with the Support Vector Machine (SVM) and Temporal Convolutional Network (TCN) to find a better model for severity prediction. The final results show that the proposed approach based on the CNN classifier performs better than the other approaches, and it shows an 81% accuracy while others have low accuracy, like 61% and 48% for SVM and TCN, respectively. The proposed model's performance was evaluated using the Bugzilla dataset, which included over 25,000 bug reports. The accuracy, precision, recall, and F1-Score are used to measure the performance.

The accurate severity classification of a bug report is an important aspect of bug fixing. The bug reports are submitted into the bug tracking system with high speed, and owing to this, bug repository size has been increasing at an enormous rate. This increased bug repository size introduces biases in the bug triage process. Therefore, it is necessary to classify the severity of a bug report to balance the bug triaging process. Previously, many machine learning models were proposed for automation of bug severity classification. The accuracy of these models is not up to the mark because they do not extract the important feature patterns for learning the classifier. This paper proposes a novel deep learning model for multiclass severity classification called Bug Severity classification to address these challenges by using a Convolutional Neural Network and Random forest with Boosting (BCR). This model directly learns the latent and highly representative features. Initially, the natural language techniques preprocess the bug report text, and then n-gram is used to extract the features. Further, the Convolutional Neural Network extracts the important feature patterns of respective severity classes. Lastly, the random forest with boosting classifies the multiple bug severity classes. The average accuracy of the proposed model is 96.34% on multiclass severity of five open source projects. The average F-measures of the proposed BCR and the existing approach were 96.43% and 84.24%, respectively, on binary class severity classification. The results prove that the proposed BCR approach enhances the performance of bug severity classification over the state-of-the-art techniques.

Bug handling is an essential part of the software development process. Ideally, in a bug‐tracking system, bugs are reported, fixed, verified, and closed. In some cases, bugs have to be reopened mostly due to an incorrect fix. However, instead of reopening the existing bug report, users may submit a new report for a previously reported bug, which results in duplicate bug reports. Users might report duplicate bugs if they miss the previous bug report or if the previous bug is unresolved (i.e., in progress) and therefore cannot be reopened. These duplicate bug reports may cost extra maintenance efforts in triaging and bug fixing. There have been several studies on characterizing reopened bugs and duplicate bug reports; however, to the best of our knowledge, there has been no prior work on understanding the dynamics of their interaction, which ismissed reopenbugs. Our study is based on analyzing the difference between duplicate and nonduplicate bugs and further categorizing the duplicate bugs. In this regard, we categorize duplicate bugs according to the original bug's resolution status at the duplicate's creation time as Master‐Unresolved bugs and Master‐Resolved (Missed Reopen) bugs to distinguish their properties. We compare these two types of bugs in terms of their relationship to their master bug, bug surface time, bug fix time, bug's severity, and the number of users involved. We perform case studies using the Eclipse and Mozilla projects' bug repositories that include more than 165,500 and 394,000 bug reports, respectively. Additionally, we investigate the perceived importance, impact, and causes of duplicate bugs, as well as the difference between nonduplicate and duplicate bugs and its categories for practitioners in the software industry by conducting a survey.

This chapter introduces information retrieval (IR)-based fault localization techniques. One of the challenges of using bug tracking systems is that they often contain too many bug reports for developers to handle. Due to the textual nature of bug reports, IR techniques are often employed to solve the problem, and many IR-based fault localization techniques have been proposed in literature. A common fault localization process consists of four steps, that is, corpus creation, indexing, query construction, and retrieval and ranking. Top- k prediction accuracy, mean reciprocal rank, and mean average precision are widely used to evaluate the performance of IR-based fault localization techniques. The chapter aims to categorize studies on IR-based fault localization into three groups: text of current bug report only, text and history, and text and stack/execution traces. Existing IR-based fault localization techniques can be categorized based on data used in the fault localization process or IR techniques used.

To manage bugs that appear in a software, developers often make use of a bug tracking system such as Bugzilla. Users can report bugs that they encounter in such a system. Whenever a user reports a new bug report, developers need to read the summary and description of the bug report and manually locate the buggy files based on this information. This manual process is often time consuming and tedious. Thus, a number of past studies have proposed bug localization techniques to automatically recover potentially buggy files from bug reports. Unfortunately, none of these techniques are integrated to bug tracking systems and thus it hinders their adoption by practitioners. To help disseminate research in bug localization to practitioners, we develop a tool named BugLocalizer, which is implemented as a Bugzilla extension and builds upon a recently proposed bug localization technique. Our tool extracts texts from summary and description fields of a bug report and source code files. It then computes similarities of the bug report with source code files to find the buggy files. Developers can use our tool online from a Bugzilla web interface by providing a link to a git source code repository and specifying the version of the repository to be analyzed. We have released our tool publicly in GitHub, which is available at: https://github.com/smagsmu/buglocalizer. We have also provided a demo video, which can be accessed at: http://youtu.be/iWHaLNCUjBY.