Post-Silicon Bug Localization in Processors Using Instruction Footprint Recording and Analysis (IFRA)

The post-silicon validation phase in a processor's design life cycle is geared towards finding all remaining bugs in the system. It is, in fact, our last opportunity to find functional and electrical bugs in the design before shipping it to customers.In this paper, we provide a high-level overview of the methodology and technologies put into use as part of the POWER8 post-silicon functional validation phase. We describe the results and list the primary factors that contributed to this highly successful bring-up.

Software bugs remain frequent in the life cycle of software development and maintenance. Automatic localization of buggy source code files is critical for timely bug fixing and improving the efficiency of software quality assurance. Various bug localization techniques have been proposed using different dimensions of features. Recent studies have shown that different dimensions of features may play different roles in bug localization. Unfortunately, how to effectively merge these dimensions of features for improving bug localization has rarely been investigated. This article presents a Multi-Dimension Convolutional Neural Network (MD-CNN) model for bug localization automatically based on a bug report. Our approach has dual-novelty. First, we identify and extract five statistical dimensions of features. Second, we design a Convolutional Neural Network (CNN) model that takes our five statistical dimensions of features as the input and iteratively learns the complex and non-linear relationship between the features and the bug locations. The MD-CNN bug localization model is verified using six large-scale open source projects. The experimental results show that our MD-CNN outperforms the existing representative bug localization techniques in terms of the Mean Average Precision (MAP) and the number of bugs successfully localized in the top 1, 5, and 10 matched source code files.

FineLocator: A novel approach to method-level fine-grained bug localization by query expansion

Long error detection latency, the time elapsed from the occurrence of an error caused by a bug to its manifestation as an observable failure, severely limits the effectiveness of existing post-silicon validation and debug techniques. Traditional post-silicon validation tests can incur very long error detection latencies of millions or even billions of clock cycles. An earlier technique called Quick Error Detection (QED) shortens error detection latencies to only few hundred (or thousand) clock cycles. However, software-only QED (i.e., QED implemented entirely in software) can result in significantly increased post-silicon validation test runtimes. We present a new technique called Fast QED that overcomes this drawback of software-only QED, while preserving the error detection latency and bug coverage benefits of software-only QED. Simulation results using an OpenSPARC T2-like multi-core SoC and bugs abstracted from multiple commercial multi-core SoCs demonstrate: 1. Fast QED achieves 4 orders of magnitude improvement in test runtime as compared to software-only QED, with only 0.4% increase in chip area; 2. Fast QED improves error detection latencies by up to 5 orders of magnitude compared to non-QED tests, and also achieves improved error detection latencies compared to software-only QED; and, 3. Fast QED improves bug coverage by up to 2-fold compared to non-QED tests (similar to software-only QED).

To improve the software debugging efficiency, bug localization techniques have been developed to automatically locate buggy files based on bug reports. Traditional information retrieval-based bug localization cannot deal with the lexical mismatch, thus its performance is limited. In recent years, some deep learning models have been proposed to learn the semantics of bug reports and source files to bridge the lexical gap. However, their accuracy is still limited as building accurate semantic representations of bug reports and source files is very challenging. Recently, relevance matching was proposed to identify whether a document is relevant to a given query by considering both local matching and global matching. In this work, we propose a novel framework DreamLoc, which utilizes a relevance matching model to locate buggy files. Specifically, DreamLoc conducts the local matching by employing an attention-based mechanism to calculate the matching scores between bug report terms and code snippets. It also conducts the global matching by employing a gating mechanism to aggregate results of local matching and obtain the final matching score between a bug report and a source file. Since the local matching considers the relevance between each word and the global matching differentiates the importance of words, DreamLoc can effectively model the characteristics of bug reports and source files. Experimental results on five benchmark datasets show that DreamLoc outperforms five state-of-the-art models. For example, compared with DeepLoc, a recently proposed approach, the evaluation measures Accuracy@10, MAP, and MRR are improved by 6.4%, 7.4%, and 7.2%, respectively.

Software bugs are a noteworthy concern for developers and maintainers. When a failure is detected late, it costs more to be fixed. To repair the bug that caused the software failure, the location of the bug must first be known. The process of finding the defective source code elements that led to the failure of the software is called bug localization. Effective approaches for automatically locating bugs using bug reports are highly desirable, as they would reduce bug-fixing time, consequently lowering software maintenance costs. With the increasing size and complexity of software projects, manual bug localization methods have become complex, challenging, and time-consuming tasks, which motivates research on automated bug localization techniques. This paper introduces a novel bug localization model, which works on two levels. The first level localizes the buggy classes using an information retrieval approach and it has two additional sub-phases, namely the class-level feature scoring phase and the class-level final score and ranking phase, which ranks the top buggy classes. The second level localizes the buggy methods inside these classes using an information retrieval approach and it has two sub-phases, which are the method-level feature scoring phase and the method-level final score and ranking phase, which ranks the top buggy methods inside the localized classes. A model is evaluated using an AspectJ dataset, and it can correctly localize and rank more than 350 classes and more than 136 methods. The evaluation results show that the proposed model outperforms several state-of-the-art approaches in terms of the mean reciprocal rank (MRR) metrics and the mean average precision (MAP) in class-level bug localization.

Statistical debugging is a powerful technique for identifying bugs that do not violate programming rules or program invariants. Previously known statistical debugging techniques are offline bug isolation (or localization) techniques. In these techniques, the program dumps data during its execution, which is used by offline statistical analysis to discover differences in passing and failing executions. The differences identify potential bug sites. Offline techniques suffer from three limitations: (i) a large number of executions are needed to provide data, (ii) each execution must be labelled as passing or failing, and (iii) they are postmortem techniques and therefore cannot raise an alert at runtime when a bug symptom occurs. In this paper, we present an online statistical bug detection tool called Argus. Argus constructs statistics at runtime using a sliding window over the program execution, is capable of detecting bugs in a single execution and can raise an alert at runtime when bug symptoms occur. Moreover, it eliminates the requirement for labelling all executions as passing or failing. We present experimental results using the Siemens bug benchmark showing that Argus is effective in detecting 102 out of 130 bugs. We introduce optimization techniques that greatly improve Argus’ detection power and control the false alarm rate when a small number of executions are available. Argus generates more precise bug reports than the best known bug localization techniques.

The exponentially growing complexity of modern processors intensifies verification challenges. Traditional pre-silicon verification covers less and less of the design space, resulting in increasing post-silicon validation effort. A critical challenge is the manual debugging of intermittent failures on prototype chips, where multiple executions of a same test do not yield a consistent outcome. We leverage the power of machine learning to support automatic diagnosis of these difficult, inconsistent bugs. During post-silicon validation, lightweight hardware logs a compact measurement of observed signal activity over multiple executions of a same test: some may pass, somemay fail. Our novel algorithm applies anomaly detection techniques similar to those used to detect credit card fraud to identify the approximate cycle of a bug's occurrence and a set of candidate root-cause signals. Compared against other state-of-the-art solutions in this space, our new approach can locate the time of a bug's occurrence with nearly 4x better accuracy when applied to the complex OpenSPARC T2 design.

Recent findings from a user study suggest that IR-based bug localization techniques do not perform well if the bug report lacks rich structured information such as relevant program entity names. On the contrary, excessive structured information such as stack traces in the bug report might always not be helpful for the automated bug localization. In this paper, we conduct a large empirical study using 5,500 bug reports from eight subject systems and replicating three existing studies from the literature. Our findings (1) empirically demonstrate how quality dynamics of bug reports affect the performances of IR-based bug localization, and (2) suggest potential ways (e.g., query reformulations) to overcome such limitations.

To manage bugs that appear in a software, developers often make use of a bug tracking system such as Bugzilla. Users can report bugs that they encounter in such a system. Whenever a user reports a new bug report, developers need to read the summary and description of the bug report and manually locate the buggy files based on this information. This manual process is often time consuming and tedious. Thus, a number of past studies have proposed bug localization techniques to automatically recover potentially buggy files from bug reports. Unfortunately, none of these techniques are integrated to bug tracking systems and thus it hinders their adoption by practitioners. To help disseminate research in bug localization to practitioners, we develop a tool named BugLocalizer, which is implemented as a Bugzilla extension and builds upon a recently proposed bug localization technique. Our tool extracts texts from summary and description fields of a bug report and source code files. It then computes similarities of the bug report with source code files to find the buggy files. Developers can use our tool online from a Bugzilla web interface by providing a link to a git source code repository and specifying the version of the repository to be analyzed. We have released our tool publicly in GitHub, which is available at: https://github.com/smagsmu/buglocalizer. We have also provided a demo video, which can be accessed at: http://youtu.be/iWHaLNCUjBY.

Debugging the pre-Silicon Register Transfer Level (RTL) design is one of the most resource and time intensive processes in contemporary hardware design cycles. Given the scale and complexity of designs, bug localization is very valuable for debugging. We present an automatic bug localization technique in RTL. Our technique is based on identifying statistically relevant common symptoms across failing simulation traces through mining, and mapping these back to the corresponding execution paths in the RTL source code. Our localized code zones are small, focused, functionally coherent and executable. We achieve localization upto 5% and an average localization of 15% in the source code and upto 80% reduction in simulation trace size for a wide variety of bugs in all the important modules of a USB design.

Numerous efforts have been invested in improving the effectiveness of bug localization techniques, whereas little attention is paid to making these tools run more efficiently in continuously evolving software repositories. This paper first analyzes the information retrieval model behind a classic bug localization tool, BugLocator, and builds a mathematical foundation illustrating that the model can be updated incrementally when codebase or bug reports evolve. Then, we present IncBL, a tool for Incremental Bug Localization in evolving software repositories. IncBL is evaluated on the Bugzbook dataset, and the results show that IncBL can significantly reduce the running time by 77.79% on average compared with the re-computing the model, while maintaining the same level of accuracy. We also implement IncBL as a Github App that can be easily integrated into open-source projects on GitHub. Users can deploy and use IncBL locally as well. The demo video for IncBL can be viewed at https://youtu.be/G4gMuvlJSb0, and the source code can be found at https://github.com/soarsmu/IncBL.

Bug or Fault localization is a process of identifying the specific location(s) or region(s) of source code (at various granularity levels such as the directory path, file, method or statement) that is faulty and needs to be modified to repair the defect. Fault localization is a routine task in software maintenance (corrective maintenance). Due to the increasing size and complexity of current software applications, automated solutions for fault localization can significantly reduce human effort and software maintenance cost.We present a technique (which falls into the class of static techniques for bug localization) for fault localization based on a character n-gram based Information Retrieval (IR) model. We frame the problem of bug localization as a relevant document(s) search task for a given query and investigate the application of character-level n-gram based textual features derived from bug reports and source-code file attributes. We implement the proposed IR model and evaluate its performance on dataset downloaded from two popular open-source projects (JBoss and Apache).We conduct a series of experiments to validate our hypothesis and present evidences to demonstrate that the proposed approach is effective. The accuracy of the proposed approach is measured in terms of the standard and commonly used SCORE and MAP (Mean Average Precision) metrics for the task of fault localization. Experimental results reveal that the median value for the SCORE metric for JBoss and Apache dataset is 99.03% and 93.70% respectively. We observe that for 16.16% of the bug reports in the JBoss dataset and for 10.67% of the bug reports in the Apache dataset, the average precision value (computed at all recall levels) is between 0.9 and 1.0.

There are several researches, which identify the following two problems as main bottlenecks of post-silicon validation: bug detection/localization and coverage calculation. Although FPGA prototyping is considered as pre-silicon verification the coverage calculation and bug localization are as much challenging as for post-silicon phase. This article presents a novel approach to solve the above mentioned issues by embedding synthesizable assertions into prototype. The experiment has been performed on USB3.0 link layer prototype. During the experiment all injected errors have been successfully detected and functional coverage has been calculated. The main drawback of the proposed approach is unacceptable resource utilization which, was solved by adding reconfigurable regions into FPGA. Synthesis results are presented for group of assertions intended to test entry/exit functionality of low power mode which show that the FPGA resource overhead is less than two percent of super speed USB link prototype. This approach gives us ability to implement USB design and all assertions for single test in one FPGA with acceptable timing results and minor area overhead.

Information retrieval-based bug localization techniques are evaluated using datasets with an oracle. However, datasets can contain non-buggy files, which affect the reliability of these techniques. To investigate the impact of non-buggy files, we show that a test file can be regarded as a buggy file. Then, we determined if this file caused inaccuracies that would eventually affect the trustworthiness of previous techniques. We further analyzed the impact of test files on IR-based bug localization through three research questions. Our results show that the test files significantly impact the performance of the techniques. Furthermore, MAP increased by a maximum of 21%, and MRR decreased by a maximum of 13%.