Recovering system specific rules from software repositories

In the era of Big Code, when researchers seek to study an increasingly large number of repositories to support their findings, the data processing stage may require manipulating millions and more of records. In this work we focus on studies involving fine-grained AST level source code changes. We present how we extended the CodeDistillery source code mining framework with data manipulation capabilities, aimed to alleviate the processing of large datasets of fine grained source code changes. The capabilities we have introduced allow researchers to highly automate their repository mining process and streamline the data acquisition and processing phases. These capabilities have been successfully used to conduct a number of studies, in the course of which dozens of millions of fine-grained source code changes have been processed.

A significant percentage of warnings reported by tools to detect coding standard violations are false positives. Thus, there are some works dedicated to provide better rules by mining them from source code history, analyzing bug-fixes or changes between system releases. However, software evolves over time, and during development not only bugs are fixed, but also features are added, and code is refactored. In such cases, changes must be consistently applied in source code to avoid maintenance problems. In this paper, we propose to extract system specific rules by mining systematic changes over source code history, i.e., not just from bug-fixes or system releases, to ensure that changes are consistently applied over source code. We focus on structural changes done to support API modification or evolution with the goal of providing better rules to developers. Also, rules are mined from predefined rule patterns that ensure their quality. In order to assess the precision of such specific rules to detect real violations, we compare them with generic rules provided by tools to detect coding standard violations on four real world systems covering two programming languages. The results show that specific rules are more precise in identifying real violations in source code than generic ones, and thus can complement them.

Regression test prioritization (RTP) is an active research field, aiming at re‐ordering the tests in a test suite to maximize the rate at which faults are detected. A number of RTP strategies have been proposed, leveraging different factors to reorder tests. Some techniques include an analysis of changed source code, to assign higher priority to tests stressing modified parts of the codebase. Still, most of these change‐based solutions focus on simple text‐level comparisons among versions. We believe that measuring source code changes in a more refined way, capable of discriminating between mere textual changes (e.g., renaming of a local variable) and more structural changes (e.g., changes in the control flow), could lead to significant benefits in RTP, under the assumption that major structural changes are also more likely to introduce faults. To this end, we propose two novel RTP techniques that leverage tree kernels (TK), a class of similarity functions largely used in Natural Language Processing on tree‐structured data. In particular, we apply TKs to abstract syntax trees of source code, to more precisely quantify the extent of structural changes in the source code, and prioritize tests accordingly. We assessed the effectiveness of the proposals by conducting an empirical study on five real‐world Java projects, also used in a number of RTP‐related papers. We automatically generated, for each considered pair of software versions (i.e., old version, new version) in the evolution of the involved projects, 100 variations with artificially injected faults, leading to over 5k different software evolution scenarios overall. We compared the proposed prioritization approaches against well‐known prioritization techniques, evaluating both their effectiveness and their execution times. Our findings show that leveraging more refined code change analysis techniques to quantify the extent of changes in source code can lead to relevant improvements in prioritization effectiveness, while typically introducing negligible overheads due to their execution.

Commit messages, which summarize the source code changes in natural language, are essential for program comprehension and software evolution understanding. Unfortunately, due to the lack of direct motivation, commit messages are sometimes neglected by developers, making it necessary to automatically generate such messages. State-of-the-art adopts learning based approaches such as neural machine translation models for the commit message generation problem. However, they tend to ignore the code structure information and suffer from the out-of-vocabulary issue. In this paper, we propose CoDiSum to address the above two limitations. In particular, we first extract both code structure and code semantics from the source code changes, and then jointly model these two sources of information so as to better learn the representations of the code changes. Moreover, we augment the model with copying mechanism to further mitigate the out-of-vocabulary issue. Experimental evaluations on real data demonstrate that the proposed approach significantly outperforms the state-of-the-art in terms of accurately generating the commit messages.

Source code comments are a valuable instrument to preserve design decisions and to communicate the intent of the code to programmers and maintainers. Nevertheless, commenting source code and keeping comments up-to-date is often neglected for reasons of time or programmers obliviousness. In this paper, we investigate the question whether developers comment their code and to what extent they add comments or adapt them when they evolve the code. We present an approach to associate comments with source code entities to track their co-evolution over multiple versions. A set of heuristics are used to decide whether a comment is associated with its preceding or its succeeding source code entity. We analyzed the co-evolution of code and comments in eight different open source and closed source software systems. We found with statistical significance that (1) the relative amount of comments and source code grows at about the same rate; (2) the type of a source code entity, such as a method declaration or an if-statement, has a significant influence on whether or not it gets commented; (3) in six out of the eight systems, code and comments co-evolve in 90% of the cases; and (4) surprisingly, API changes and comments do not co-evolve but they are re-documented in a later revision. As a result, our approach enables a quantitative assessment of the commenting process in a software system. We can, therefore, leverage the results to provide feedback during development to increase the awareness of when to add comments or when to adapt comments because of source code changes.

In recent years, researchers have intensively investigated various topics in test prioritization, which aims to re-order tests to increase the rate of fault detection during regression testing. While the main research focus in test prioritization is on proposing novel prioritization techniques and evaluating on more and larger subject systems, little effort has been put on investigating the threats to validity in existing work on test prioritization. One main threat to validity is that existing work mainly evaluates prioritization techniques based on simple artificial changes on the source code and tests. For example, the changes in the source code usually include only seeded program faults, whereas the test suite is usually not augmented at all. On the contrary, in real-world software development, software systems usually undergo various changes on the source code and test suite augmentation. Therefore, it is not clear whether the conclusions drawn by existing work in test prioritization from the artificial changes are still valid for real-world software evolution. In this paper, we present the first empirical study to investigate this important threat to validity in test prioritization. We reimplemented 24 variant techniques of both the traditional and time-aware test prioritization, and investigated the impacts of software evolution on those techniques based on the version history of 8 real-world Java programs from GitHub. The results show that for both traditional and time-aware test prioritization, test suite augmentation significantly hampers their effectiveness, whereas source code changes alone do not influence their effectiveness much.

Regression tests ensure that existing functionality is not impaired by changes to an existing software system. However, executing complete test suites often takes much time. Therefore, a subset of tests has to be found that is sufficient to validate whether the system under test is still valid after it has been changed. This test case selection is especially important if regression tests are executed manually, since manual execution is time intensive and costly. To select manual test cases, many regression testing techniques exist that aim on achieving coverage of changed or even new code. Many of these techniques base on coverage data from prior test runs or logical properties such as annotated pre and post conditions in the source code. However, coverage information becomes outdated if a system is changed extensively. Also annotated logical properties are often not available in industrial software systems. We present an approach for test selection that is based on static analyses of the test suite and the system's source code. We combine trace link recovery using latent semantic indexing with the metric change coverage, which assesses the coverage of source code changes. The proposed approach works automatically without the need to execute tests beforehand or annotate source code. Furthermore, we present a first evaluation of the approach.

To quickly locate the source code that maps to a specific change described in change history, establishing traceability links between release notes and source code is a necessary task. Current works on the traceability link recovery can be used to find out source code changes which are of higher textual similarities with the release note. However, these approaches rely on consistency of the text used in artifacts at various abstraction levels, and the completeness of text descriptions. In this paper, we propose to leverage source code change information for improving the accuracy of release note to source code traceability recovery tasks. In order to reduce the complexity of link recovery, our approach first performs change impact analysis to cluster the source code changes for the same purpose as a virtual class. After that, our approach employs a natural language generation algorithm to generate readable summary sentence for each virtual class. The traceability links are built between release notes and clusters of program entities by computing the linguistic similarity of sentences. We conduct case studies on 26 releases of 3 popular softwares to evaluate the approach, and the results indicate that our proposed method can improve the accuracy of traceability link recovery compared to other IR-based techniques.

Tao is a system that optimizes the execution of unit tests in large software programs and reduces the programmer wait time from minutes to seconds. Tao is based on two key ideas: First, Tao focuses on efficiency, unlike past work that focused on avoiding false negatives. Tao implements simple and fast function-level dependency tracking that identifies tests to run on a code change; any false negatives missed by this dependency tracking are caught by running the entire test suite on a test server once the code change is committed. Second, to make it easy for programmers to adopt Tao, it incorporates the dependency information into the source code repository. This paper describes an early prototype of Tao and demonstrates that Tao can reduce unit test execution time in two large Python software projects by over 96% while incurring few false negatives.

Scrum of scrums is an approach used to scale the traditional Scrum methodology to fit for the development of complex and large projects. However, scaling the roles of scrum members brought new challenges especially in distributed and large software projects. This paper describes in details the roles of each scrum member in scrum of scrum to propose a solution to use a dedicated product owner for a team and inclusion of sub-backlog. The main goal of the proposed solution is to optimize the role of product owner for distributed large projects. The proposed changes will increase cohesiveness among scrum teams and it will also eliminate duplication of work. Survey is used as a research design to evaluate the proposed solution. The results are found encouraging supporting the proposed solution. It is anticipated that the proposed solution will help the software companies to scale Scrum methodology effectively for large and complex software projects.

We present a tool that predicts whether the software under development inside an IDE has a bug. An IDE plugin performs this prediction, using the Change Classification technique to classify source code changes as buggy or clean during the editing session. Change Classification uses Support Vector Machines (SVM), a machine learning classifier algorithm, to classify changes to projects mined from their configuration management repository. This technique, besides being language independent and relatively accurate, can (a) classify a change immediately upon its completion and (b) use features extracted solely from the change delta (added, deleted) and the source code to predict buggy changes. Thus, integrating change classification within an IDE can predict potential bugs in the software as the developer edits the source code, ideally reducing the amount of time spent on fixing bugs later. To this end, we have developed a Change Classification plugin for Eclipse based on client-server architecture, described in this paper.

The Bison fuel performance code is being developed by Idaho National Laboratory (INL) to analyze a wide range of fuel forms and operating conditions with the flexibility to conduct simulations in 1D through 3D geometries. Bison plays an important role in the development of advanced fuel with improved accident tolerance for existing light water reactors (LWRs), the improved understanding of mechanisms in fuel designs in a wider range of operating conditions, and the facilitation of future development of fuel for advanced reactor designs. Additionally, industry interest in advanced fuels has also driven Bison development for advanced fuels in the last few years. All of these factors combine to create a potential increase of new Bison users. These new users will potentially be interested in the Bison LWR validation base for examples of Bison's capabilities and how to use Bison. Current users also rely on the Bison validation base as a demonstration of the best practices settings and material behavioral models. Bison, both through direct updates to the Bison source code and to the nearly daily changes and improvements in the underlying MOOSE framework source code, undergoes nearly continuous changes. It is necessary that these source code changes are reflected with updated input files in the Bison validation base. Periodic updates of the validation case input files should also be accompanied by a review and revision of the validation case documentation on which our users rely. Therefore our efforts under this milestone were guided by two objectives: (1) Update the Bison LWR validation input files to use current Bison simulation settings, and (2) Enhance the ease-of-use for Bison through improved documentation These objectives build on the goals identified FY18 CASL milestone reports. During FY18 we addressed issues with the robustness of Bison and identified a need to migrate the assessment suite to the more reliable tensor mechanics module system and set a goal of standardizing the Bison input files. The FY18 CASL milestone report on updates to the Bison documentation system listed continued development of user-focused guides and examples along with revisions to the validation case documentation as future development goals. During FY19 we have separated these two objectives into four different focus areas for our work on this milestone. Our first two focus areas target the existing Bison validation suite: update the existing LWR validation cases and the accompanying documentation. The second two focus areas center on adding new capabilities: a new Bison LWR validation case and expansion of user support documentation. In the following report each of these four focus areas are discussed as a separate section. We conclude the report with a summary of potential future work to build on the efforts of this year.

“When you can measure what you are speaking about and express it in numbers, you know something about it; but when you cannot measure, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely, in your thoughts, advanced to the stage of science.” LORD WILLIAM KELVIN (1824 – 1907). During the last decade, the quantification of software engineering process has got a pace due to availability of a huge amount of software repositories. These repositories include source code, bug, communication among developers/users, changes in code, etc. Researchers are trying to find out useful information from these repositories for improving the quality of software.The absence of bugs in the software is a major factor that decides the quality of software. In the available literature, researchers have proposed and implemented a plethora of bug prediction approaches varying in terms of accuracy, complexity and input data. The code change metric based bug prediction is proven to be very useful. In the literature, decay functions have been proposed that decay the complexity of code changes over a period of time in either exponential or linear fashion but they do not fit in open source software development paradigm because in open source software development paradigm, the development team is geographical dispersed and there is an irregular fluctuation in the code changes and bug detection/fixing process. The complexity of code changes reduces over a period of time that may be less than exponential or more than linear. This paper presents the method that quantifies the code change metric and also proposed decay functions that capture the variability in the decay curves represented the complexity of code changes. The proposed decay functions model the complexity of code changes which reduces over a period of time and follows different types of decay curves. We have collected the source code change data of Mozilla components and applied simple linear regression (SLR) and support vector regression (SVR) techniques to validate the proposed method and predict the bugs yet to come in future based on the current year complexity of code changes (entropy). The performance of proposed models has been compared using different performance criteria namely R2, Adjusted R2, Variation and Root Mean Squared Prediction Error (RMSPE).

The lack of compatibility of High‐Performance Fortran (HPF) between vender implementations has been disheartening scientific application users so as to hinder the development of portable programs. Thus parallel computing is still unpopular in the computational science community, even though parallel programming is common to the computer science community. As users would like to run the same source code on parallel machines with different architectures as fast as possible, we have investigated the compatibility of source codes for Japanese HPF compilers (NEC, Fujitsu and Hitachi) with two real‐world applications: a 3D fluid code and a 2D particle code. We have found that the source‐level compatibility between Japanese HPF compilers is almost preserved, but more effort will be needed to sustain complete compatibility. We have also evaluated parallel performance and found that HPF can achieve good performance for the 3D fluid code with almost the same source code. For the 2D particle code, good results have also been obtained with a small number of processors, but some changes in the original source code and the addition of interface blocks is required. Copyright © 2002 John Wiley & Sons, Ltd.

Towards exact and inexact approximate matching of executable binaries