Deep learning code fragments for code clone detection

Software clone detection, aiming at identifying out code fragments with similar functionalities, has played an important role in software maintenance and evolution. Many clone detection approaches have been proposed. However, most of them represent source codes with hand-crafted features using lexical or syntactical information, or unsupervised deep features, which makes it difficult to detect the functional clone pairs, i.e., pieces of codes with similar functionality but differing in both syntactical and lexical level. In this paper, we address the software functional clone detection problem by learning supervised deep features. We formulate the clone detection as a supervised learning to hash problem and propose an end-to-end deep feature learning framework called CDLH for functional clone detection. Such framework learns hash codes by exploiting the lexical and syntactical information for fast computation of functional similarity between code fragments. Experiments on software clone detection benchmarks indicate that the CDLH approach is effective and outperforms the state-of-the-art approaches in software functional clone detection.

Exact or nearly similar code fragments in a software system's source code are referred to as code clones. It is often the case that updates (i.e., changes) to a code clone will need to be propagated to its related code clones to preserve their similarity and to maintain source code consistency. When there is a delay in propagating the changes (possibly because the developer is unaware of the related cloned code), the system might behave incorrectly. A delay in propagating a change is referred to as `late propagation,' and a number of studies have investigated this phenomenon. However, these studies did not investigate the intensity of late propagation nor how late propagation differs by clone type. In this research, we investigate late propagation separately for each of the three clone types (Type 1, Type 2, and Type 3). According to our experimental results on thousands of revisions of eight diverse subject systems written in two programming languages, late propagation occurs more frequently in Type 3 clones compared with the other two clone types. More importantly, there is a higher probability that Type 3 clones will experience buggy late propagations compared with the other two clone types. Also, we discovered that block clones are more involved in late propagation than method clones. Refactoring and tracking of Similarity Preserving Change Pattern (SPCP) clones (i.e., the clone fragments that evolve following a SPCP) can help us minimize the occurrences of late propagation in clones.

The ecosystem of open source software (OSS) has been growing considerably in size. In addition, code clones - code fragments that are copied and pasted within or between software systems - are also proliferating. Although code cloning may expedite the process of software development, it often critically affects the security of software because vulnerabilities and bugs can easily be propagated through code clones. These vulnerable code clones are increasing in conjunction with the growth of OSS, potentially contaminating many systems. Although researchers have attempted to detect code clones for decades, most of these attempts fail to scale to the size of the ever-growing OSS code base. The lack of scalability prevents software developers from readily managing code clones and associated vulnerabilities. Moreover, most existing clone detection techniques focus overly on merely detecting clones and this impairs their ability to accurately find "vulnerable" clones. In this paper, we propose VUDDY, an approach for the scalable detection of vulnerable code clones, which is capable of detecting security vulnerabilities in large software programs efficiently and accurately. Its extreme scalability is achieved by leveraging function-level granularity and a length-filtering technique that reduces the number of signature comparisons. This efficient design enables VUDDY to preprocess a billion lines of code in 14 hour and 17 minutes, after which it requires a few seconds to identify code clones. In addition, we designed a security-aware abstraction technique that renders VUDDY resilient to common modifications in cloned code, while preserving the vulnerable conditions even after the abstraction is applied. This extends the scope of VUDDY to identifying variants of known vulnerabilities, with high accuracy. In this study, we describe its principles and evaluate its efficacy and effectiveness by comparing it with existing mechanisms and presenting the vulnerabilities it detected. VUDDY outperformed four state-of-the-art code clone detection techniques in terms of both scalability and accuracy, and proved its effectiveness by detecting zero-day vulnerabilities in widely used software systems, such as Apache HTTPD and Ubuntu OS Distribution.

As Code Clones are defined on the notion of similarity in code fragments, it is necessary to first know what a code is meant by in accordance with Code Clones. A Source Code Fragment, which is a sequence of source code lines, is the basic entity that is used to analyze similarity/relation between Code Clones. For analysis, removal, avoidance, and management of Code Clones we have to first detect clones in software systems. There are more than 40 clone detection tools that implement some clone detection techniques to detect clones, but it is not well-defined what could be the appropriate minimum threshold for Clone length and with which unit of estimation. This paper, on the basis of Code Clone literature, presents different Units of Measurement of Clone Size and a comprehensive review of minimum Clone Size based on a particular technique used in Clone Detection and also argues that a unique Unit of Measurement and Minimum Clone Size should be presented.

A code clone is a code fragment identical or similar to another code fragment in source code. Some of code clones are considered as a factor of bug replications and make it more difficult to maintain software. Various code clone detection tools have been proposed so far. However, in most algorithms adopted by existing clone detection tools, if program statements are reordered, they are not detected as code clones. In this research, we examined how clone detection results change by rearranging the order of program statements. We performed preprocessing to rearranging the order of program statements using program dependency graph (PDG). We compared clone detection results with and without preprocessing. As a result, by rearranging the order of program statements, the number of detected code clones is almost the same in most projects. We classified newly detected or disappeared clones manually. From our experimental results, we show that there is no newly detected clone whose statements are reordered and that there are four disappeared clones whose statements are reordered. We think three out of the four clones occurred by copy-and-paste operations. Therefore, we conclude that rearranging the order of program statements is not effective to detect reordered code clones.

Code clones are prevalent in software systems due to many factors in software development. Detecting code clones and managing consistency between them along code evolution can be very useful for reducing clone-related bugs and maintenance costs. Despite some early attempts at detecting code clones and managing the consistency between them, the state-of-the-art tool can only handle simple code clones whose structures are identical or quite similar. However, existing empirical studies show that clones can have quite different structures with their evolution, which can easily go beyond the capability of the state-of-the-art tool. In this paper, we propose CCSync, a novel, rule-directed approach, which paves the structure differences between the code clones and synchronizes them even when code clones become quite different in their structures. The key steps of this approach are, given two code clones, to (1) extract a synchronization rule from the relationship between the clones, and (2) once one code fragment is updated, propagate the modifications to the other following the synchronization rule. We have implemented a tool for CCSync and evaluated its effectiveness on five Java projects. Our results shows that there are many code clones suitable for synchronization, and our tool achieves precisions of up to 92% and recalls of up to 84%. In particular, more than 76% of our generated revisions are identical with manual revisions.

A code clone (in short, clone) is a code fragment that is identical or similar to other code fragments in source code. The presence of clone is known as bad smell, which is phenomena of source code to be refactored. A motivation of refactoring (merging) clones is to reduce the size of source code. An existing study proposed a technique to estimate reduced lines of code by merging clones; however, there are two issues in the existing technique: (1) the existing technique does not consider the refactorability of clones in spite that it is difficult or even impossible to merge some clones due to the limitation of programming languages; (2) in the case that multiple clones are overlapping, the existing technique only considers one of them can be merged. Due to the above issues, estimated reducible LoC is occasionally different from the actual number. Consequently, in this research, we propose a new technique to calculate a reducible LoC. The proposed technique is free from the two issues, and it calculates a reducible LoC fully automatically. The proposed technique performs a loop processing of (a) detecting clones, (b) merging them, (c) compiling the edited source files, and (d) testing them. After finishing the loop, reducible LoC is calculated from the edited source files. This paper also includes comparison results of the proposed technique and the existing one. In the comparisons, we confirmed that a reducible LoC which was calculated with considering refactorability is 25% of a reducible LoC which was estimated without considering refactorability. We also confirmed that the proposed technique was able to merge clones that were not counted in the existing technique.

It has been noted in recent years that the presence of code clones makes software maintenance more difficult. Unintended code inconsistencies may occur due to the presence of code clones. In order to avoid problems caused by code clones, it is necessary to identify where code clones exist in a software system. Consequently, various kinds of code clone detection techniques have been proposed before now. In particular, incremental code clone detection attracts much attention in the last few years, and line-based and token-based incremental detection methods have been proposed. In incremental detections, code clone detection results or their intermediate products persist by using databases, and it is used in next code clone detection. However, no incremental detection technique has been proposed for PDG-based detection, which requires much more time to detect code clones than line-or token-based detection. In this paper, we propose a PDG-based incremental code clone detection technique for improving practicality of PDG-based detection. A prototype tool has been developed, and it has been applied to open source software. We confirmed that detection time is extremely shortened and its detection result is almost the same as one of an existing PDG-based detection technique.

Code segments are similar or identical to each other by copy-paste the original code at different locations with or without modifications are called code clones. Code clones are either by intentional or unintentional. In software development reprocess, the original code is most common. If any bug exists in original code segment, the bug remains in all the cloned code. As a result, bug propagation in cloning process affects in software maintenance. Detection and corrections are required. Code clone detection appears as an ongoing research area to solve these issues. In this paper, we carried out literature survey of 43 studies to summarize techniques and tools utilized for code clone detection. Six code clone detection techniques are based on text, token, tree, metric, graph, and hybrid to detect type-1, type-2, type-3, and type-4 clones. Our study is concluded with several studies that exist to detect the types 1–4 clones. There is a requirement of novel approach to develop to detect all four types of clones together. In addition, need to develop more approaches to simplify to create program dependency graph for type-4 clone detection.

Code clone detection technology aims to automatically detect code similarity and help developers identify and reduce code duplication.While code syntax analysis-based methods are commonly used for clone detection, they may not capture semantic information due to bypassing the analysis of code text.To address this issue, this paper proposes a new method called visualization representation learning for code clone detection (VRL4CCD).This method converts source code fragments into grayscale images to preserve textual information and then utilizes VGG16 and a self-attention mechanism to extract features related to code semantic similarity.A siamese neural network is used to learn the similarity pattern between code features.Experimental results on the Big Clone Bench and Google Code Jam datasets demonstrate that VRL4CCD outperforms current clone detection methods regarding precision, recall, and F1-score, indicating the effectiveness of code visualization technology in clone detection tasks.

Detecting code clones in a program has many applications in software engineering and other related fields. In this paper, we present Boreas, an accurate and scalable token-based approach for code clone detection. Boreas introduces a novel counting-based method to define the characteristic matrices, which are able to describe the program segments distinctly and effectively for the purpose of clone detection. We conducted experiments on JDK 7 and Linux kernel 2.6.38.6 source code. Experimental results show that Boreas is able to match the detecting accuracy of a recently proposed syntactic-based tool Deckard, with the execution time reduced by more than an order of magnitude.

Precision and recall are the main metrics used to measure the correctness of clone detectors. These metrics require the existence of labeled datasets containing the ground truth - samples of clone and non-clone pairs. For source code clone detectors, in particular, there are some techniques, as well as a concrete framework, for automatically evaluating recall, down to different types of clones. However, evaluating precision is still challenging, because of the intensive and specialized manual effort required to accomplish the task. Moreover, when precision is reported, it is typically done over all types of clones, making it hard to assess the strengths and weaknesses of the corresponding clone detectors. This paper presents systematic experiments to evaluate precision of eight code clone detection tools. Three judges independently reviewed 12,800 clone pairs to compute the undifferentiated and type-based precision of these tools. Besides providing a useful baseline for future research in code clone detection, another contribution of our work is to unveil important considerations to take into account when doing precision measurements and reporting the results. Specifically, our work shows that the reported precision of these tools leads to significantly different conclusions and insights about the tools when different types of clones are taken into account. It also stresses, once again, the importance of reporting inter-rater agreement.

So far, a lot of techniques have been developed on the detection of code clones (i.e., duplicated code) in large-scale source code. Currently, the code clone research community is gradually shifting its focus of attention from the detection to the management (e.g., clone refactoring). During clone management, developers need to understand how and why code clones are scattered in source code, and then decide how to handle those code clones. In this paper, we present a clone analysis tool with tag cloud visualization. This tool is aimed at helping to understand why code clone are concentrated in a part of a software system by generating tag clouds from a collection of identifier names in source code.

With the rise of deep learning in recent years, many code clone detection (CCD) methods use deep learning techniques and achieve promising results, so is cross-language CCD. However, deep learning techniques require a dataset to train the models. The dataset is typically small and has a gap between real-world clones due to the difficulty of collecting datasets for cross-language CCD. This creates a data bottleneck problem: data scale and quality issues will cause that model with a better design can still not reach its full potential. To mitigate this, we propose a tree autoencoder (TAE) architecture. It uses unsupervised learning to pretrain with abstract syntax trees (ASTs) of a large-scale dataset, then fine-tunes the trained encoder in the downstream CCD task. Our proposed TAE contains a tree Long Short-Term Memory (LSTM) encoder and a tree LSTM decoder. We design a novel embedding method for AST nodes, including type embedding and value embedding. In the training of TAE, we present an “encode and decode by layers” strategy and a node-level batch size design. For the CCD dataset, we propose a negative sampling method based on probability distribution. The experimental results on two datasets verify the effeteness of our embedding method, as well as that TAE and its pretrain enhance the performance of the CCD model. The node context information is well captured, and the reconstruction accuracy of the node-value reaches 95.45%. TAE pretrain improves the performance of CCD with a 4% increase in F1 score, which alleviates the data bottleneck problem.

Activities such as copying, pasting and modifying codes usually lead to a lot of code clones in the software systems. These code clones have a great impact on software development and maintenance. This paper presents a new clone detection approach. That is to uses a algorithm based on suffix array to efficiently search repeated token substrings and finally to detect code clones. We also develop a clone detection tool SaCD in order to implement the method. This tool has detected statement clones in 31 open source software systems of C, C++ and Java language. The experimental results show that the SaCD can efficiently detect code clones at the statement granularity in the software.