Quick Similarity Measurement of Source Code Based on Suffix Array

Suffix arrays are used in various application and research areas like data compression or computational biology. In this work, our goal is to characterize the combinatorial properties of suffix arrays and their enumeration. For fixed alphabet size and string length we count the number of strings sharing the same suffix array and the number of such suffix arrays. Our methods have applications to succinct suffix arrays and build the foundation for the efficient generation of appropriate test data sets for suffix array based algorithms.We also show that summing up the strings for all suffix arrays builds a particular instance for some summation identities of Eulerian numbers.

Given a sequence T = t0t1 . . . tn-1 of size n = |T|, with symbols from a fixed alphabet Σ, (|Σ| ≤ n), the suffix array provides a listing of all the suffixes of T in a lexicographic order. Given T, the suffix sorting problem is to construct its suffix array. The direct suffix sorting problem is to construct the suffix array of T directly without using the suffix tree data structure. While algorithims for linear time, linear space direct suffix sorting have been proposed, the actual constant in the linear space is still a major concern, given that the applications of suffix trees and suffix arrays (such as in whole-genome analysis) often involve huge data sets. In this work, we reduce the gap between current results and the minimal space requirement. We introduce an algorithm for the direct suffix sorting problem with worst case time complexity in O(n), requiring only (1 2 3 n log n - n log | ∑ |+O(1)) bits in memory space. This implies 5 2 3 n+O(1) bytes for total space requirment, (including space for both the output suffix array and the input sequence T) assuming n ≤ 2 32 ,| ∑ |≤256 , and 4 bytes per integer. The basis of our algorithm is an extension of Shannon-Fano-Elias codes used in source coding and information theory. This is the first time information-theoretic methods have been used as the basis for solving the suffix sorting problem.

BackgroundAn absent word of a word y of length n is a word that does not occur in y. It is a minimal absent word if all its proper factors occur in y. Minimal absent words have been computed in genomes of organisms from all domains of life; their computation also provides a fast alternative for measuring approximation in sequence comparison. There exists an tttttttmathcal {O}(n)tttttt-time and tttttttmathcal {O}(n)tttttt-space algorithm for computing all minimal absent words on a fixed-sized alphabet based on the construction of suffix automata (Crochemore et al., 1998). No implementation of this algorithm is publicly available. There also exists an tttttttmathcal {O}(n^{2})tttttt-time and tttttttmathcal {O}(n)tttttt-space algorithm for the same problem based on the construction of suffix arrays (Pinho et al., 2009). An implementation of this algorithm was also provided by the authors and is currently the fastest available.ResultsOur contribution in this article is twofold: first, we bridge this unpleasant gap by presenting an tttttttmathcal {O}(n)tttttt-time and tttttttmathcal {O}(n)tttttt-space algorithm for computing all minimal absent words based on the construction of suffix arrays; and second, we provide the respective implementation of this algorithm. Experimental results, using real and synthetic data, show that this implementation outperforms the one by Pinho et al. The open-source code of our implementation is freely available at http://github.com/solonas13/maw.ConclusionsClassical notions for sequence comparison are increasingly being replaced by other similarity measures that refer to the composition of sequences in terms of their constituent patterns. One such measure is the minimal absent words. In this article, we present a new linear-time and linear-space algorithm for the computation of minimal absent words based on the suffix array.

String matching is a ubiquitous problem that arises in a wide range of applications in computing, e.g., packet routing, intrusion detection, web querying, and genome analysis. Due to its importance, dozens of algorithms and several data structures have been developed over the years. A recent breakthrough in this field is the FM-index, a data structure that synergistically combines the Burrows-Wheeler transform and the suffix array. In software, the FM-index allows searching (exact and approximate) in times comparable to the fastest known indices for large texts (suffix trees and suffix arrays), but has the additional advantage of being more space-efficient than those approaches. In this paper, we describe the first FPGA-based hardware implementation of the FM-index for exact pattern matching. We report experimental results on the problem of mapping short DNA sequences to a reference genome. We show that the throughput of the FM-index is significantly higher than the naive (brute force) approach. Like the Bowtie software tool, the FM-index can abandon early the hardware matching. It outperforms Bowtie by two orders of magnitude.

Summary: We introduce the tool mkESA, an open source program for constructing enhanced suffix arrays (ESAs), striving for low memory consumption, yet high practical speed. mkESA is a user-friendly program written in portable C99, based on a parallelized version of the Deep-Shallow suffix array construction algorithm, which is known for its high speed and small memory usage. The tool handles large FASTA files with multiple sequences, and computes suffix arrays and various additional tables, such as the LCP table (longest common prefix) or the inverse suffix array, from given sequence data.Availability: The source code of mkESA is freely available under the terms of the GNU General Public License (GPL) version 2 at http://bibiserv.techfak.uni-bielefeld.de/mkesa/.Contact: rhomann@techfak.uni-bielefeld.de

With the use of increasingly complex software, software bugs are inevitable. Software developers rely on bug reports to identify and fix these issues. In this process, developers inspect suspected buggy source code files, relying heavily on a bug report. This process is often time-consuming and increases the cost of software maintenance. To resolve this problem, we propose a novel bug localization method using topic-based similar commit information. First, the method determines similar topics for a given bug report. Then, it extracts similar bug reports and similar commit information for these topics. To extract similar bug reports on a topic, a similarity measure is calculated for a given bug report. In the process, for a given bug report and source code, features shared by similar source codes are classified and extracted; combining these features improves the method’s performance. The extracted features are presented to the convolutional neural network’s long short-term memory algorithm for model training. Finally, when a bug report is submitted to the model, a suspected buggy source code file is detected and recommended. To evaluate the performance of our method, a baseline performance comparison was conducted using code from open-source projects. Our method exhibits good performance.

In this paper we describe a new algorithm for building the suffix array of a string. This task is equivalent to the problem of lexicographically sorting all the suffixes of the input string. Our algorithm is based on a new approach called deep–shallow sorting: we use a “shallow” sorter for the suffixes with a short common prefix, and a “deep” sorter for the suffixes with a long common prefix. All the known algorithms for building the suffix array either require a large amount of space or are inefficient when the input string contains many repeated substrings. Our algorithm has been designed to overcome this dichotomy. Our algorithm is “lightweight” in the sense that it uses very small space in addition to the space required by the suffix array itself. At the same time our algorithm is fast even when the input contains many repetitions: this has been shown by extensive experiments with inputs of size up to 110 Mb. The source code of our algorithm, as well as a C library providing a simple API, is available under the GNU GPL.

Illegal code reuse has become a serious threat to the software community. Identifying similar or identical code fragments becomes much more challenging in code theft cases where plagiarizers can use various automated code transformation or obfuscation techniques to hide stolen code from being detected. Previous works in this field are largely limited in that (i) most of them cannot handle advanced obfuscation techniques, and (ii) the methods based on source code analysis are not practical since the source code of suspicious programs typically cannot be obtained until strong evidences have been collected. Based on the observation that some critical runtime values of a program are hard to be replaced or eliminated by semantics-preserving transformation techniques, we introduce a novel approach to dynamic characterization of executable programs. Leveraging such invariant values, our technique is resilient to various control and data obfuscation techniques. We show how the values can be extracted and refined to expose the critical values and how we can apply this runtime property to help solve problems in software plagiarism detection. We have implemented a prototype with a dynamic taint analyzer atop a generic processor emulator. Our value-based plagiarism detection method (VaPD) uses the longest common subsequence based similarity measuring algorithms to check whether two code fragments belong to the same lineage. We evaluate our proposed method through a set of real-world automated obfuscators. Our experimental results show that the value-based method successfully discriminates 34 plagiarisms obfuscated by SandMark, plagiarisms heavily obfuscated by KlassMaster, programs obfuscated by Thicket, and executables obfuscated by Loco/Diablo.

The source code similarity problem consists in defining if two given distinct codes are the same program or not. This problem is valuable for polymorphic malware detection, which can generate distinct versions of code applying rules of obfuscation that change the original code. A technique to measure the similarity between source codes is to model the codes as program dependency graphs (PDG) and find an alignment between the graphs that maximizes a similarity function. This work investigates the performance of distinct similarity metrics applied to a genetic algorithm (GA). To distinguish between similar codes and non-similar codes, we use the k-nearest-neighbors (KNN) algorithm based on the similarity of the alignment found by the GA. The experiments are conducted with a database proposed by this work, where the source codes were retrieved from the Codeforces website and obfuscated by the tool CXX-obfuscator.

In the software development cycle, the requirement traceability link is one of the important factors. Information retrieval techniques typically produce links with low precision and/or recall because, due to their very nature, they depend on the textual similarity between requirements and source code. The developers may not evolve requirements in synchronization with source code. But, they frequently update other sources of information including CVS/SVN repositories, bug-tracking systems, mailing lists, forums, and blogs. These sources of information will be used to build improved requirement traceability-recovery approaches. After the source code was developed, the developer compiles the source code and commits the source code into versions. Then admin needs to compare the templates and versions based on similarity measurement. Based on textual comparison, the similarity is calculated. The admin can do the same process for the forum requirements.

The delay and throughput characteristics of a packet switch depend mainly on the queueing scheme and the scheduling algorithm deployed at the switch. Early research on scheduling algorithms has mainly focused on maximum weight matching scheduling algorithms. It is known that maximum weight matching algorithms guarantee the stability of input-queued switches, but are impractical due to their high computational complexity. Later research showed that the less complex maximal matching algorithms can stabilize input-queued switches when they are deployed with a speed-up of two. For practical purposes, neither a high computational complexity nor a speed-up of two is desirable.In this paper, we investigate the application of matching algorithms that approximate maximum weight matching algorithms to scheduling problems. We show that while having a low computational complexity, they guarantee the stability of input queued switches when they are deployed with a moderate speed-up.In particular, we show that the improve_matching algorithm stabilizes input-queued switches when it is deployed with a speed-up of 3<over>2+ε.In a second step, we further improve on these results by proposing a class of maximal weight matching algorithms that stabilize an input-queued switch without any speed-up.Whereas initial research has only focused on scheduling algorithms that guarantee the stability of a single switch, recent work has shown how scheduling algorithms for single switches can be modified in order to design distributed scheduling algorithms that stabilize networks of input-queued switches. Using those results, we show that the switching algorithms proposed in this paper do not only stabilize a single switch, but also networks of input-queued switches.

SummaryPlagiarism is becoming an increasingly serious problem in academic environment. In this paper, we deal with a specific kind of plagiarism: source code plagiarism. In this case, there is no software available for detecting plagiarism on a larger scale (hundreds of student submissions every year). We propose algorithms for source code parsing and processing as a part of a complex system for plagiarism detection. A source code vectorization using characteristic vectors is a vital piece of the whole process, and k‐means algorithm helps with the classification and clustering of vectors. Student assignments are submitted regularly, and any plagiarism detection system needs to handle them as they come. For this reason, we propose a modified incremental k‐means algorithm and a method for determining the number of clusters. We also consider methods for vector search among clusters and suggest the use of conditional entropy to select the important vector elements used in the search algorithm. Our results show how the proposed algorithms and methods work on real student submissions.

Plagiarism detection in software programs is very important in both academia and industry. There are many tools available today to aid in detecting plagiarism in text and source documents. Most of the plagiarism detection tools for source code do some sort of pre-processing, like removing comments & blank spaces and using uniform tokens for primitive types & variables, before actually applying intelligent matching techniques so that the tool can detect the copied code even if it is changed in format and variable names etc. In this paper, we propose a novel preprocessing approach based on design patterns such that any code that is intelligently plagiarized can also be easily located.

Third-generation sequencing technologies can sequence long reads that contain as many as 2 million base pairs. These long reads are used to construct an assembly (i.e. the subject's genome), which is further used in downstream genome analysis. Unfortunately, third-generation sequencing technologies have high sequencing error rates and a large proportion of base pairs in these long reads is incorrectly identified. These errors propagate to the assembly and affect the accuracy of genome analysis. Assembly polishing algorithms minimize such error propagation by polishing or fixing errors in the assembly by using information from alignments between reads and the assembly (i.e. read-to-assembly alignment information). However, current assembly polishing algorithms can only polish an assembly using reads from either a certain sequencing technology or a small assembly. Such technology-dependency and assembly-size dependency require researchers to (i) run multiple polishing algorithms and (ii) use small chunks of a large genome to use all available readsets and polish large genomes, respectively. We introduce Apollo, a universal assembly polishing algorithm that scales well to polish an assembly of any size (i.e. both large and small genomes) using reads from all sequencing technologies (i.e. second- and third-generation). Our goal is to provide a single algorithm that uses read sets from all available sequencing technologies to improve the accuracy of assembly polishing and that can polish large genomes. Apollo (i) models an assembly as a profile hidden Markov model (pHMM), (ii) uses read-to-assembly alignment to train the pHMM with the Forward-Backward algorithm and (iii) decodes the trained model with the Viterbi algorithm to produce a polished assembly. Our experiments with real readsets demonstrate that Apollo is the only algorithm that (i) uses reads from any sequencing technology within a single run and (ii) scales well to polish large assemblies without splitting the assembly into multiple parts. Source code is available at https://github.com/CMU-SAFARI/Apollo. Supplementary data are available at Bioinformatics online.

Under the background of big data era, massive trajectory data is constantly generated, which contains great value. The analysis and mining of spatiotemporal trajectory data is a research hotspot of spatial data, including trajectory retrieval, trajectory classification etc. In the process of analysis and mining, the similarity measurement between different trajectories is a key problem. Considering the amount of data, computational complexity, noise and other factors, different metrics need to be selected in different situations. In order to understand the performance of different similarity measurement algorithms, this paper selects the Longest Common Sub-Sequence Algorithm (LCSS), Fréchet Distance Algorithm and One Way Distance Algorithm (OWD) among the three kinds of trajectory similarity measurement algorithms and they were used for comparison. Experimental results show that LCSS algorithm can effectively resist the interference of noise points, Fréchet Distance Algorithm has strong robustness, and OWD Algorithm has low time complexity and short execution time.