Suffix trees for inputs larger than main memory

A close inspection of Fiala and Green's implementation of Ziv-Lempel'77 dictionary compression method reveals a surprising inefficiency. The match they are searching for from the root down the suffix tree can be obtained for free from the suffix tree construction algorithm! This observation suggests that the output of this compression method is in a way just a description of the suffix tree construction for the input string. If taken one step further this leads to the idea that there exists a whole family of compression methods replacing the input string with a description of the suffix tree construction for this string. This family contains some (multi-)dictionary methods not dissimilar to those of Bloom or Hoang et.al., as well as some prediction by partial matching variants. We give the general description of this family along with details about some of its members, discuss implementation issues and show some early experimental results. A possible application to other suffix structures like the compact directed acyclic word graph is also outlined.

We will present two methods for constructing suffix trees in detail, Ukkonen's method and Weiner's method. Weiner was the first to show that suffix trees can be built in linear time, and his method is presented both for its historical importance and for some different technical ideas that it contains. However, Ukkonen's method is equally fast and uses far less space (i.e., memory) in practice than Weiner's method. Hence Ukkonen is the method of choice for most problems requiring the construction of a suffix tree. We also believe that Ukkonen's method is easier to understand. Therefore, it will be presented first. A reader who wishes to study only one method is advised to concentrate on it. However, our development of Weiner's method does not depend on understanding Ukkonen's algorithm, and the two algorithms can be read independently (with one small shared section noted in the description of Weiner's method). Ukkonen's linear-time suffix tree algorithm Esko Ukkonen [438] devised a linear-time algorithm for constructing a suffix tree that may be the conceptually easiest linear-time construction algorithm. This algorithm has a space-saving improvement over Weiner's algorithm (which was achieved first in the development of McCreight's algorithm), and it has a certain “on-line” property that may be useful in some situations. We will describe that on-line property but emphasize that the main virtue of Ukkonen's algorithm is the simplicity of its description, proof, and time analysis.

Improving on-line construction of two-dimensional suffix trees for square matrices

The construction of suffix trees in secondary storage was considered impractical due to its excessive I/O cost. Algorithms developed in the last decade show that a suffix tree can efficiently be built in secondary storage for inputs which fit the main memory. In this paper, we analyze the details of algorithmic approaches to the external memory suffix tree construction and compare the performance and scalability of existing state‐of‐the‐art software based on these algorithms. Copyright © 2010 John Wiley & Sons, Ltd.

We discuss an intrinsic generalization of the suffix tree, designed to index a string of length n which has a natural partitioning into m multicharacter substrings or words . This word suffix tree represents only the m suffixes that start at word boundaries. These boundaries are determined by delimiters , whose definition depends on the application. Since traditional suffix tree construction algorithms rely heavily on the fact that all suffixes are inserted, construction of a word suffix tree is nontrivial, in particular when only O(m) construction space is allowed. We solve this problem, presenting an algorithm with O(n) expected running time. In general, construction cost is Ω(n) due to the need of scanning the entire input. In applications that require strict node ordering, an additional cost of sorting O(m') characters arises, where m' is the number of distinct words. In either case, this is a significant improvement over previously known solutions. Furthermore, when the alphabet is small, we may assume that the n characters in the input string occupy o(n) machine words. We illustrate that this can allow a word suffix tree to be built in sublinear time.

Online persistent suffix tree construction has been considered impractical due to its excessive I/O costs. However, these prior studies have not taken into account the effects of the buffer management policy and the internal node structure of the suffix tree on I/O behavior of construction and subsequent retrievals over the tree. We study these two issues in detail in the context of large genomic DNA and protein sequences. In particular, we make the following contributions: (i) a novel, low-overhead buffering policy called TOP-Q which improves the on-disk behavior of suffix tree construction and subsequent retrievals, and (ii) empirical evidence that the space efficient linked-list representation of suffix tree nodes provides significantly inferior performance when compared to the array representation. These results demonstrate that a careful choice of implementation strategies can make online persistent suffix tree construction considerably more scalable - in terms of length of sequences indexed with a fixed memory budget, than currently perceived.

Generally, a suffix tree is an efficient data structure since it reveals the detailed internal structures of given sequences within linear time. However, it is difficult to implement a suffix tree for a large number of sequences in the ubiquitous devices because of the memory constraints. Therefore, in order to compare multi-megabase genomic DNA sequence sets using suffix trees, there is a need to re-construct the suffix tree algorithm. This paper introduces a new algorithm for constructing a suffix tree on the secondary storage with a large number of sequences. Our algorithm divides a suffix tree into three files, in a designated sequence, into parts, storing references to the locations of edges in hash tables.

We present an improved wavelet tree construction algorithm and discuss its applications to a number of rank/select problems for integer keys and strings.Given a string of length n over an alphabet of size σ ≤ n, our method builds the wavelet tree in O(n log σ / [EQUATION]log n) time, improving upon the state-of-the-art algorithm by a factor of [EQUATION]log n. As a consequence, given an array of n integers we can construct in O(n[EQUATION]log n) time a data structure consisting of O(n) machine words and capable of answering rank/select queries for the subranges of the array in O(log n/ log log n) time. This is a log log n-factor improvement in query time compared to Chan and Patrascu (SODA 2010) and a [EQUATION]log n-factor improvement in construction time compared to Brodal et al. (Theor. Comput. Sci. 2011).Next, we switch to stringological context and propose a novel notion of wavelet suffix trees. For a string w of length n, this data structure occupies O(n) words, takes O(n[EQUATION]log n) time to construct, and simultaneously captures the combinatorial structure of substrings of w while enabling efficient top-down traversal and binary search. In particular, with a wavelet suffix tree we are able to answer in O(log|x|) time the following two natural analogues of rank/select queries for suffixes of substrings:1) For substrings x and y of w (given by their endpoints) count the number of suffixes of x that are lexicographically smaller than y;2) For a substring x of w (given by its endpoints) and an integer k, find the k-th lexicographically smallest suffix of x.We further show that wavelet suffix trees allow to compute a run-length-encoded Burrows-Wheeler transform of a substring x of w (again, given by its endpoints) in O(s log |x|) time, where s denotes the length of the resulting run-length encoding. This answers a question by Cormode and Muthukrishnan (SODA 2005), who considered an analogous problem for Lempel-Ziv compression.All our algorithms, except for the construction of wavelet suffix trees, which additionally requires O(n) time in expectation, are deterministic and operate in the word RAM model.

We introduce a new text-indexing data structure, the String B-Tree , that can be seen as a link between some traditional external-memory and string-matching data structures. In a short phrase, it is a combination of B-trees and Patricia tries for internal-node indices that is made more effective by adding extra pointers to speed up search and update operations. Consequently, the String B-Tree overcomes the theoretical limitations of inverted files, B-trees, prefix B-trees, suffix arrays, compacted tries and suffix trees. String B-trees have the same worst-case performance as B-trees but they manage unbounded-length strings and perform much more powerful search operations such as the ones supported by suffix trees. String B-trees are also effective in main memory (RAM model) because they improve the online suffix tree search on a dynamic set of strings. They also can be successfully applied to database indexing and software duplication.

Genome comparison is central to contemporary genomics and typically relies on sequence alignment. However, genome-wide alignments are difficult to compute. We have, therefore, recently developed an accurate alignment-free estimator of the number of substitutions per site based on the lengths of exact matches between pairs of sequences. The previous implementation of this measure requires n(n-1) suffix tree constructions and traversals, where n is the number of sequences analyzed. This does not scale well for large n. We present an algorithm to extract pairwise distances in a single traversal of a single suffix tree containing n sequences. As a result, the run time of the suffix tree construction phase of our algorithm is reduced from O(n(2)L) to O(nL), where L is the length of each sequence. We implement this algorithm in the program kr version 2 and apply it to 825 HIV genomes, 13 genomes of enterobacteria and the complete genomes of 12 Drosophila species. We show that, depending on the input dataset, the new program is at least 10 times faster than its predecessor. Version 2 of kr can be tested via a web interface at http://guanine.evolbio.mpg.de/kr2/. It is written in standard C and its source code is available under the GNU General Public License from the same web site. haubold@evolbio.mpg.de Supplementary informations: Supplementary data are available at Bioinformatics online.

With the exponential growth of biological sequence databases, it has become critical to develop effective techniques for storing, querying, and analyzing these massive data. Suffix trees are widely used to solve many sequence-based problems, and they can be built in linear time and space, provided the resulting tree fits in main-memory. To index larger sequences, several external suffix tree algorithms have been proposed in recent years. However, they suffer from several problems such as susceptibility to data skew, non-scalability to genome-scale sequences, and non-existence of suffix links, which are crucial in various suffix tree based algorithms. In this paper, we target DNA sequences and propose a novel disk-based suffix tree algorithm called TRELLIS, which effectively scales up to genome-scale sequences. Specifically, it can index the entire human genome using 2GB of memory, in about 4 hours and can recover all its suffix links within 2 hours. TRELLIS was compared to various state-of-the-art persistent disk-based suffix tree construction algorithms, and was shown to outperform the best previous methods, both in terms of indexing time and querying time.

Pattern matching with wildcards is very important in many fields such as information retrieval and bioinformatics. Suffix trees are used in pattern matching with variable length wildcards. But the construction of a suffix tree needs significant time and space overload. This paper presents a new pattern matching algorithm, PST, based on multiple suffix trees. The PST algorithm uses a cutting process to divide a string S into several parts firstly, and then establishes a suffix tree for each part of S respectively. If multiple patterns are to be retrieved, the suffix trees should be adjusted according to the cutting points: prefix sequence deletion and suffix sequence addition; prefix sequence addition and suffix sequence deletion. Theoretical analysis and experiments show that the PST algorithm can decrease the time and space overload than other peers.

Main memory virtualization is used to expand the effective main memory capacity meaning that more virtual main memory is represented to guest operating systems than physical memory is actually available. This is done by compressing and uncompressing memory pages. Obviously the price for main memory virtualization is a higher CPU utilization, especially when dealing with high workloads. SAP ERP systems represent the backbone of today's enterprises and have a very high resource demand. The combination of main memory virtualization and SAP ERP system is of great importance but has not yet been thoroughly researched. This paper is the first approach to a quantitative research for evaluating the performance impact of memory virtualization on SAP ERP systems. We utilize a synthetic main memory benchmark, called Zachmanntest, to evaluate the performance impact of main memory virtualization on a SAP ERP system. As an exemplary main memory virtualization implementation IBM's Active Memory Expansion is used, where a so called expansion factor can be used to specify the size of the virtual main memory. Performance results show that the greater the expansion factor, the greater the performance impact. Regarding the peak performance, throughput of SAP ERP systems may be decreased till -17%, whereas the overall throughput of such systems may experience a decrease up to -42% when dealing with a very high compression factor.

Current processors employ aggressive prediction mechanisms to improve performance and reduce power. It is increasingly important to understand and quantify a program's dynamic behavior to effectively design next-generation prediction mechanisms. In this paper, we develop algorithms and mechanisms inspired by DNA discovery tools to analyze and quantify program dynamic behavior in terms of regularities and patterns. We describe our PatternFinder tool and analyze its results to summarize most important branch and data address pattern behaviors for a set of program traces and SPEC CPU 2006 benchmarks. M aking the common case fast is a design principle that has been used in microprocessor design for decades. This principle applies when determining how to spend resources, since the performance impact on making some occurrence faster is higher if the occurrence is frequent. Although quantifying frequent behavior in an application's dynamic execution behavior is trivial in cases such as observing the frequency of each type of instruction, it is very challenging to summarize dynamic data reference behavior (1). As a finding overlapping or approximate patterns. In this paper, inspired by DNA discovery tools (5, 6), we adopt and revise the methods motivated by suffix trees (5) in order to develop comprehensive pattern discovery tools targeted for computer architecture. Suffix trees have several advantages over SEQUITUR in designing such a tool, which we discuss in Section 3. In this paper, we present a novel pattern analysis tool, the PatternFinder, which is designed to discover and quantify patterns, and the results produced by the tool that quantify data address and branch patterns in dynamic program behavior. PatternFinder is an offline analysis tool that can summarize input sequences using patterns, quantify spatial and temporal regularities in input sequences and enumerate hot patterns based on specified scoring criteria (e.g., coverage). It allows user customizable search with a number of input parameters. PatternFinder can be used to gain insights into hardware and software optimization opportunities as well as perform pattern-centric

Suffix tree characterization of maximal motifs in biological sequences