Efficient pattern matching in compressed text via bit-parallel word-based encoding

The field of stringology studies algorithms and data structures used for processing strings efficiently. The goal of this thesis is to investigate 2-dimensional (2D) variants of some fundamental string problems, including \textit{Exact Pattern Matching} and \textit{Longest Common Substring}. In the 2D pattern matching problem, we are given a matrix $\M[1\dd n,1\dd n]$ that consists of $N = n \times n$ symbols drawn from an alphabet $\Sigma$ of size $\sigma$. The query consists of a $ m \times m$ square matrix $\PP[1\dd m, 1\dd m]$ drawn from the same alphabet, and the task is to find all the locations of $\PP$ in $\M$. For such square patterns, data structures such as suffix trees and suffix arrays exist for the task of efficient pattern matching. However, a suffix tree occupies $O(N \log N)$ bits, which is significantly more than that of the original text's size of $N\log \sigma$ bits. Therefore, the design of compressed data structures, that supports pattern matching queries efficiently and occupies space close to the original text's size, is imperative. In this thesis, we show an interesting result by designing a compact text index of size $O(N \log\log N + N \log\sigma)$ bits that at least supports efficient inverse suffix array queries. Although, the question of designing a compressed text index that would lead to efficient pattern matching is still evasive, this index gives a hope on the existence of a full 2D compressed text index with all functionalities similar to that of 1D case. On the other hand, the Longest Common 2D substring problem consists of two 2D strings (matrices), and the task is to report the size of the longest common 2D substring (submatrix) of these 2D strings. It is interesting to know if there exists a sublinear-time algorithm for solving this task. We answer this question positively by presenting a sublinear-time \textit{quantum} algorithm. In addition to this, we prove that any quantum algorithm requires at least $\tilde{\Omega}(N^{2/3})$ time to solve this problem.

In a 2-dimensional (2D) pattern matching problem, the text is arranged as a matrix 𝖬[1..n, 1..n] and consists of N = n × n symbols drawn from alphabet set Σ of size σ. The query consists of a m × m square matrix 𝖯[1..m, 1..m] drawn from the same alphabet set Σ and the task is to find all the locations in 𝖬 where 𝖯 appears as a (contiguous) submatrix. The patterns can be of any size, but as long as they are square in shape data structures like suffix trees and suffix array exist [Raffaele Giancarlo, 1995; Dong Kyue Kim et al., 1998] for the task of efficient pattern matching. These are essentially 2D counterparts of classic suffix trees and arrays known for traditional 1-dimensional (1D) pattern matching. They work based on linearization of 2D suffixes which would preserve the prefix match property (i.e., every pattern match is a prefix of some suffix). The main limitation of the suffix trees and the suffix arrays (in 1D) was their space utilization of O(N log N) bits, where N is the size of the text. This was suboptimal compared to Nlog σ bits of space, which is information theoretic optimal for the text. With the advent of the field of succinct/compressed data structures, it was possible to develop compressed variants of suffix trees and array based on Burrows-Wheeler Tansform and LF-mapping (or Φ function) [Roberto Grossi and Jeffrey Scott Vitter, 2005; Paolo Ferragina and Giovanni Manzini, 2005; Kunihiko Sadakane, 2007]. These data structures indeed achieve O(N log σ) bits of space or better. This gives rise to the question: analogous to 1D case, can we design a succinct or compressed index for 2D pattern matching? Can there be a 2D compressed suffix tree? Are there analogues of Burrows-Wheeler Transform or LF-mapping? The problem has been acknowledged for over a decade now and there have been a few attempts at applying Φ function [Ankur Gupta, 2004] and achieving entropy based compression [Veli Mäkinen and Gonzalo Navarro, 2008]. However, achieving the complexity breakthrough akin to 1D case has yet to be found. In this paper, we still do not know how to answer suffix array queries in O(N log σ) bits of space - which would have led to efficient pattern matching. However, for the first time, we show an interesting result that it is indeed possible to compute inverse suffix array (ISA) queries in near compact space in O(polylog n) time. Our 2D succinct text index design is based on two 1D compressed suffix trees and it takes O(N log log N + N logσ) bits of space which is much smaller than its naive design that takes O(N log N) bits. Although the main problem is still evasive, this index gives a hope on the existence of a full 2D succinct index with all functionalities similar to that of 1D case.

Efficient pattern matching is fundamental for practical term rewrite engines. By preprocessing the given patterns into a finite deterministic automaton the matching patterns can be decided in a single traversal of the relevant parts of the input term. Most automaton-based techniques are restricted to linear patterns, where each variable occurs at most once, and require an additional post-processing step to check so-called variable consistency. However, we can show that interleaving the variable consistency and pattern matching phases can reduce the number of required steps to find a match all matches. Therefore, we take the existing adaptive pattern matching automata as introduced by Sekar et al and extend it these with consistency checks. We prove that the resulting deterministic pattern matching automaton is correct, and show that its evaluation depth is can be shorter than two-phase approaches.

Efficient pattern matching is fundamental for practical term rewrite engines. By preprocessing the given patterns into a finite deterministic automaton the matching patterns can be decided in a single traversal of the relevant parts of the input term. Most automaton-based techniques are restricted to linear patterns, where each variable occurs at most once, and require an additional post-processing step to check so-called variable consistency. However, we can show that interleaving the variable consistency and pattern matching phases can reduce the number of required steps to find all matches. Therefore, we take the existing adaptive pattern matching automata as introduced by Sekar et al and extend these with consistency checks. We prove that the resulting deterministic pattern matching automaton is correct, and show several examples where some reduction can be achieved.

Counting problems, determining the number of possible states of a large system under certain constraints, play an important role in many areas of science. They naturally arise for complex disordered systems in physics and chemistry, in mathematical graph theory, and in computer science. Counting problems, however, are among the hardest problems to access computationally. Here, we suggest a novel method to access a benchmark counting problem, finding chromatic polynomials of graphs. We develop a vertex-oriented symbolic pattern matching algorithm that exploits the equivalence between the chromatic polynomial and the zero-temperature partition function of the Potts antiferromagnet on the same graph. Implementing this bottom-up algorithm using appropriate computer algebra, the new method outperforms standard top-down methods by several orders of magnitude, already for moderately sized graphs. As a first application, we compute chromatic polynomials of samples of the simple cubic lattice, for the first time computationally accessing three-dimensional lattices of physical relevance. The method offers straightforward generalizations to several other counting problems.

In the recent years stream time series data management has become an important research area due to its wide variety of real life applications. In this paper, we propose a technique for similarity matching between staiic/dynamic patterns and stream time-series image data. In order to perform effective retrieval we propose a multi scale segment median approximation representation for stream time-series image data, which can be computed with respect to change in temporal events and hence is suitable for cooperating with temporal behavior. In addition, we propose an efficient pruning algorithm over the multiscale representation. Mainly, the pruning scheme is to reduce the search space and to retrieve the similar patterns effectively. Experiments carried out in this work show that the techniques for stream time series image data representation and the level by level pruning scheme can efficiently generate all candidate patterns and perform similarity matching over stream time series image datasets and performs well compared to the existing methods such as multiscale wavelet.

This paper considers the Shift-And approach to the problem of pattern matching in LZW compressed text, and gives a new algorithm that solves it. The algorithm is indeed fast when a pattern length is at most 32, or the word length. After an O(m + |∑|) time and O(|∑|) space preprocessing of a pattern, it scans an LZW compressed text in O(n + r) time and reports all occurrences of the pattern, where n is the compressed text length, m is the pattern length, and r is the number of the pattern occurrences. Experimental results show that it runs approximately 1.5 times faster than a decompression followed by a simple search using the Shift-And algorithm. Moreover, the algorithm can be extended to the generalized pattern matching, to the pattern matching with k mismatches, and to the multiple pattern matching, like the Shift-And algorithm.

Several formalisms and tools for software development use hierarchy for system design, for instance statecharts and diagrams in UML. Message sequence charts are an ITU standardized notation for asynchronously communicating processes. The standard Z.120 allows (high-level) MSC-references that correspond to the use of macros. We consider in this paper two basic verification tasks for hierarchical MSCs (nested high-level MSCs, nHMSC), the membership and the pattern matching problem. We show that the membership problem for nHMSCs is PSPACE-complete, even using a weaker semantics for nMSCs than the partial-order semantics. For pattern matching nMSCs M,N we exhibit a polynomial algorithm of time O(M 2 · N 2). We use here techniques stemming from algorithms on compressed texts.

Pattern Matching in Compressed Texts and Images surveys and appraises techniques for pattern matching in compressed text and images. Normally compressed data needs to be decompressed before it is processed. If however the compression has been done in the right way, it is often possible to search the data without having to decompress it, or, at least, only partially decompress it. The problem can be divided into lossless and lossy compression methods, and then in each of these cases the pattern matching can be either exact or inexact. Much work has been reported in the literature on techniques for all of these cases. It includes algorithms that are suitable for pattern matching for various compression methods, and compression methods designed specifically for pattern matching. This monograph provides a survey of this work while also identifying the important relationship between pattern matching and compression, and proposing some performance measures for compressed pattern matching algorithms. Pattern Matching in Compressed Texts and Images is an excellent reference text for anyone who has an interest in the problem of searching compressed text and images. It concludes with a particularly insightful section on the ideas and research directions that are likely to occupy researchers in this field in the short and long term.

Pattern matching in Huffman encoded texts

Intrusion detection systems are treated as vital elements of protective measures to computer systems and networks from abuse. The drastic increase in network speed and detection workloads necessitates the need for highly efficient Network Intrusion Detection Systems(NIDS). Since most NIDSs need to check for a large number of known attack patterns in every packet, pattern matching becomes the most significant part of signature-based NIDSs in terms of processing and memory resources. To support segmentation of network traffic and to detect fragmented attacks, we propose a method which performs both 'partial' and 'full' pattern matching using the data structure CDAWG (Compact Direct Acyclic Word Graph). In the present work, we designed and implemented an efficient string matching algorithm using CDAWG structure. Experimental results show that this algorithm is 2.5 times faster than the currently used Aho-Corasick algorithm.

This paper defines a challenging problem of pattern matching between a pattern P and a text T, with wildcards and length constraints, and designs an efficient algorithm to return each pattern occurrence in an online manner. In this pattern matching problem, the user can specify the constraints on the number of wildcards between each two consecutive letters of P and the constraints on the length of each matching substring in T. We design a complete algorithm, SAIL that returns each matching substring of P in T as soon as it appears in T in an O(n+klmg) time with an O(lm) space overhead, where n is the length of T, k is the frequency of P's last letter occurring in T, l is the user-specified maximum length for each matching substring, m is the length of P, and g is the maximum difference between the user-specified maximum and minimum numbers of wildcards allowed between two consecutive letters in P.

With the advent of the era of big data, the scale of data has grown dramatically, and there is a close correlation between massive multi-source heterogeneous data, which can be visually depicted by a big graph. Big graph, especially from Web data, social networks, or biometric data, has attracted more and more attention from researchers, which usually contains complex relationships and multiple attributes. How to perform efficient query and matching on big graph data is the basic problem on analyzing big graph. Using multi-constrained graph pattern matching, we can design patterns that meet our specific requirements, and find matched subgraphs to locate the required patterns to accomplish specific tasks. So how to find matched subgraphs with good attributes in big graph becomes the key problem on big graph research. Considering the possibility that a node in a subgraph may fail due to reliability, in order to select more and better matched subgraphs, in this paper, we introduce fuzziness and reliability into multi-objective graph pattern matching, and then use a multi-objective genetic algorithm NSGA-II to find the subgraphs with higher reliability and better attributes including social trust and social relationship. Finally, a reliability-based multi-fuzzy-objective graph pattern matching method (named as RMFO-GPM) is proposed. The experimental results on real data sets show the effectiveness of the proposed RMFO-GPM method comparing with other state-of-art methods.

Inplace run-length 2d compressed search

We introduce a new variant of the popular Burrows-Wheeler transform (BWT), called Geometric Burrows-Wheeler Transform (GBWT), which converts a text into a set of points in 2-dimensional geometry. We also introduce a reverse transform, called Points2Text, which converts a set of points into text. Using these two transforms, we show strong equivalence between data structural problems in geometric range searching and text pattern matching. This allows us to apply the lower bounds known in the field of orthogonal range searching to the problems in compressed text indexing. In addition, we give the first succinct (compact) index for I/O-efficient pattern matching in external memory, and show how this index can be further improved to achieve higher-order entropy compressed space.