Shift-And Approach to Pattern Matching in LZW Compressed Text

Several formalisms and tools for software development use hierarchy for system design, for instance statecharts and diagrams in UML. Message sequence charts are an ITU standardized notation for asynchronously communicating processes. The standard Z.120 allows (high-level) MSC-references that correspond to the use of macros. We consider in this paper two basic verification tasks for hierarchical MSCs (nested high-level MSCs, nHMSC), the membership and the pattern matching problem. We show that the membership problem for nHMSCs is PSPACE-complete, even using a weaker semantics for nMSCs than the partial-order semantics. For pattern matching nMSCs M,N we exhibit a polynomial algorithm of time O(M 2 · N 2). We use here techniques stemming from algorithms on compressed texts.

We propose a protocol for the problem of secure two-party pattern matching, where Alice holds a text t?{0,1}? of length n, while Bob has a pattern p?{0,1}? of length m. The goal is for Bob to (only) learn where his pattern occurs in Alice's text, while Alice learns nothing. Private pattern matching is an important problem that has many applications in the area of DNA search, computational biology and more. Our construction guarantees full simulation in the presence of malicious, polynomial-time adversaries (assuming the hardness of DDH assumption) and exhibits computation and communication costs of O(n+m) group elements in a constant round complexity. This improves over previous work by Gennaro et al. (Public Key Cryptography, pp. 145---160, 2010) whose solution requires overhead of O(nm) group elements and exponentiations in O(m) rounds. In addition to the above, we propose a collection of protocols for important variations of the secure pattern matching problem that are significantly more efficient than the current state of art solutions: First, we deal with secure pattern matching with wildcards. In this variant the pattern may contain wildcards that match both 0 and 1. Our protocol requires O(n+m) communication and O(1) rounds using O(nm) computation. Then we treat secure approximate pattern matching. In this variant the matches may be approximated, i.e., have Hamming distance less than some threshold, ?. Our protocol requires O(n?) communication in O(1) rounds using O(nm) computation. Third, we have secure pattern matching with hidden pattern length. Here, the length, m, of Bob's pattern remains a secret. Our protocol requires O(n+M) communication in O(1) rounds using O(n+M) computation, where M is an upper bound on m. Finally, we have secure pattern matching with hidden text length. Finally, in this variant the length, n, of Alice's text remains a secret. Our protocol requires O(N+m) communication in O(1) rounds using O(N+m) computation, where N is an upper bound on n.

Recently, there has been a growing focus in solving approximate pattern matching problems in the streaming model. Of particular interest are the pattern matching with k-mismatches (KMM) problem and the pattern matching with w-wildcards (PMWC) problem. Motivated by reductions from these problems in the streaming model to the dictionary matching problem, this paper focuses on designing algorithms for the dictionary matching problem in the multi-stream model where there are several independent streams of data (as opposed to just one in the streaming model), and the memory complexity of an algorithm is expressed using two quantities: (1) a read-only shared memory storage area which is shared among all the streams, and (2) local stream memory that each stream stores separately. In the dictionary matching problem in the multi-stream model the goal is to preprocess a dictionary D={P_1,P_2,...,P_d} of d=|D| patterns (strings with maximum length m over alphabet Sigma) into a data structure stored in shared memory, so that given multiple independent streaming texts (where characters arrive one at a time) the algorithm reports occurrences of patterns from D in each one of the texts as soon as they appear. We design two efficient algorithms for the dictionary matching problem in the multi-stream model. The first algorithm works when all the patterns in D have the same length m and costs O(d log m) words in shared memory, O(log m log d) words in stream memory, and O(log m) time per character. The second algorithm works for general D, but the time cost per character becomes O(log m+log d log log d). We also demonstrate the usefulness of our first algorithm in solving both the KMM problem and PMWC problem in the streaming model. In particular, we obtain the first almost optimal (up to poly-log factors) algorithm for the PMWC problem in the streaming model. We also design a new algorithm for the KMM problem in the streaming model that, up to poly-log factors, has the same bounds as the most recent results that use different techniques. Moreover, for most inputs, our algorithm for KMM is significantly faster on average.

Efficient protocols for private wildcards pattern matching

In a 2-dimensional (2D) pattern matching problem, the text is arranged as a matrix 𝖬[1..n, 1..n] and consists of N = n × n symbols drawn from alphabet set Σ of size σ. The query consists of a m × m square matrix 𝖯[1..m, 1..m] drawn from the same alphabet set Σ and the task is to find all the locations in 𝖬 where 𝖯 appears as a (contiguous) submatrix. The patterns can be of any size, but as long as they are square in shape data structures like suffix trees and suffix array exist [Raffaele Giancarlo, 1995; Dong Kyue Kim et al., 1998] for the task of efficient pattern matching. These are essentially 2D counterparts of classic suffix trees and arrays known for traditional 1-dimensional (1D) pattern matching. They work based on linearization of 2D suffixes which would preserve the prefix match property (i.e., every pattern match is a prefix of some suffix). The main limitation of the suffix trees and the suffix arrays (in 1D) was their space utilization of O(N log N) bits, where N is the size of the text. This was suboptimal compared to Nlog σ bits of space, which is information theoretic optimal for the text. With the advent of the field of succinct/compressed data structures, it was possible to develop compressed variants of suffix trees and array based on Burrows-Wheeler Tansform and LF-mapping (or Φ function) [Roberto Grossi and Jeffrey Scott Vitter, 2005; Paolo Ferragina and Giovanni Manzini, 2005; Kunihiko Sadakane, 2007]. These data structures indeed achieve O(N log σ) bits of space or better. This gives rise to the question: analogous to 1D case, can we design a succinct or compressed index for 2D pattern matching? Can there be a 2D compressed suffix tree? Are there analogues of Burrows-Wheeler Transform or LF-mapping? The problem has been acknowledged for over a decade now and there have been a few attempts at applying Φ function [Ankur Gupta, 2004] and achieving entropy based compression [Veli Mäkinen and Gonzalo Navarro, 2008]. However, achieving the complexity breakthrough akin to 1D case has yet to be found. In this paper, we still do not know how to answer suffix array queries in O(N log σ) bits of space - which would have led to efficient pattern matching. However, for the first time, we show an interesting result that it is indeed possible to compute inverse suffix array (ISA) queries in near compact space in O(polylog n) time. Our 2D succinct text index design is based on two 1D compressed suffix trees and it takes O(N log log N + N logσ) bits of space which is much smaller than its naive design that takes O(N log N) bits. Although the main problem is still evasive, this index gives a hope on the existence of a full 2D succinct index with all functionalities similar to that of 1D case.

패턴 매칭(Pattern Matching)은 네트워크 침입방지 시스템에서 가장 중요한 부분의 하나며 많은 연산을 필요로 한다. 날로 증가되는 많은 수의 공격 패턴을 다루기 위해, 네트워크 침입방지 시스템에서는 회선 속도로 들어오는 패킷을 처리 할 수 있는 다중 패턴 매칭 방법이 필수적이다. 본 논문에서는 현재 많이 사용되고있는 네트워크 침입방지 및 탐지 시스템인 Snort와 이것의 패턴 매칭 특성을 분석한다. 침입방지 시스템을 위한 패턴 매칭 방법은 다양한 길이를 갖는 많은 수의 패턴과 대소문자 구분 없는 패턴 매칭을 효과적으로 다룰 수 있어야 한다. 또한 여러 개의 입력 문자들을 동시에 처리 할 수 있어야 한다. 본 논문에서 Shift-OR 패턴 매칭 알고리즘에 기반을 둔 다중 패턴 매칭 하드웨어 가속기를 제시하고 여러 가지 가정 하에서 성능 측정을 하였다. 성능 측정에 따르면 제시된 하드웨어 가속기는 현재 Snort에서 사용되는 가장 빠른 소프트웨어 다중 패턴 매칭 보다 80배 이상 빠를 수 있다. Pattern matching is one of critical parts of Network Intrusion Prevention Systems (NIPS) and computationally intensive. To handle a large number of attack signature fattens increasing everyday, a network intrusion prevention system requires a multi pattern matching method that can meet the line speed of packet transfer. In this paper, we analyze Snort, a widely used open source network intrusion prevention/detection system, and its pattern matching characteristics. A multi pattern matching method for NIPS should efficiently handle a large number of patterns with a wide range of pattern lengths and case insensitive patterns matches. It should also be able to process multiple input characters in parallel. We propose a multi pattern matching hardware accelerator based on Shift-OR pattern matching algorithm. We evaluate the performance of the pattern matching accelerator under various assumptions. The performance evaluation shows that the pattern matching accelerator can be more than 80 times faster than the fastest software multi-pattern matching method used in Snort.

Pattern matching with wildcards using words of shorter length

Generation of Pattern-Matching Algorithms by Extended Regular Expressions

Pattern Matching in Compressed Texts and Images surveys and appraises techniques for pattern matching in compressed text and images. Normally compressed data needs to be decompressed before it is processed. If however the compression has been done in the right way, it is often possible to search the data without having to decompress it, or, at least, only partially decompress it. The problem can be divided into lossless and lossy compression methods, and then in each of these cases the pattern matching can be either exact or inexact. Much work has been reported in the literature on techniques for all of these cases. It includes algorithms that are suitable for pattern matching for various compression methods, and compression methods designed specifically for pattern matching. This monograph provides a survey of this work while also identifying the important relationship between pattern matching and compression, and proposing some performance measures for compressed pattern matching algorithms. Pattern Matching in Compressed Texts and Images is an excellent reference text for anyone who has an interest in the problem of searching compressed text and images. It concludes with a particularly insightful section on the ideas and research directions that are likely to occupy researchers in this field in the short and long term.

Pattern matching in Huffman encoded texts

Parameterized matching on non-linear structures

With the wide adoption of internet into our everyday lives, internet security becomes an important issue. Intrusion detection at the network level is an effective way of stopping malicious attacks at the source and preventing viruses and worms from wide spreading. The key component in a successful network intrusion detection system is a high performance pattern matching engine that can uncover the malicious activities in real time. In this paper, we propose a highly parallel, scalable hardware based network intrusion detection system, that can handle variable pattern length efficiently and effectively. Pattern matching for a packet is completed in O(N log M) time where N is the size of the packet and M is the longest pattern length. Implementation is done on a standard off-the-shelf field-programmable gate array. Comparison with the other techniques shows promising results.

The field of stringology studies algorithms and data structures used for processing strings efficiently. The goal of this thesis is to investigate 2-dimensional (2D) variants of some fundamental string problems, including \textit{Exact Pattern Matching} and \textit{Longest Common Substring}. In the 2D pattern matching problem, we are given a matrix $\M[1\dd n,1\dd n]$ that consists of $N = n \times n$ symbols drawn from an alphabet $\Sigma$ of size $\sigma$. The query consists of a $ m \times m$ square matrix $\PP[1\dd m, 1\dd m]$ drawn from the same alphabet, and the task is to find all the locations of $\PP$ in $\M$. For such square patterns, data structures such as suffix trees and suffix arrays exist for the task of efficient pattern matching. However, a suffix tree occupies $O(N \log N)$ bits, which is significantly more than that of the original text's size of $N\log \sigma$ bits. Therefore, the design of compressed data structures, that supports pattern matching queries efficiently and occupies space close to the original text's size, is imperative. In this thesis, we show an interesting result by designing a compact text index of size $O(N \log\log N + N \log\sigma)$ bits that at least supports efficient inverse suffix array queries. Although, the question of designing a compressed text index that would lead to efficient pattern matching is still evasive, this index gives a hope on the existence of a full 2D compressed text index with all functionalities similar to that of 1D case. On the other hand, the Longest Common 2D substring problem consists of two 2D strings (matrices), and the task is to report the size of the longest common 2D substring (submatrix) of these 2D strings. It is interesting to know if there exists a sublinear-time algorithm for solving this task. We answer this question positively by presenting a sublinear-time \textit{quantum} algorithm. In addition to this, we prove that any quantum algorithm requires at least $\tilde{\Omega}(N^{2/3})$ time to solve this problem.

Pattern matching is an important task, which is widely used in many fields, such as information retrieval and bioinformatics. Recently, a much more flexible pattern matching problem with wildcards has been proposed. Chen et al. introduced local constraints, global constraints and the one-off condition into the task of pattern matching, and the most representative algorithm SAIL was designed. However, the performance of SAIL is not analyzed well, which affects its application. Therefore, this paper analyzes the performance of SAIL in-depth, and discovers that the matching result is closely related to the features of patterns. Meanwhile, the completeness of SAIL in the pattern matching with no-recurring characters is proved, and an improved algorithm, named RSAIL, is proposed for pattern matching with recurring tail characters. Extensive experiments demonstrate that RSAIL improves the number of matches by 2.2% compared to SAIL.

A fast algorithm for order-preserving pattern matching