Adaptive Non-linear Pattern Matching Automata

Efficient pattern matching is fundamental for practical term rewrite engines. By preprocessing the given patterns into a finite deterministic automaton the matching patterns can be decided in a single traversal of the relevant parts of the input term. Most automaton-based techniques are restricted to linear patterns, where each variable occurs at most once, and require an additional post-processing step to check so-called variable consistency. However, we can show that interleaving the variable consistency and pattern matching phases can reduce the number of required steps to find a match all matches. Therefore, we take the existing adaptive pattern matching automata as introduced by Sekar et al and extend it these with consistency checks. We prove that the resulting deterministic pattern matching automaton is correct, and show that its evaluation depth is can be shorter than two-phase approaches.

In a 2-dimensional (2D) pattern matching problem, the text is arranged as a matrix 𝖬[1..n, 1..n] and consists of N = n × n symbols drawn from alphabet set Σ of size σ. The query consists of a m × m square matrix 𝖯[1..m, 1..m] drawn from the same alphabet set Σ and the task is to find all the locations in 𝖬 where 𝖯 appears as a (contiguous) submatrix. The patterns can be of any size, but as long as they are square in shape data structures like suffix trees and suffix array exist [Raffaele Giancarlo, 1995; Dong Kyue Kim et al., 1998] for the task of efficient pattern matching. These are essentially 2D counterparts of classic suffix trees and arrays known for traditional 1-dimensional (1D) pattern matching. They work based on linearization of 2D suffixes which would preserve the prefix match property (i.e., every pattern match is a prefix of some suffix). The main limitation of the suffix trees and the suffix arrays (in 1D) was their space utilization of O(N log N) bits, where N is the size of the text. This was suboptimal compared to Nlog σ bits of space, which is information theoretic optimal for the text. With the advent of the field of succinct/compressed data structures, it was possible to develop compressed variants of suffix trees and array based on Burrows-Wheeler Tansform and LF-mapping (or Φ function) [Roberto Grossi and Jeffrey Scott Vitter, 2005; Paolo Ferragina and Giovanni Manzini, 2005; Kunihiko Sadakane, 2007]. These data structures indeed achieve O(N log σ) bits of space or better. This gives rise to the question: analogous to 1D case, can we design a succinct or compressed index for 2D pattern matching? Can there be a 2D compressed suffix tree? Are there analogues of Burrows-Wheeler Transform or LF-mapping? The problem has been acknowledged for over a decade now and there have been a few attempts at applying Φ function [Ankur Gupta, 2004] and achieving entropy based compression [Veli Mäkinen and Gonzalo Navarro, 2008]. However, achieving the complexity breakthrough akin to 1D case has yet to be found. In this paper, we still do not know how to answer suffix array queries in O(N log σ) bits of space - which would have led to efficient pattern matching. However, for the first time, we show an interesting result that it is indeed possible to compute inverse suffix array (ISA) queries in near compact space in O(polylog n) time. Our 2D succinct text index design is based on two 1D compressed suffix trees and it takes O(N log log N + N logσ) bits of space which is much smaller than its naive design that takes O(N log N) bits. Although the main problem is still evasive, this index gives a hope on the existence of a full 2D succinct index with all functionalities similar to that of 1D case.

The field of stringology studies algorithms and data structures used for processing strings efficiently. The goal of this thesis is to investigate 2-dimensional (2D) variants of some fundamental string problems, including \textit{Exact Pattern Matching} and \textit{Longest Common Substring}. In the 2D pattern matching problem, we are given a matrix $\M[1\dd n,1\dd n]$ that consists of $N = n \times n$ symbols drawn from an alphabet $\Sigma$ of size $\sigma$. The query consists of a $ m \times m$ square matrix $\PP[1\dd m, 1\dd m]$ drawn from the same alphabet, and the task is to find all the locations of $\PP$ in $\M$. For such square patterns, data structures such as suffix trees and suffix arrays exist for the task of efficient pattern matching. However, a suffix tree occupies $O(N \log N)$ bits, which is significantly more than that of the original text's size of $N\log \sigma$ bits. Therefore, the design of compressed data structures, that supports pattern matching queries efficiently and occupies space close to the original text's size, is imperative. In this thesis, we show an interesting result by designing a compact text index of size $O(N \log\log N + N \log\sigma)$ bits that at least supports efficient inverse suffix array queries. Although, the question of designing a compressed text index that would lead to efficient pattern matching is still evasive, this index gives a hope on the existence of a full 2D compressed text index with all functionalities similar to that of 1D case. On the other hand, the Longest Common 2D substring problem consists of two 2D strings (matrices), and the task is to report the size of the longest common 2D substring (submatrix) of these 2D strings. It is interesting to know if there exists a sublinear-time algorithm for solving this task. We answer this question positively by presenting a sublinear-time \textit{quantum} algorithm. In addition to this, we prove that any quantum algorithm requires at least $\tilde{\Omega}(N^{2/3})$ time to solve this problem.

Counting problems, determining the number of possible states of a large system under certain constraints, play an important role in many areas of science. They naturally arise for complex disordered systems in physics and chemistry, in mathematical graph theory, and in computer science. Counting problems, however, are among the hardest problems to access computationally. Here, we suggest a novel method to access a benchmark counting problem, finding chromatic polynomials of graphs. We develop a vertex-oriented symbolic pattern matching algorithm that exploits the equivalence between the chromatic polynomial and the zero-temperature partition function of the Potts antiferromagnet on the same graph. Implementing this bottom-up algorithm using appropriate computer algebra, the new method outperforms standard top-down methods by several orders of magnitude, already for moderately sized graphs. As a first application, we compute chromatic polynomials of samples of the simple cubic lattice, for the first time computationally accessing three-dimensional lattices of physical relevance. The method offers straightforward generalizations to several other counting problems.

Many network security applications such as Intrusion Detection System (IDS), Firewall and Data Loss Prevention System (DLPS) are based on deep packet inspection, in this packets header as well as payload of the packets are checked with predefined attack signature to identify whether it contains malicious traffic or not. To perform this checking different pattern matching methods are used by NIDS. The most popular method to implement pattern matching is to use of Finite Automata (FA). Generally, regular expressions are used to represent most of the attack signatures defined by NIDS. They are implemented using finite automata, which takes the payload of packet as input string. However, existing approaches of Finite Automata (FA), both deterministic finite automata (DFA) and non-deterministic finite automata (NFA) for pattern matching are having their own advantages and some drawbacks. The DFA based pattern matching methods are fast enough but require more memory. However, NFA based pattern matching methods are comparatively takes less memory but the speed of matching is very slow, to overcome these drawbacks of finite automata there are many approaches have been proposed. This paper discuses comparative study of some Finite Automata (FA) based techniques for pattern matching in network intrusion detection system (NIDS).

Efficient pattern matching in compressed text via bit-parallel word-based encoding

Intrusion detection systems are treated as vital elements of protective measures to computer systems and networks from abuse. The drastic increase in network speed and detection workloads necessitates the need for highly efficient Network Intrusion Detection Systems(NIDS). Since most NIDSs need to check for a large number of known attack patterns in every packet, pattern matching becomes the most significant part of signature-based NIDSs in terms of processing and memory resources. To support segmentation of network traffic and to detect fragmented attacks, we propose a method which performs both 'partial' and 'full' pattern matching using the data structure CDAWG (Compact Direct Acyclic Word Graph). In the present work, we designed and implemented an efficient string matching algorithm using CDAWG structure. Experimental results show that this algorithm is 2.5 times faster than the currently used Aho-Corasick algorithm.

Two equivalence relations for finite acyclic automata are considered and minimization algorithms for these automata are described. From these algorithms we develop algorithms for deciding identity of terms, matching of subterms, and pattern matching in terms, which allow for commutativity and associativity of operations.

In the recent years stream time series data management has become an important research area due to its wide variety of real life applications. In this paper, we propose a technique for similarity matching between staiic/dynamic patterns and stream time-series image data. In order to perform effective retrieval we propose a multi scale segment median approximation representation for stream time-series image data, which can be computed with respect to change in temporal events and hence is suitable for cooperating with temporal behavior. In addition, we propose an efficient pruning algorithm over the multiscale representation. Mainly, the pruning scheme is to reduce the search space and to retrieve the similar patterns effectively. Experiments carried out in this work show that the techniques for stream time series image data representation and the level by level pruning scheme can efficiently generate all candidate patterns and perform similarity matching over stream time series image datasets and performs well compared to the existing methods such as multiscale wavelet.

Generation of Pattern-Matching Algorithms by Extended Regular Expressions

Subtree matching is an important problem in Computer Science on which a number of tasks, such as mechanical theorem proving, term-rewriting, symbolic computation and nonprocedural programming languages are based on. A systematic approach to the construction of subtree pattern matchers by deterministic pushdown automata, which read subject trees in prefix and postfix notation, is presented. The method is analogous to the construction of string pattern matchers: for a given pattern, a nondeterministic pushdown automaton is created and is then determinised. In addition, it is shown that the size of the resulting deterministic pushdown automata directly corresponds to the size of the existing string pattern matchers based on finite automata.

Signature-based security tools such as network intrusion detection systems, anti-virus scanners, filters against network worms and other similar systems perform in real time computation-intensive task of multi-pattern string matching against tens of thousands or even millions of predefined malicious patterns. Due to rising traffic rates, increasing number and sophistication of attacks and the collapse of Moore's law for sequential processing, traditional software solutions can no longer meet the high requirements of today’s security challenges. Therefore, designers pay more attention to hardware approaches to accelerate pattern matching. The reconfigurable devices based on Field Programmable Gate Arrays (FPGA) combining the flexibility of software and the near-ASIC performance, have become increasingly popular for this purpose. The state-of-the-art solutions made in this area around the world were analyzed. There are three main approaches to fulfill the pattern matching using FPGA. The techniques (and underlying technologies) of these approaches are: content addressable memory (based on digital comparators), Bloom filter (based on hash-functions) and Aho-Corasick algorithm (based on finite automata). But none of them shows clear advantages over others. In this article, we propose a set of methods to increase the effectiveness of reconfigurable security tools by synthesizing optimal recognition modules that maximize the benefits of each approach. The Parallel Combination Method divides a set of patterns between several matching blocks that use different approaches to better fit each of them. The Sequential Cascading Method processes patterns in parts: if the first fragment does not match, the rest can be ignored. The Vertical Join Method couples together different approaches or techniques in a single unit to provide higher efficiency of the resulting device. The optimization procedure maximizes efficiency gains for each method. The methods and methodologies presented in this study will allow developers to create more efficient reconfigurable tools for information security systems.

State complexity of pattern matching in regular languages

Computer Science and Engineering have given us the field of automata theory, one of the largest areas that is concerned with the efficiency of an algorithm in solving a problem on a computational model. Various classes of formal languages are represented using Chomsky hierarchy. These languages are described as a set of specific strings over a given alphabet and can be described using state or transition diagrams. The state/transition diagram for regular languages is called a finite automaton which is used in compiler design for recognition of tokens. Other applications of finite automata include pattern matching, speech and text processing, CPU machine operations, etc. The construction of finite automata is a complicated and challenging process as there is no fixed mathematical approach that exists for designing Deterministic Finite Automata (DFA) and handling the validations for acceptance or rejection of strings. Consequently, it is difficult to represent the DFA’s transition table and graph. Novel learners in the field of theoretical computer science often feel difficulty in designing of DFA The present paper proposes an algorithm for designing of deterministic finite automata (DFA) for a regular language with a given prefix. The proposed method further aims to simplify the lexical analysis process of compiler design.

This paper defines a challenging problem of pattern matching between a pattern P and a text T, with wildcards and length constraints, and designs an efficient algorithm to return each pattern occurrence in an online manner. In this pattern matching problem, the user can specify the constraints on the number of wildcards between each two consecutive letters of P and the constraints on the length of each matching substring in T. We design a complete algorithm, SAIL that returns each matching substring of P in T as soon as it appears in T in an O(n+klmg) time with an O(lm) space overhead, where n is the length of T, k is the frequency of P's last letter occurring in T, l is the user-specified maximum length for each matching substring, m is the length of P, and g is the maximum difference between the user-specified maximum and minimum numbers of wildcards allowed between two consecutive letters in P.