A novel algorithm for the conversion of shuffle regular expressions into non-deterministic finite automata

Many difficult open problems in theoretical computer science center around nondeterminism. We study the fundamental problem of converting a given deterministic finite automaton (DFA) to a minimal nondeterministic finite automaton (NFA). Despite extensive work on finite automata, this fundamental problem has remained open. Recently, in [Ji91] we studied this problem and showed that this (and related) problems are computationally hard. Here we study the restriction of this problem to the case when the input DFA is over a one-letter alphabet. Even in this restricted case the problem is computationally hard even though our evidence of hardness is different from (and is weaker than) the standard ones such as NP-hardness. Let A → B denote the problem of converting a finite automaton of type A to a minimal finite automaton of type B. Our main result is that DFA → NFA, when the input is a unary cyclic DFA (a DFA whose graph is a simple cycle), is in NP but not in P unless NP ⊑ DTIME(nO(log n)). Our work was also motivated by the problem of finding structurally simple ‘normal forms’ of NFA's over a unary alphabet. We present some normal forms for minimal NFA's over a unary alphabet and present an application to lower bounds on the size complexity of an NFA. In fact, the normal form result is used in a nontrivial manner to show the NP membership result stated above.

Regular Expressions (RegExes) are widely used in various applications to identify strings of text. Their flexibility, however, increases the complexity of the detection system and often limits the detection speed as well as the total number of RegExes that can be detected using limited resources. The two classical detection methods, Deterministic Finite Automaton (DFA) and Non-Deterministic Finite Automaton (NFA), have the potential problems of prohibitively large memory requirements and a large number of concurrent operations, respectively. Although recent schemes addressing these problems to improve DFA and NFA are promising, they are inherently limited by their scalability, since they follow the state transition model in DFA and NFA, where the state transitions occur per each character of the input. We recently proposed a scalable RegEx detection system called Lookahead Finite Automata (LaFA) to solve these problems with three novel ideas: 1. Provide specialized and optimized detection modules to increase resource utilizations. 2. Systematically reordering the RegEx detection sequence to reduce number of concurrent operations. 3. Sharing states among automata for different RegExes to reduce resource requirements. In this paper, we propose an efficient hardware architecture and prototype design implementation based on LaFA. Our proof-of-concept prototype design is built on a fraction of a single commodity Field Programmable Gate Array (FPGA) chip and can accommodate up to twenty-five thousand (25k) RegExes. Using only 7% of the logic area and 25% of the memory on a Xilinx Virtex-4 FX100, the prototype design can achieve 2-Gbps (gigabits-per-second) detection throughput with only one detection engine. We estimate that 34-Gbps detection throughput can be achieved if the entire resources of a state-of-the-art FPGA chip are used to implement multiple detection engines.

Many network security applications in today's networks are based on deep packet inspection, checking not only the header portion but also the payload portion of a packet. For example, traffic monitoring, layer-7 filtering, and network intrusion detection all require an accurate analysis of packet content in search for predefined patterns to identify specific classes of applications, viruses, attack signatures, etc. Regular expressions are often used to represent such patterns. They are implemented using finite automata, which take the payload of a packet as an input string. However, existing approaches, both non-deterministic finite automata (NFA) and deterministic finite automata (DFA), do not deal with compressed traffic, which becomes more and more popular in HTTP applications. In this paper, we propose an efficient algorithm for regular expression matching to implement deep packet inspection on compressed traffic. Based on the observations of DFA, we design a scheme to skip most of the matching process in the compressed parts of traffic. To the best of our knowledge, this is the first effort to design an efficient regular expression matching on compressed traffic. We evaluate our algorithm using rule sets provided by Snort, a popular open-source intrusion detection system. The evaluation results show that our approach can reduce the number of state access in the DFA significantly.

Meyer and Fischer b][MF] proved that nondeterministic finite automata (NFA) can be exponentially more concise than deterministic finite automata (DFA) in their representations of regular languages. Several variants of that basic finite state machine model are now being used to analyze parallelism and to build real-time software systems [HL+]. Even though these variants can sometimes represent regular languages in a more concise manner than NFA, the underlying models fundamentally differ from NFA in how they operate. Degree automata [W] (DA), however, differ from NFA only in their acceptance criteria and accept only regular languages. We show here that DA are also exponentially more concise than NFA on some sequences of regular languages. We also show that the conciseness of probabilistic automata [R] with isolated cutpoints can be unbounded over DA and, concurrently, i.e., over the same sequence of languages, those DA can be exponentially more concise than NFA.

The equivalence of non-deterministic finite automata (NFA ), deterministic finite automata (DFA), regular expressions (RE), and parallel finite automata (PFA) with augmented regular expressions (PRE) is used to relate parallel finite automata to the shuffle operation on regular languages. Our goal is to capitalize on the equivalence of regular expressions and augmented regular expressions to increase parallelization efficiency. We use the existing automated transformation algorithms for the various finite automata listed above, as well as one of our own, to inform a transformation from augmented regular expressions to minimized deterministic finite automata (DFAmin). We then discuss the serialization of augmented regular expressions, and present a brief summary of potential applications, related research, and future directions

Chapter 1 - Formal Requirements and Finite Automata Overview

Many network security applications such as Intrusion Detection System (IDS), Firewall and Data Loss Prevention System (DLPS) are based on deep packet inspection, in this packets header as well as payload of the packets are checked with predefined attack signature to identify whether it contains malicious traffic or not. To perform this checking different pattern matching methods are used by NIDS. The most popular method to implement pattern matching is to use of Finite Automata (FA). Generally, regular expressions are used to represent most of the attack signatures defined by NIDS. They are implemented using finite automata, which takes the payload of packet as input string. However, existing approaches of Finite Automata (FA), both deterministic finite automata (DFA) and non-deterministic finite automata (NFA) for pattern matching are having their own advantages and some drawbacks. The DFA based pattern matching methods are fast enough but require more memory. However, NFA based pattern matching methods are comparatively takes less memory but the speed of matching is very slow, to overcome these drawbacks of finite automata there are many approaches have been proposed. This paper discuses comparative study of some Finite Automata (FA) based techniques for pattern matching in network intrusion detection system (NIDS).

There is an increasing demand for network devices to perform deep packet inspection (DPI) in order to enhance network security. In DPI, the packet payload is compared against a set of predefined patterns that can be specified using regular expressions (regexes). It is well-known that mapping regexes to deterministic finite automaton (DFA) may suffer from the state explosion problem. Through observation, we attribute DFA explosion to the necessity of remembering matching history. In this paper, we investigate how to manage matching history efficiently and propose an extended DFA approach for regex matching called fcq-FA, which can make a memory size reduction of about 1,000 times with a fully automated approach. In fcq-FA, we use pipeline queues and counters to help record the matching history. Hence, state explosion caused by Kleene closure and length restriction can be completely avoided. Furthermore, it achieves a fully automated signature compilation with polynomial running time and space. The equivalence between fcq-FA and the traditional DFA is guaranteed by a strict theoretical proof, which means fcq-FA can process all the regexes supported by the traditional DFA.

We introduce E-equivalence, which is a straightforward generalization of almost-equivalence. While almost-equivalence asks for ordinary equivalence up to a finite number of exceptions, in E-equivalence these exceptions or errors must belong to a (regular) set E. The computational complexity of minimization problems and their variants w.r.t. almost- and E-equivalence are studied. Roughly speaking, whenever nondeterministic finite automata (NFAs) are involved, most minimization problems, and their equivalence problems they are based on, become PSPACE-complete, while for deterministic finite automata (DFAs) the situation is more subtle. For instance, hyper-minimizing DFAs is NL-complete, but E-minimizing DFA s is NP-complete, even for finite E. The obtained results nicely fit to the known ones on ordinary minimization for finite automata. Moreover, since hyper-minimal and E-minimal automata are not necessarily unique (up to isomorphism as for minimal DFAs), we consider the problem of counting the number of these minimal automata. It turns out that counting hyper-minimal DFAs can be done in FP, while counting E-minimal DFA s is #P-hard, and belongs to the counting class #·coNP.

The finite automata are mathematical models for finite state systems. More general finite automaton is the nondeterministic finite automaton (NFA) that cannot be directly used. It is usually transformed to the deterministic finite automaton (DFA) that then runs in time O(n), where n is the size of the input text. We present two main approaches to practical implementation of DFA considering CPU cache. The first approach (represented by Table Driven and Hard Coded implementations) is suitable forautomata being run very frequently, typically having cycles. The other approach is suitable for a collection of automata from which various automata are retrieved and then run. This second kind of automata are expected to be cycle-free.

Regular expression constrained sequence alignment

We give two new algorithms for constructing small nondeterministic finite automata (NFA) from regular expressions. The first constructs NFAs with ?-moves (?NFA) which are smaller than all the other ?NFAs obtained by similar constructions. Their size is at most 3/2 |?|+ 5/2, where ? is the regular expression. The second constructs NFAs. It uses ?-elimination in the ?NFAs we just introduced and builds a quotient of the well-known position automaton. Our NFA is always smaller and faster to compute than the position automaton. It uses optimally the information from the positions of a regular expression.

Intelligent and efficient grouping algorithms for large-scale regular expressions

[Formula: see text] is a binary coded unary regular language, if there exists a unary regular language [Formula: see text] such that [Formula: see text] is in [Formula: see text] if and only if the binary representation of [Formula: see text] is in [Formula: see text]. If a unary language [Formula: see text] is accepted by a minimal deterministic finite automaton (DFA) [Formula: see text] with [Formula: see text] states, then its binary coded version [Formula: see text] is regular and can be accepted by a DFA [Formula: see text] using at most [Formula: see text] states, but at least [Formula: see text] states. There are witness languages matching these upper and lower bounds exactly, for each [Formula: see text]. More precisely, if [Formula: see text] uses [Formula: see text] states in the initial segment and [Formula: see text] states in the loop, where [Formula: see text] is odd and [Formula: see text], then the minimal [Formula: see text] for [Formula: see text] consists of a preamble with at most [Formula: see text] but at least [Formula: see text] states, except for [Formula: see text] with no preamble, and a kernel with at most [Formula: see text] but at least [Formula: see text] states. Also these lower bounds are matched exactly by witness languages, for each [Formula: see text]. If the length of the loop is fixed, the size of the preamble is bounded by [Formula: see text]. The conversion in the opposite way is not always granted: there are binary regular languages the unary versions of which are not even context free. The conversion of a unary nondeterministic finite automaton (NFA) to a binary NFA uses [Formula: see text] states and introduces a binary version of Chrobak normal form.

Minimization of nondeterministic finite automata (NFA) is a hard problem (PSPACE-complete). Bisimulations are then an attractive alternative for reducing the size of NFAs, as even bisimilarity (the largest bisimulation) is almost linear using the Paige and Tarjan algorithm. NFAs obtained from regular expressions (REs) can have the number of states linear with respect to the size of the REs and conversion methods from REs to equivalent NFAs can produce NFAs without or with transitions labelled with the empty word (e-NFA). The standard conversion without e-transitions is the position automaton, \(\mathcal{A}_{pos}\). Other conversions, such as partial derivative automata (\(\mathcal{A}_{pd}\)) or follow automata (\(\mathcal{A}_{f}\)), were proven to be quotients of the position automata (by some bisimulations). Recent experimental results suggested that for REs in (normalized) star normal form the position bisimilarity almost coincide with the \(\mathcal{A}_{pd}\) automaton. Our goal is to have a better characterization of \(\mathcal{A}_{pd}\) automata and their relation with the bisimilarity of the position automata. In this paper, we consider \(\mathcal{A}_{pd}\) automata for regular expressions without Kleene star and establish under which conditions they are isomorphic to the bisimilarity of \(\mathcal{A}_{pos}\).