Compiling regular patterns to sequential machines

R. McNaughton and H. Yamada. Regular expressions and state graphs for automata. Sequential machines, Selected papers, edited by Edward F. Moore, Addison-Wesley Publishing Company, Inc., Reading, Massachusetts, Palo Alto, and London, 1964, pp. 157–174. (Reprinted from IRE transactions on electronic computers, vol. EC-9 (1960), pp. 39–47.) - Volume 32 Issue 3

The Markov chain embedding technique is commonly used to study the distribution of statistics associated with regular patterns (i.e. set of strings described by a regular expression) in random strings. In this extended abstract, we formalize the concept Markov chain embedding for random strings produced by a possibly non-stationary Markov source. A notion of memory conveyed by the states of a deterministic finite automaton is introduced. This notion is used to characterize the smallest state-space size Markov chain required to specify the distribution of the count statistic of a given regular pattern. The research finds applications in problems associated with regular patterns in random strings that demand exponentially large state spaces.

Regular expression has been widely used in network intrusion detection system to represent attack patterns due to its expressive power and flexibility. However, the traditional memory architecture suffers from the problem of memory explosion for certain types of complex regular expressions. In this paper, we propose a hierarchical state machine architecture which can significantly reduce the memory required to accommodate complex regular expression patterns. The experiments demonstrate a significant reduction in memory for the complex regular expression patterns commonly used in network intrusion detection systems.

A noticeable fraction of Algorithms papers in the last few decades improve the running time of well-known algorithms for fundamental problems by logarithmic factors. For example, the {O}(n^2) dynamic programming solution to the Longest Common Subsequence problem (LCS) was improved to O(n^2/log^{2}n) in several ways and using a variety of ingenious tricks. This line of research, also known as the art of shaving log factors, lacks a tool for proving negative results. Specifically, how can we show that it is unlikely that LCS can be solved in time O(n^2/log^3n)?\nPerhaps the only approach for such results was suggested in a recent paper of Abboud, Hansen, Vassilevska W. and Williams (STOC'16). The authors blame the hardness of shaving logs on the hardness of solving satisfiability on boolean formulas (Formula-SAT) faster than exhaustive search. They show that an O(n^2/log^{1000} n) algorithm for LCS would imply a major advance in circuit lower bounds. Whether this approach can lead to tighter barriers was unclear.\nIn this paper, we push this approach to its limit and, in particular, prove that a well-known barrier from complexity theory stands in the way for shaving five additional log factors for fundamental combinatorial problems. For LCS, regular expression pattern matching, as well as the Fréchet distance problem from Computational Geometry, we show that an O(n^2/log^{7+epsilon}{n}) runtime would imply new Formula-SAT algorithms.\nOur main result is a reduction from SAT on formulas of size s over n variables to LCS on sequences of length N=2^{n/2} * s^{1+o(1)}. Our reduction is essentially as efficient as possible, and it greatly improves the previously known reduction for LCS with N=2^{n/2} * s^c, for some c >= 100.

Generation of Pattern-Matching Algorithms by Extended Regular Expressions

Mealy machines are a better model of lexical analyzers

Self-organizing systems called wheel systems have been developed in response to the problem of finding periodic recurrences in sequences of units (observations of the behaviour of individual animals). In the introduction there are presented concepts of ethological the available mathematical methods for analysis of sequences of units, and an overview of the field of self-organizing systems. This is followed by a description of the A wheel system. Rules of two other self-organizing systems, a Q wheel system and an H wheel system are then given. Illustrations of the operation of Q, and H wheel systems are given in appendices. The A wheel system is then selected for detailed study. This begins with descriptions of several experiments which empirically demonstrate certain properties of the system, with emphasis on certain effects of the fact that it is a self-organizing system (SOS). The possibility of using the A wheel system as an adjunct to pattern recognition and simulation is then studied by development of methods of interpretation of the results (wheel tracks) of the A wheel system. Two lines of approach are developed according to what are conveniently called structural patterns and functional patterns. The latter approach is more general and was developed as an understanding grew of the limitations of the former approach. Certain concepts from automata theory are introduced to show that a certain kind of automaton, an acceptor, can be used to model the behaviour of a wheel. Also, the behaviour of a set of wheels can be modelled by an acceptor. A method is given for constructing a sequential machine which will print the input sequence to the A wheel system (i.e., the original sequence of units). It receives, on an input tape, blanks and non-blank symbols. For each blank input symbol, the machine generates a non-blank output symbol autonomously. The non-blank input symbols are printed in the output sequence and mark events when operation is interrupted. A sequential machine is obtained which simulates the singing of morning twilight song by an individual of the North American Wood Peewee. The sequence simulated was Record 5 of Craig (1943)*. This sequence has previously been analysed by other authors using Markov chain models and the two methods are contrasted. An input tape is then obtained which drives the same machine in such a way that the machine produces an output sequence which is Record 4 (Craig, ibid.), the sequence of phrases sung by another individual Wood Peewee. An appendix contains instructions for use of three computer programs for the implementation of A, Q, and H wheel systems. A major advantage of the A wheel system is its adaptive property enabling successive changes of pattern to be found. It can accommodate relatively short data sequences compared with Markov models, and allows construction of a deterministic sequential machine to generate the observed sequence. Input symbols to the sequential machine support a classification of output symbols as autonomous productions or non-autonomous productions. A study is mads of the non-autonomous productions of the machine which simulates Record 5 and this confirms some of the earlier findings of Craig and suggests alternate interpretations for other of Craig's findings.*The song of the wood peewee Myochanes virens Linnaeus : a study of bird music. New York State Museum Bulletin, No. 334, June, 1943, page 41.

We consider the problem of restoring regular expressions from good examples. We describe a natural learning algorithm for obtaining a “plausible” regular expression from one example. The algorithm is based on finding the longest substring which can be matched by some part of the so far obtained expression. We believe that the algorithm to a certain extent mimics humans guessing regular expressions from the same sort of examples. We show that for regular expressions of bounded length successful learning takes time linear in the length of the example, provided that the example is “good”. Under certain natural restrictions the run-time of the learning algorithm is polynomial also in unsuccessful cases. In the end we discuss the computer experiment of learning regular expressions via the described algorithm, showing that the proposed learning method is quite practical.

Introduction: Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Vitae sapien pellentesque habitant morbi tristique senectus et netus. Dignissim cras tincidunt lobortis feugiat vivamus at augue eget arcu. At risus viverra adipiscing at in. Cras semper auctor neque vitae tempus quam. Sed cras ornare arcu dui. Turpis massa sed elementum tempus. Risus commodo viverra maecenas accumsan lacus vel facilisis volutpat est.
 Objectives: Effective data management has become an absolute necessity in today's world as a direct result of the widespread adoption of electronic medical records for patients. For medicine and healthcare domains, particularly effective web healthcare providers, computer-assisted categorization of such information into functional classifications such as medical issues or illnesses may save time and efforts. However, a substantial volume of data in the healthcare domain is still unstructured.
 Methods: The utilization of unstructured text datasets is a challenge in comparison with structured text datasets. Many researchers in various domains have proposed to convert unstructured text into structured text. The text needs to be built in a fixed and aligned pattern for easy comparison, matching, and classification. In the text classification process, accuracy is one of the challenges when text is unstructured. The text with a fixed pattern is easy to classify. The regular expressions are sequences of characters with fixed patterns of text. This strength of regular expression is used in the proposed text classification. Finding patterns in unstructured text is possible with the use of sequence alignment. Basically, sequence alignment is a concept of biomedical research, but the same concept is used to align text snippets that are not perfectly similar to each other.
 Results: The sequence alignment technique detects the similarity score assigned to each potential alignment in order to choose among the numerous local alignments of sequences. The proposed local pairwise alignment method is used to get sequence alignments, which are useful to generate regular expressions.
 Conclusions: A regular expression is a string of characters used to describe a text pattern. The regular expressions are typically created manually by the domain experts. The proposed work is the automatic generation of regular expressions from aligned sequences with a bottom up approach. The generated regular expressions are used as a dataset on which various machine learning algorithms are applied for text classification.

Many network security applications such as Intrusion Detection System (IDS), Firewall and Data Loss Prevention System (DLPS) are based on deep packet inspection, in this packets header as well as payload of the packets are checked with predefined attack signature to identify whether it contains malicious traffic or not. To perform this checking different pattern matching methods are used by NIDS. The most popular method to implement pattern matching is to use of Finite Automata (FA). Generally, regular expressions are used to represent most of the attack signatures defined by NIDS. They are implemented using finite automata, which takes the payload of packet as input string. However, existing approaches of Finite Automata (FA), both deterministic finite automata (DFA) and non-deterministic finite automata (NFA) for pattern matching are having their own advantages and some drawbacks. The DFA based pattern matching methods are fast enough but require more memory. However, NFA based pattern matching methods are comparatively takes less memory but the speed of matching is very slow, to overcome these drawbacks of finite automata there are many approaches have been proposed. This paper discuses comparative study of some Finite Automata (FA) based techniques for pattern matching in network intrusion detection system (NIDS).

In this article we describe how to expand a partially dynamic reconfigurable pattern matcher for regular expressions presented in previous work by Divyasree and Rajashekar [2]. The resulting, extended, pattern matcher is fully dynamically reconfigurable. First, the design is adapted for use with parameterisable configurations, a method for Dynamic Circuit Specialization. Using parameterisable configurations allows us to achieve the same area gains as the hand crafted reconfigurable design, with the benefit that parameterisable configurations can be applied automatically. This results in a design that is more easily adaptable to specific applications and allows for an easier design exploration. Additionally, the parameterisable configuration implementation is also generated automatically, which greatly reduces the design overhead of using dynamic reconfiguration. Secondly, we propose a number of expansions to the original design to overcome several limitations in the original design that constrain the dynamic reconfigurability of the pattern matcher. We propose two different solutions to dynamically change the character that is matched in a certain block. The resulting pattern matcher, after these changes, is fully dynamically reconfigurable, all aspects of the implemented regular expression can be changed at run-time.

In this paper, we propose a systolic pattern-independent hardware regular expression matching (REM) engine which handles nested Kleene operators used in virus patterns. Pattern-independent systolic REM engines are suitable to network intrusion detection systems for quick updating of virus pattern. In the proposed engine, we introduce a compact pattern-independent NFA circuit, which can handle any small regular expression patterns, into a systolic REM engine to handle nested Kleene operators. Experimental results show that the extended engine implemented on an FPGA handles nested Kleene operators with efficient circuit size and high performance (2.17Gbps).

Deep packet inspection (DPI) is widely used in content-aware network applications such as network intrusion detection systems, traffic billing, load balancing, and government surveillance. Pattern matching is a core and critical step in DPI, which checks the payload of each packet for known signatures (patterns) in order to identify packets with certain characteristics (e.g., malicious packets that carry viruses or worms). Regular expression is the major tool for signature description due to its powerful and flexible expressive ability. However, this flexibility also brings great challenges for efficient implementation in practice. Despite of hundreds to thousands of empirical proposals, wire-speed matching for large scale regular expressions still remains a big challenge. The gap between the matching throughput and the link speed is widening with the ever-increasing network link speed and pattern scale. This survey begins with a full-scale application background of DPI and technical background of regular expression matching in order to provide a global view and essential knowledge for readers. We then analyze the challenges in regular expression matching originated from the state explosion of finite state automaton used for regular expression matching. The nature of state explosion is analyzed in details, and the state-of-the-art solutions are grouped into categories of methods to relieve state expansion and methods to avoid state explosion, suggestions are also provided for building compact and efficient automata in different scenarios. Furthermore, proposals employing parallel platforms, including field-programmable gate array, GPU, general multi-processors, and ternary content addressable memory, to accelerate the matching process are introduced and thoroughly discussed. We also provide guidelines for efficient deployment for each of these platforms.

Complex processing is a technology that obtains valuable complex events from the low value of single events, and is widely used in the network system of big data era. In order to solve the need of rich the description language of complex event and promote the efficiency of stream pattern matching, an original scheme of complex detection is proposed in this study. It adopts the regular tree pattern to describe the complex events, which can not only describe the structure but also the timing constraints of events. Based on the means of tree ordering, automaton and mapping table, the scheme combines data-parallel and task-parallel and make full use of the multi-core advantage of computers. The experimental results show that the scheme has a high performance on complex detection.

Systolic circuits have attracted considerable attention as a means of implementing parallel algorithms in areas such as linear algebra, signal processing, pattern matching, etc. A systolic circuit is composed of a number of computation cells which are connected in a regular pattern. Each cell can perform computations, store data and communicate with other cells in the circuit. We define a language to describe implementations and specifications of a class of systolic circuits whose cells compute over a commutative ring, and present a decision method to check whether or not a circuit implementation fulfils a specification. The main advantage of our approach, compared with earlier work in the field, is that the verification is performed automatically, without user interaction. We give an example of how the method may be applied to verify a convolution circuit.