Generation of Pattern-Matching Algorithms by Extended Regular Expressions

Introduction: Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Vitae sapien pellentesque habitant morbi tristique senectus et netus. Dignissim cras tincidunt lobortis feugiat vivamus at augue eget arcu. At risus viverra adipiscing at in. Cras semper auctor neque vitae tempus quam. Sed cras ornare arcu dui. Turpis massa sed elementum tempus. Risus commodo viverra maecenas accumsan lacus vel facilisis volutpat est.
 Objectives: Effective data management has become an absolute necessity in today's world as a direct result of the widespread adoption of electronic medical records for patients. For medicine and healthcare domains, particularly effective web healthcare providers, computer-assisted categorization of such information into functional classifications such as medical issues or illnesses may save time and efforts. However, a substantial volume of data in the healthcare domain is still unstructured.
 Methods: The utilization of unstructured text datasets is a challenge in comparison with structured text datasets. Many researchers in various domains have proposed to convert unstructured text into structured text. The text needs to be built in a fixed and aligned pattern for easy comparison, matching, and classification. In the text classification process, accuracy is one of the challenges when text is unstructured. The text with a fixed pattern is easy to classify. The regular expressions are sequences of characters with fixed patterns of text. This strength of regular expression is used in the proposed text classification. Finding patterns in unstructured text is possible with the use of sequence alignment. Basically, sequence alignment is a concept of biomedical research, but the same concept is used to align text snippets that are not perfectly similar to each other.
 Results: The sequence alignment technique detects the similarity score assigned to each potential alignment in order to choose among the numerous local alignments of sequences. The proposed local pairwise alignment method is used to get sequence alignments, which are useful to generate regular expressions.
 Conclusions: A regular expression is a string of characters used to describe a text pattern. The regular expressions are typically created manually by the domain experts. The proposed work is the automatic generation of regular expressions from aligned sequences with a bottom up approach. The generated regular expressions are used as a dataset on which various machine learning algorithms are applied for text classification.

Many network security applications such as Intrusion Detection System (IDS), Firewall and Data Loss Prevention System (DLPS) are based on deep packet inspection, in this packets header as well as payload of the packets are checked with predefined attack signature to identify whether it contains malicious traffic or not. To perform this checking different pattern matching methods are used by NIDS. The most popular method to implement pattern matching is to use of Finite Automata (FA). Generally, regular expressions are used to represent most of the attack signatures defined by NIDS. They are implemented using finite automata, which takes the payload of packet as input string. However, existing approaches of Finite Automata (FA), both deterministic finite automata (DFA) and non-deterministic finite automata (NFA) for pattern matching are having their own advantages and some drawbacks. The DFA based pattern matching methods are fast enough but require more memory. However, NFA based pattern matching methods are comparatively takes less memory but the speed of matching is very slow, to overcome these drawbacks of finite automata there are many approaches have been proposed. This paper discuses comparative study of some Finite Automata (FA) based techniques for pattern matching in network intrusion detection system (NIDS).

Pattern matching and regular expression matching are all the critical components for content inspection based applications. But current regular expression matching algorithms or architecture cannot provide a perfect solution for whole matching problem. In some real network security applications, exact strings are the biggest part of rule set, and the second part is simple regular expressions (Dot-Star and AND-Logic), and the other complex regular expressions only occupy a very small part. So, we propose a new hardware-based multiple simple regular expression matching architecture, called MSRM, for Dot-Star and AND-Logic regular expressions. Firstly, software compiler splits simple regular expressions into exact strings and relations. Multi-string-matching module judges whether strings match and outputs the matched ID. Based on these matched information and pre-generated RAM data, MSRM can judge whether Dot-Star and AND-Logic regular expressions are satisfied easily and quickly. Experiments with random test data and ClamAV rule set show that MSRM can achieve a high throughput of 2.1 and 2.8 Gbps using Virtex2 and Virtex4 devices respectively which is much higher than software algorithms.

Pattern matching algorithms have gained a lot of importance in computer science, primarily because they are used in various domains such as computational biology, video retrieval, intrusion detection systems, and fraud detection. Finding one or more patterns in a given text is known as pattern matching. Two important things that are used to judge how well exact pattern matching algorithms work are the total number of attempts and the character comparisons that are made during the matching process. The primary focus of our proposed method is reducing the size of both components wherever possible. Despite sprinting, hash-based pattern matching algorithms may have hash collisions. The Efficient Hashing Method (EHM) algorithm is improved in this research. Despite the EHM algorithm’s effectiveness, it takes a lot of time in the preprocessing phase, and some hash collisions are generated. A novel hashing method has been proposed, which has reduced the preprocessing time and hash collision of the EHM algorithm. We devised the Hashing Approach for Pattern Matching (HAPM) algorithm by taking the best parts of the EHM and Quick Search (QS) algorithms and adding a way to avoid hash collisions. The preprocessing step of this algorithm combines the bad character table from the QS algorithm, the hashing strategy from the EHM algorithm, and the collision-reducing mechanism. To analyze the performance of our HAPM algorithm, we have used three types of datasets: E. coli, DNA sequences, and protein sequences. We looked at six algorithms discussed in the literature and compared our proposed method. The Hash-q with Unique FNG (HqUF) algorithm was only compared with E. coli and DNA datasets because it creates unique bits for DNA sequences. Our proposed HAPM algorithm also overcomes the problems of the HqUF algorithm. The new method beats older ones regarding average runtime, number of attempts, and character comparisons for long and short text patterns, though it did worse on some short patterns.

We consider the problem of restoring regular expressions from good examples. We describe a natural learning algorithm for obtaining a “plausible” regular expression from one example. The algorithm is based on finding the longest substring which can be matched by some part of the so far obtained expression. We believe that the algorithm to a certain extent mimics humans guessing regular expressions from the same sort of examples. We show that for regular expressions of bounded length successful learning takes time linear in the length of the example, provided that the example is “good”. Under certain natural restrictions the run-time of the learning algorithm is polynomial also in unsuccessful cases. In the end we discuss the computer experiment of learning regular expressions via the described algorithm, showing that the proposed learning method is quite practical.

Deep packet inspection (DPI) is widely used in content-aware network applications such as network intrusion detection systems, traffic billing, load balancing, and government surveillance. Pattern matching is a core and critical step in DPI, which checks the payload of each packet for known signatures (patterns) in order to identify packets with certain characteristics (e.g., malicious packets that carry viruses or worms). Regular expression is the major tool for signature description due to its powerful and flexible expressive ability. However, this flexibility also brings great challenges for efficient implementation in practice. Despite of hundreds to thousands of empirical proposals, wire-speed matching for large scale regular expressions still remains a big challenge. The gap between the matching throughput and the link speed is widening with the ever-increasing network link speed and pattern scale. This survey begins with a full-scale application background of DPI and technical background of regular expression matching in order to provide a global view and essential knowledge for readers. We then analyze the challenges in regular expression matching originated from the state explosion of finite state automaton used for regular expression matching. The nature of state explosion is analyzed in details, and the state-of-the-art solutions are grouped into categories of methods to relieve state expansion and methods to avoid state explosion, suggestions are also provided for building compact and efficient automata in different scenarios. Furthermore, proposals employing parallel platforms, including field-programmable gate array, GPU, general multi-processors, and ternary content addressable memory, to accelerate the matching process are introduced and thoroughly discussed. We also provide guidelines for efficient deployment for each of these platforms.

Intrusion Detection System has become the important part of the computer and network security because it can effectively compensate for the lack of network security measures. However, the intrusion detection technology relies heavily on pattern matching algorithms, as the choice of pattern matching algorithms directly affect the detection rate, so it is more important. First the paper is detailed analysis the characteristics of the four kinds of single pattern matching which based on prefix search, and then it tests by the number of different pattern strings. The experimental results show that the KMP algorithm can improve effectively the speed of pattern matching. So the application of KMP algorithm can improve the efficiency of intrusion detection.

In a 2-dimensional (2D) pattern matching problem, the text is arranged as a matrix 𝖬[1..n, 1..n] and consists of N = n × n symbols drawn from alphabet set Σ of size σ. The query consists of a m × m square matrix 𝖯[1..m, 1..m] drawn from the same alphabet set Σ and the task is to find all the locations in 𝖬 where 𝖯 appears as a (contiguous) submatrix. The patterns can be of any size, but as long as they are square in shape data structures like suffix trees and suffix array exist [Raffaele Giancarlo, 1995; Dong Kyue Kim et al., 1998] for the task of efficient pattern matching. These are essentially 2D counterparts of classic suffix trees and arrays known for traditional 1-dimensional (1D) pattern matching. They work based on linearization of 2D suffixes which would preserve the prefix match property (i.e., every pattern match is a prefix of some suffix). The main limitation of the suffix trees and the suffix arrays (in 1D) was their space utilization of O(N log N) bits, where N is the size of the text. This was suboptimal compared to Nlog σ bits of space, which is information theoretic optimal for the text. With the advent of the field of succinct/compressed data structures, it was possible to develop compressed variants of suffix trees and array based on Burrows-Wheeler Tansform and LF-mapping (or Φ function) [Roberto Grossi and Jeffrey Scott Vitter, 2005; Paolo Ferragina and Giovanni Manzini, 2005; Kunihiko Sadakane, 2007]. These data structures indeed achieve O(N log σ) bits of space or better. This gives rise to the question: analogous to 1D case, can we design a succinct or compressed index for 2D pattern matching? Can there be a 2D compressed suffix tree? Are there analogues of Burrows-Wheeler Transform or LF-mapping? The problem has been acknowledged for over a decade now and there have been a few attempts at applying Φ function [Ankur Gupta, 2004] and achieving entropy based compression [Veli Mäkinen and Gonzalo Navarro, 2008]. However, achieving the complexity breakthrough akin to 1D case has yet to be found. In this paper, we still do not know how to answer suffix array queries in O(N log σ) bits of space - which would have led to efficient pattern matching. However, for the first time, we show an interesting result that it is indeed possible to compute inverse suffix array (ISA) queries in near compact space in O(polylog n) time. Our 2D succinct text index design is based on two 1D compressed suffix trees and it takes O(N log log N + N logσ) bits of space which is much smaller than its naive design that takes O(N log N) bits. Although the main problem is still evasive, this index gives a hope on the existence of a full 2D succinct index with all functionalities similar to that of 1D case.

In this article we describe how to expand a partially dynamic reconfigurable pattern matcher for regular expressions presented in previous work by Divyasree and Rajashekar [2]. The resulting, extended, pattern matcher is fully dynamically reconfigurable. First, the design is adapted for use with parameterisable configurations, a method for Dynamic Circuit Specialization. Using parameterisable configurations allows us to achieve the same area gains as the hand crafted reconfigurable design, with the benefit that parameterisable configurations can be applied automatically. This results in a design that is more easily adaptable to specific applications and allows for an easier design exploration. Additionally, the parameterisable configuration implementation is also generated automatically, which greatly reduces the design overhead of using dynamic reconfiguration. Secondly, we propose a number of expansions to the original design to overcome several limitations in the original design that constrain the dynamic reconfigurability of the pattern matcher. We propose two different solutions to dynamically change the character that is matched in a certain block. The resulting pattern matcher, after these changes, is fully dynamically reconfigurable, all aspects of the implemented regular expression can be changed at run-time.

Efficient protocols for private wildcards pattern matching

We propose a protocol for the problem of secure two-party pattern matching, where Alice holds a text t?{0,1}? of length n, while Bob has a pattern p?{0,1}? of length m. The goal is for Bob to (only) learn where his pattern occurs in Alice's text, while Alice learns nothing. Private pattern matching is an important problem that has many applications in the area of DNA search, computational biology and more. Our construction guarantees full simulation in the presence of malicious, polynomial-time adversaries (assuming the hardness of DDH assumption) and exhibits computation and communication costs of O(n+m) group elements in a constant round complexity. This improves over previous work by Gennaro et al. (Public Key Cryptography, pp. 145---160, 2010) whose solution requires overhead of O(nm) group elements and exponentiations in O(m) rounds. In addition to the above, we propose a collection of protocols for important variations of the secure pattern matching problem that are significantly more efficient than the current state of art solutions: First, we deal with secure pattern matching with wildcards. In this variant the pattern may contain wildcards that match both 0 and 1. Our protocol requires O(n+m) communication and O(1) rounds using O(nm) computation. Then we treat secure approximate pattern matching. In this variant the matches may be approximated, i.e., have Hamming distance less than some threshold, ?. Our protocol requires O(n?) communication in O(1) rounds using O(nm) computation. Third, we have secure pattern matching with hidden pattern length. Here, the length, m, of Bob's pattern remains a secret. Our protocol requires O(n+M) communication in O(1) rounds using O(n+M) computation, where M is an upper bound on m. Finally, we have secure pattern matching with hidden text length. Finally, in this variant the length, n, of Alice's text remains a secret. Our protocol requires O(N+m) communication in O(1) rounds using O(N+m) computation, where N is an upper bound on n.

Constrained sequence alignment has been proposed as a way for incorporating biologists' knowledge about common structures or functions into the alignment process. For alignment of protein sequences, several studies have suggested taking into account the motifs (a restricted regular expression) from the PROSITE database to guide alignments. The regular expression constrained sequence alignment has been introduced for this purpose. An alignment satisfies the constraint if part of it matches a given regular expression in each dimension (i.e. in each sequence aligned). There is a method that rewards the alignments that include a region matching the given regular expression. This method does not always guarantee the satisfaction of the constraint. Another method constructs a weighted finite automaton from the given regular expression, and presents a dynamic programming solution that simulates copies of this automaton to find an alignment with maximum score satisfying the regular expression constraint. We propose a new algorithm for the regular expression constrained multiple sequence alignment problem. Our algorithm considers two layers each of which corresponds to part of the dynamic programming matrix for the alignment of the given sequences. We compute each layer differently using dynamic programming. We propose the following modification in the definition of the problem: the region satisfying the constraint does not contribute to the total score. This modification is not necessary for the correctness and the performance in certain cases such as the constraint involves only one motif or motif-matching regions span short distance in each sequence but we believe that with this modification we achieve the same goal by doing less work in practice. Our algorithm is much more efficient than a previously proposed algorithm that uses weighted automata, and its performance in practice is comparable to (and under certain conditions even better than) that of the ordinary (unconstrained) multiple sequence alignment algorithm. Our experiments on real biological sequences, and regular expressions each composed of a sequence of motifs verify this

Background SAS has a set of RX (“regular expression”) pattern matching and string manipulation functions. They provide a great deal more flexibility and power than string functions such as INDEX(), SCAN(), SUBSTR(), etc. in managing character strings. These functions enable one to locate, extract, and change patterns of character strings and are especially useful in situations of varying data patterns, formats, and placement within your source data.

Finite state automata are of fundamental importance in theory as well as applications in computer science. This dissertation presents several parallel algorithms for processing of them, including minimizing the number of states for a given finite state automaton, finding homing sequences for a given finite state automaton, converting a regular expression into an $\\varepsilon$-free nondeterministic finite automaton. Implementation of some algorithms, as well as their experimental results, are also given. For minimization of deterministic finite automata, we present a parallel algorithm and discuss its implementation on a practical parallel machine using data parallel and message passing paradigms. Its time complexity on a p processor EREW PRAM $(p\\le n)$ for inputs of size n is $O({n\\log\\sp2n\\over p}+\\log n\\ \\log p)$ uniformly on almost all instances. The work done by our algorithm is thus within a factor of $O(\\log n)$ of the best known sequential algorithm. The space used by our algorithm is linear in the input size. The actual resource requirements of our implementations are consistent with these estimates. For finding homing sequences of deterministic finite automata, we present both deterministic and randomized algorithms. Two randomized algorithms of time complexity $O(log\\sp2n)$ are given. One of the randomized algorithms produces a homing sequence of length $O(n\\ log\\sp2n)$ for almost all DFA's with n states. The deterministic algorithm is based on a well known sequential algorithm. The implementation of the deterministic parallel algorithm shows that it can handle randomly generated instances with millions of states, and all DFA's with thousands of states using currently available parallel machines. Finally, we present an efficient parallel algorithm for converting a regular expression into an $\\varepsilon$-free nondeterministic finite automaton. The resulting automaton is natural representation of a regular expression and has some important and practical uses in the document processing area. The time complexity of the algorithm is $O(\\log n)$ and the number of processors used is $O(n\\sp2).$

Intelligent and efficient grouping algorithms for large-scale regular expressions