Efficient regular expression matching on LZ77 compressed strings using negative factors

Given a string S of n integers in \([0,\sigma )\), a range minimum query \({\textsf {RMQ}}(i, j)\) asks for the index of the smallest integer in \(S[i \dots j]\). It is well known that the problem can be solved with a succinct data structure of size \(2n + o(n)\) and constant query-time. In this paper we show how to preprocess S into a compressed representation that allows fast range minimum queries. This allows for sublinear size data structures with logarithmic query time. The most natural approach is to use string compression and construct a data structure for answering range minimum queries directly on the compressed string. We investigate this approach using grammar compression. We then consider an alternative approach. Even if S is not compressible, its Cartesian tree necessarily is. Therefore, instead of compressing S using string compression, we compress the Cartesian tree of S using tree compression. We show that this approach can be exponentially better than the former, and is never worse by more than an \(O(\sigma )\) factor (i.e. for constant alphabets it is never asymptotically worse).

The problem of finding matches of a regular expression (RE) on a string exists in many applications, such as text editing, biosequence search, and shell commands. Existing techniques first identify candidates using substrings in the RE, then verify each of them using an automaton. These techniques become inefficient when there are many candidate occurrences that need to be verified. In this article, we propose a novel technique that prunes false negatives by utilizing negative factors , which are substrings that cannot appear in an answer. A main advantage of the technique is that it can be integrated with many existing algorithms to improve their efficiency significantly. We present a detailed description of this technique. We develop an efficient algorithm that utilizes negative factors to prune candidates, then improve it by using bit operations to process negative factors in parallel. We show that negative factors, when used with necessary factors (substrings that must appear in each answer), can achieve much better pruning power. We analyze the large number of negative factors, and develop an algorithm for finding a small number of high-quality negative factors. We conducted a thorough experimental study of this technique on real datasets, including DNA sequences, proteins, and text documents, and show significant performance improvement of the state-of-the-art tools by an order of magnitude.

Packet content scanning at high speed has become extremely important due to its applications in network security, network monitoring, HTTP load balancing, etc. In content scanning, the packet payload is compared to a set of patterns specified as regular expressions. In this chapter, we first describe the typical patterns used in packet-scanning applications and show that for some of these patterns the memory requirements can be prohibitively high when traditional matching methods are used. We then review techniques for efficient regular expression matching and explore regular expression rewrite techniques that can significantly reduce memory usage. Based on new rewrite insights, we propose guidelines for pattern writers to make matching fast and practical. Furthermore, we discuss deterministic finite automaton (DFA) link compression techniques and review algorithms and data structures that are specifically designed for matching regular expressions in networking applications.

Data processing and information retrieval is key technologies in information technology area. Regular expression matching plays a crucial role in information retrieval. However, in regular expression matching, the traditional DFA method has serious performance degradation with a large-scale rule set. And building a hybrid DFA will lead to a substantial increase in space requirements. To address these problems, according to the rule set’s characteristics and combining with multi-pattern idea, a matching mechanism based on multi-pattern finite automation is proposed. Compared with the traditional matching algorithm, the new mechanism’s time complexity can be reduced to O(M+n). Using experiment, we demonstrate that by parallel processing, it can achieve an efficient regular expression based information retrieval and matching. This provides a new technical support for regular expression in information retrieval and language processing fields, and has a wider range of applications.Keywordsinformation technologyretrievalregular expressionmulti-patternparallel processing

Intelligent and efficient grouping algorithms for large-scale regular expressions

In this article we study a very simple linear reduction algorithm that is specific to regular expressions. It aims to detect, in a bottom-up fashion, universal subtrees in regular expressions trees, and replace them by the smallest equivalent to \(\varSigma ^*\). Of course, this does not detect every universal subtree, as the universality problem is PSPACE-complete. However, we prove that for the uniform random tree model, this simple algorithm detects a large proportion of universal trees. Furthermore, we prove that on average this algorithm reduces uniform regular expressions to a constant size that is very small, and that can be computed efficiently. For example, for two letters the limit expected size is \(\sim \)77.8. Our theoretical constants are backed-up by the experimental evidence. This confirms the phenomena reported in [13], and further it completely discards the usefulness of the uniform distribution on regular expressions.

Many network security applications in today's networks are based on deep packet inspection, checking not only the header portion but also the payload portion of a packet. For example, traffic monitoring, layer-7 filtering, and network intrusion detection all require an accurate analysis of packet content in search for predefined patterns to identify specific classes of applications, viruses, attack signatures, etc. Regular expressions are often used to represent such patterns. They are implemented using finite automata, which take the payload of a packet as an input string. However, existing approaches, both non-deterministic finite automata (NFA) and deterministic finite automata (DFA), do not deal with compressed traffic, which becomes more and more popular in HTTP applications. In this paper, we propose an efficient algorithm for regular expression matching to implement deep packet inspection on compressed traffic. Based on the observations of DFA, we design a scheme to skip most of the matching process in the compressed parts of traffic. To the best of our knowledge, this is the first effort to design an efficient regular expression matching on compressed traffic. We evaluate our algorithm using rule sets provided by Snort, a popular open-source intrusion detection system. The evaluation results show that our approach can reduce the number of state access in the DFA significantly.

For the Government projects, Public–Private Partnership (PPP) is an important parameter based on which the project is procured through a private-sector. The objective of this paper was to identify and evaluate the perception of respondents for negative factors affecting Indian road projects undertaken by PPP. Thirteen negative factors were identified through a literature survey and validated with experienced professionals from the public-sector, private-sector and researcher. A questionnaire survey was conducted among major PPP project participants and researchers of Indian PPP road projects to investigate the most important negative factors. Identified factors were grouped into four categories: lack of experience, high cost, project's effectiveness and high risk using factor analysis. The four most important negative factors of Indian PPP Road Projects were identified using the Importance Index Method, which is: excessive restriction on participation, lengthy delays in negotiation, confusion over government objectives and evolution criteria, and fewer employment opportunities. The perception was analyzed with various statistical tools. This paper gives recognition of very important negative factors in the PPP road sector in the Indian context which will help consider road projects under PPP during the development phase itself.

Cyber-physical systems (CPS) and the Internet-of-Things (IoT) result in a tremendous amount of generated, measured and recorded time-series data. Extracting temporal segments that encode patterns with useful information out of these huge amounts of data is an extremely difficult problem. We propose shape expressions as a declarative formalism for specifying, querying and extracting sophisticated temporal patterns from possibly noisy data. Shape expressions are regular expressions with arbitrary (linear, exponential, sinusoidal, etc.) shapes with parameters as atomic predicates and additional constraints on these parameters. We equip shape expressions with a novel noisy semantics that combines regular expression matching semantics with statistical regression. We characterize essential properties of the formalism and propose an efficient approximate shape expression matching procedure. We demonstrate the wide applicability of this technique on two case studies.

Bulk-Synchronous Parallel (BSP) is a bridging model between abstract execution and concrete parallel architectures that retains enough information to model performance scalability. In order to model BSP program executions, Hains adapted the theory of finite automata to the BSP paradigm by introducing BSP automata [12]. In the initial description of the theory, BSP automata had to be explicitly defined as structured sets of states and transitions. The lack of a clean and efficient algorithm for generating them from regular expressions would have prevented this theory from being used in practice. To alleviate this problem we introduce in this paper an algorithm that generates a BSP automaton recognizing a BSP language defined by a BSP regular expression. The main practical use of BSP automata developed in this paper is the parallelization of finite state automata with an explicit distribution and a performance model, that enable parallel matching of regular expressions. Secondarily, BSP regular expressions provide a convenient structure for automatic code generation of imperative BSP programs that is also developed in this paper.

We present an efficient algorithm for regular expression matching on streams with out of order data, while maintaining a small state and without complete stream reconstruction. We have implemented three versions of the algorithm - sequential, parallel and mixed - and show by experimental study that the algorithms are highly effective in matching regular expressions on IP packet streams.

We show in this paper that parsing with regular expressions instead of context-free grammars, when it is possible, is desirable. We present efficient algorithms for performing different tasks that concern parsing: producing the external representation and the internal representation of parse trees; producing all possible parse trees or a single one. Each of our algorithms to produce a parse tree from an input string has an optimal time complexity, linear with the length of the string. Moreover, ambiguous regular expressions can be used.

Outlier detection is a well-researched and crucial problem in machine learning. However, there is little research on string data outlier detection, as most literature focuses on outlier detection of numerical data. A robust string data outlier detection algorithm could assist with data cleaning or anomaly detection in system log files. In this thesis, we compare two string outlier detection algorithms. Firstly, we introduce a variant of the well-known local outlier factor algorithm, which we tailor to detect outliers on string data using the Levenshtein measure to calculate the density of the dataset. We present a differently weighted Levenshtein measure, which considers hierarchical character classes and can be used to tune the algorithm to a specific string dataset. Secondly, we introduce a new kind of outlier detection algorithm based on the hierarchical left regular expression learner, which infers a regular expression for the expected data. Using various datasets and parameters, we experimentally show that both algorithms can conceptually find outliers in string data. We show that the regular expression-based algorithm is especially good at finding outliers if the expected values have a distinct structure that is sufficiently different from the structure of the outliers. In contrast, the local outlier factor algorithms are best at finding outliers if their edit distance to the expected data is sufficiently distinct from the edit distance between the expected data.

In this paper, we present PaTSI, a tool for analyzing evolutions of objects in time series of satellite images. This tool is a plugin integrated in the KNIME Analytics Platform. PaTSI is a workflow composed of several nodes assembled together to form a whole KDD process (data selection, pre-processing, image segmentation, pattern mining and visualization). Input data consists of a time series of satellite images and GIS information on the studied area. This data is transformed in a single attributed directed acyclic graph (a-DAG), where nodes represent objects described by several attributes and edges represent temporal relationships. This graph is then mined to extract frequent evolutions (weighted path patterns) using an efficient graph mining algorithm. At the end of the process, extracted patterns can be filtered using regular expressions and displayed on the original images in order to facilitate the experts' interpretation of the results. In the present demo, the pertinence of PaTSI is illustrated through its application to soil erosion monitoring.

We propose quantitative regular expressions (QREs) as a high-level programming abstraction for specifying complex numerical queries over data streams in a modular way. Our language allows the arbitrary nesting of orthogonal sets of combinators: (a) generalized versions of choice, concatenation, and Kleene-iteration from regular expressions, (b) streaming (serial) composition, and (c) numerical operators such as min, max, sum, difference, and averaging. Instead of requiring the programmer to figure out the low-level details of what state needs to be maintained and how to update it while processing each data item, the regular constructs facilitate a global view of the entire data stream splitting it into different cases and multiple chunks. The key technical challenge in defining our language is the design of typing rules that can be enforced efficiently and which strike a balance between expressiveness and theoretical guarantees for well-typed programs. We describe how to compile each QRE into an efficient streaming algorithm. The time and space complexity is dependent on the complexity of the data structure for representing terms over the basic numerical operators. In particular, we show that when the set of numerical operations is sum, difference, minimum, maximum, and average, the compiled algorithm uses constant space and processes each symbol in the data stream in constant time outputting the cost of the stream processed so far. Finally, we prove that the expressiveness of QREs coincides with the streaming composition of regular functions, that is, MSO-definable string-to-term transformations, leading to a potentially robust foundation for understanding their expressiveness and the complexity of analysis problems.