Minimal Markov chain embeddings of pattern problems

We study the set of incompressible strings for various resource bounded versions of Kolmogorov complexity. The resource bounded versions of Kolmogorov complexity we study are polynomial time CD complexity defined by Sipser, the nondeterministic variant CND due to Buhrman and Fortnow, and the polynomial space bounded Kolmogorov complexity CS introduced by Hartmanis. For all of these measures we define the set of random strings $\mathrm{R}^{\mathit{CD}}_t$, $\mathrm{R}^{\mathit{CND}}_t$, and $\mathrm{R}^{\mathit{CS}}_t$ as the set of strings x such that $\mathit{CD}^t(x)$, $\mathit{CND}^t(x)$, and $\mathit{CS}^s(x)$ is greater than or equal to the length of x for s and t polynomials. We show the following: $\mathrm{MA} \subseteq \mathrm{NP}^{\mathrm{R}^{\mathit{CD}}_t}$, where $\mathrm{MA}$ is the class of Merlin--Arthur games defined by Babai. $\mathrm{AM} \subseteq \mathrm{NP}^{\mathrm{R}^{\mathit{CND}}_t}$, where $\mathrm{AM}$ is the class of Arthur--Merlin games. $\mathrm{PSPACE} \subseteq \mathrm{NP}^{\mathrm{cR}^{\mathit{CS}}_s}$. In the last item $\mathrm{cR}^{\mathit{CS}}_s$ is the set of pairs $\langle x,y \rangle$ so that x is random given y. These results show that the set of random strings for various resource bounds is hard for complexity classes under nondeterministic reductions. This paper contrasts the earlier work of Buhrman and Mayordomo where they show that for polynomial time deterministic reductions the set of exponential time Kolmogorov random strings is not complete for EXP.

Many-core accelerator chips are becoming increasingly popular these days for its high performance floating-point performance exceeding 1 Tflops per chip. Aho-Corasick (AC) is a multiple patterns string matching algorithm commonly used in computer and network security, bioinformatics, among others. In order to simultaneously match a number of string patterns with respect to the input text data, a Deterministic Finite Automata (DFA) is constructed from a given set of pattern strings. The DFA is referenced almost randomly, whereas the input data is sequentially accessed. As the number of pattern strings increases, the irregular DFA accesses lead to poor cache locality and low overall performance. In this paper, we present a cache locality optimizing parallelization on the many-core accelerator chip, the Intel Xeon Phi. A given set of pattern strings is partitioned into into multiple sets of a smaller number of patterns so that multiple small DFAs are constructed instead of single large DFA. The accesses to multiple small DFAs lead to significantly smaller cache footprints in each core's cache and result in impressive performance improvements. Experimental results on the Intel Xeon Phi 5110P show that our approach delivers up to 2.00 × speedup compared with the previous approach using single large DFA.

Pattern matching combined with regular expressions has many applications including semistructured data matching and lexical analysis in compilers. Variables in patterns allow one to refer to parts of the matching input. But some regular patterns suffer from inherent ambiguity, yielding more than one valid result. A match policy like shortest or longest match can disambiguate such patterns.In this paper, we show that regular pattern matching corresponds to sequential transduction. We derive straightforward ways to optimally compile regular patterns to sequential machines and to decide when regular patterns are unambiguous. Unambiguous patterns can be matched in a single traversal of the input. Ambiguities in patterns correspond to nondeterminism in sequential machines. Applying the match policy optimally yields two deterministic sequential machines, which produce the shortest match in two consecutive runs.

This paper proposes new notions of polynomial depth (called monotone poly depth), based on a polynomial version of monotone Kolmogorov complexity. We show that monotone poly depth satisfies all desirable properties of depth notions i.e., both trivial and random sequences are not monotone poly deep, monotone poly depth satisfies the slow growth law i.e., no simple process can transform a non deep sequence into a deep one, and monotone poly deep sequences exist (unconditionally).We give two natural examples of deep sets, by showing that both the set of Levin-random strings and the set of Kolmogorov random strings are monotone poly deep.KeywordsKolmogorov ComplexityRandom StringPolynomial VersionUniversal Turing MachineLogical DepthThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

A pattern is a string consisting of constant symbols and variables. The language of a pattern is the set of constant strings obtained by substituting nonempty constant strings for variables in the pattern. For any fixed k, the class of unions of at most k pattern languages is already shown to be inferable from positive data. The class of all the unions of arbitrarily finitely many pattern languages is not inferable, because any constant string defines a singleton set consisting of itself, and the class of unions contains all the finite languages. A proper pattern is a pattern that contains at least one variable. The language of a proper pattern is infinite. In this paper, we consider the class of unions when patterns are restricted to be proper and show that the class is not inferable from positive data. A regular pattern is a pattern that contains at most one occurrence of every variable. When regular patterns are restricted not to contain more than l consecutive occurrences of constant symbols for some l, the class of unions is shown to be inferable from positive data.

Regular expression has been widely used in network intrusion detection system to represent attack patterns due to its expressive power and flexibility. However, the traditional memory architecture suffers from the problem of memory explosion for certain types of complex regular expressions. In this paper, we propose a hierarchical state machine architecture which can significantly reduce the memory required to accommodate complex regular expression patterns. The experiments demonstrate a significant reduction in memory for the complex regular expression patterns commonly used in network intrusion detection systems.

We tackle the problem of analyzing filter and sanitizer programs remotely, i.e. given only the ability to query the targeted program and observe the output. We focus on two important and widely used program classes: regular expression (RE) filters and string sanitizers. We demonstrate that existing tools from machine learning that are available for analyzing RE filters, namely automata learning algorithms, require a very large number of queries in order to infer real life RE filters. Motivated by this, we develop the first algorithm that infers symbolic representations of automata in the standard membership/equivalence query model. We show that our algorithm provides an improvement of x15 times in the number of queries required to learn real life XSS and SQL filters of popular web application firewall systems such as mod-security and PHPIDS. % Active learning algorithms require the usage of an equivalence oracle, i.e. an oracle that tests the equivalence of a hypothesis with the target machine. We show that when the goal is to audit a target filter with respect to a set of attack strings from a context free grammar, i.e. find an attack or infer that none exists, we can use the attack grammar to implement the equivalence oracle with a single query to the filter. Our construction finds on average 90% of the target filter states when no attack exists and is very effective in finding attacks when they are present. For the case of string sanitizers, we show that existing algorithms for inferring sanitizers modelled as Mealy Machines are not only inefficient, but lack the expressive power to be able to infer real life sanitizers. We design two novel extensions to existing algorithms that allow one to infer sanitizers represented as single-valued transducers. Our algorithms are able to infer many common sanitizer functions such as HTML encoders and decoders. Furthermore, we design an algorithm to convert the inferred models into BEK programs, which allows for further applications such as cross checking different sanitizer implementations and cross compiling sanitizers into different languages supported by the BEK backend. We showcase the power of our techniques by utilizing our black-box inference algorithms to perform an equivalence checking between different HTML encoders including the encoders from Twitter, Facebook and Microsoft Outlook email, for which no implementation is publicly available.

Consider a given pattern and a random text generated by a Markovian source of any order. We study the frequency of pattern occurrences in a random text when overlapping copies of the pattern are counted separately. More precisely: we consider two strings, a given pattern string H=h/sub 1/h/sub 2/....h/sub m/ and a random text string T=t/sub 1/t/sub 2/...t/sub n/ of respective lengths equal to m and n over an alphabet S of size V. The results for the Markovian model are summarized in two theorems.

LogExtractor: Extracting digital evidence from android log messages via string and taint analysis

PERG is a pattern matching engine designed for locating pre-defined byte string patterns (rules) from the ClamAV virus signature database in a data stream. This paper presents PERG-Rx, an extension of PERG that adds limited regular expression support for wildcard patterns used by rules that represent polymorphic viruses. To reduce the amount of state needed to track so many regular expressions, PERG-Rx employs a lossy scheme which increases the rate of false positives detected as the required state grows. The scalability and dynamic updatability of the PERG-Rx architecture to database updates are also evaluated.

Intelligent and efficient grouping algorithms for large-scale regular expressions

In this paper, we propose a systolic pattern-independent hardware regular expression matching (REM) engine which handles nested Kleene operators used in virus patterns. Pattern-independent systolic REM engines are suitable to network intrusion detection systems for quick updating of virus pattern. In the proposed engine, we introduce a compact pattern-independent NFA circuit, which can handle any small regular expression patterns, into a systolic REM engine to handle nested Kleene operators. Experimental results show that the extended engine implemented on an FPGA handles nested Kleene operators with efficient circuit size and high performance (2.17Gbps).

An analytical comparison of two string searching algorithms

The aim of the paper is to concoct a novel algorithm for the metamorphosis of parallel regular expressions to e-free non- deterministic finite automata. For a given parallel regular expression r, let m be the number of symbols that occur in r and let C denote the number of concatenation operators in r. In the worst case, 2 m+1 states are required for the construction of the non-deterministic finite automaton using the novel algorithm. In the earlier existing approaches, th e number of states of the non-deterministic finite automaton in the worst case is equal to 2 2jrj−3C.

We provide syntactic derivative-like operations, defined by recursion on regular expressions, in the styles of both Brzozowski and Antimirov, for trace closures of regular languages. Just as the Brzozowski and Antimirov derivative operations for regular languages, these syntactic reordering derivative operations yield deterministic and nondeterministic automata respectively. But trace closures of regular languages are in general not regular, hence these automata cannot generally be finite. Still, as we show, for star-connected expressions, the Antimirov and Brzozowski automata, suitably quotiented, are finite. We also define a refined version of the Antimirov reordering derivative operation where parts-of-derivatives (states of the automaton) are nonempty lists of regular expressions rather than single regular expressions. We define the uniform scattering rank of a language and show that, for a regexp whose language has finite uniform scattering rank, the truncation of the (generally infinite) refined Antimirov automaton, obtained by removing long states, is finite without any quotienting, but still accepts the trace closure. We also show that star-connected languages have finite uniform scattering rank.