Parsing with derivatives

Parsing with Derivatives (PwD) is an elegant approach to parsing context-free grammars (CFGs). It takes the equational theory behind Brzozowski's derivative for regular expressions and augments that theory with laziness, memoization, and fixed points. The result is a simple parser for arbitrary CFGs. Although recent work improved the performance of PwD, it remains inefficient due to the algorithm repeatedly traversing some parts of the grammar. In this functional pearl, we show how to avoid this inefficiency by suspending the state of the traversal in a zipper. When subsequent derivatives are taken, we can resume the traversal from where we left off without retraversing already traversed parts of the grammar. However, the original zipper is designed for use with trees, and we want to parse CFGs. CFGs can include shared regions, cycles, and choices between alternates, which makes them incompatible with the traditional tree model for zippers. This paper develops a generalization of zippers to properly handle these additional features. Just as PwD generalized Brzozowski's derivatives from regular expressions to CFGs, we generalize Huet's zippers from trees to CFGs. Abstract The resulting parsing algorithm is concise and efficient: it takes only 31 lines of OCaml code to implement the derivative function but performs 6,500 times faster than the original PwD and 3.24 times faster than the optimized implementation of PwD.

We propose a novel extension of Brzozowski's derivative of regular expressions, called product derivatives. It takes two regular expressions and the meaning of its result is stated as follows: a product derivative of R with respect to S is a regular expression obtained by the consumption of all the sequences in the first regular expression by the second, i.e., the result is the product of Brzozowski's derivatives of R with respect to sequences in S. We develop an algorithm for our derivatives in coinductive manner. The termination of the algorithm is shown based on the proof method by Brandt and Henglein. We expect the derivative proposed in this talk is an important step to the development of the derivatives of a regular hedge expression taking a regular hedge expression as input, which is useful for transformation of XML documents with function terms typed with regular hedge expressions. The situation, for example, occurs in the transformation of XML documents with the links to the other Web services, such as Active XML documents.

We investigate the proof theory of regular expressions with fixed points, construed as a notation for ($$\omega $$ ω -)context-free grammars. Starting with a hypersequential system for regular expressions due to Das and Pous [15], we define its extension by least fixed points and prove the soundness and completeness of its non-wellfounded proofs for the standard language model. From here we apply proof-theoretic techniques to recover an infinitary axiomatisation of the resulting equational theory, complete for inclusions of context-free languages. Finally, we extend our syntax by greatest fixed points, now computing $$\omega $$ ω -context-free languages. We show the soundness and completeness of the corresponding system using a mixture of proof-theoretic and game-theoretic techniques.

We tackle the problem of analyzing filter and sanitizer programs remotely, i.e. given only the ability to query the targeted program and observe the output. We focus on two important and widely used program classes: regular expression (RE) filters and string sanitizers. We demonstrate that existing tools from machine learning that are available for analyzing RE filters, namely automata learning algorithms, require a very large number of queries in order to infer real life RE filters. Motivated by this, we develop the first algorithm that infers symbolic representations of automata in the standard membership/equivalence query model. We show that our algorithm provides an improvement of x15 times in the number of queries required to learn real life XSS and SQL filters of popular web application firewall systems such as mod-security and PHPIDS. % Active learning algorithms require the usage of an equivalence oracle, i.e. an oracle that tests the equivalence of a hypothesis with the target machine. We show that when the goal is to audit a target filter with respect to a set of attack strings from a context free grammar, i.e. find an attack or infer that none exists, we can use the attack grammar to implement the equivalence oracle with a single query to the filter. Our construction finds on average 90% of the target filter states when no attack exists and is very effective in finding attacks when they are present. For the case of string sanitizers, we show that existing algorithms for inferring sanitizers modelled as Mealy Machines are not only inefficient, but lack the expressive power to be able to infer real life sanitizers. We design two novel extensions to existing algorithms that allow one to infer sanitizers represented as single-valued transducers. Our algorithms are able to infer many common sanitizer functions such as HTML encoders and decoders. Furthermore, we design an algorithm to convert the inferred models into BEK programs, which allows for further applications such as cross checking different sanitizer implementations and cross compiling sanitizers into different languages supported by the BEK backend. We showcase the power of our techniques by utilizing our black-box inference algorithms to perform an equivalence checking between different HTML encoders including the encoders from Twitter, Facebook and Microsoft Outlook email, for which no implementation is publicly available.

Learning TOC – begins is a web application which is useful for Learning TOC (Theory of Computation). It covers hypothesis of the basic points with cases and it likewise has Exercise segment in which client can check different speculations for all intents and purposes. It likewise creates drawing of different cases. So the client can learn it adequately and additionally quick. Client can build FA of the string without anyone else and print or fare it for his task work. It has the office of Test to check his score and readiness work. So it is extremely valuable for client as exam arrangement. This Web Application is valuable for the educators and understudies and additionally different clients which are has a place with the Computer Science field. The fundamental reason for this web application is to pick up everything outwardly and graphically. It covers the listed topics given below- Regular Expression, Finite Automata, Context Free Grammar, Push Down Automata, Turing Machine, Exercise of the topics, Mock test.

We describe the formalization of a certified algorithm for regular expression parsing based on Brzozowski derivatives, in the dependently typed language Idris. The formalized algorithm produces a proof that an input string matches a given regular expression or a proof that no matching exists. A tool for regular expression based search in the style of the well known GNU grep has been developed with the certified algorithm, and practical experiments were conducted with this tool.

Regular expressions, or regexes, have been used traditionally as a pattern matching tool to search for structures in a set of objects, like flies, text documents or folders. Pattern matching can be used to look for flies whose name contains a given string, to search flies that contain a specific pattern within them, or simply to extract text in a set of documents. It is very popular to apply regexes to detect and extract patterns that represent phone numbers, URLs, email addresses, etc. These kind of information can be characterized because it has a well defined structure. Nevertheless, regexes are not very frequently used because its high complexity in both, syntax and grammatical rules, makes regexes difficult to understand. For this reason, the development of programs able to automatically generate, and evaluate, regexes has become a valuable task. This work analyzes the performance of different grammatical evolutionary approaches in the generation of regexes able to extract URL patterns. Four different types of grammars have been evaluated: a context-free grammar, a context-free grammar with a penalized fitness function, an extensible context-free grammar, and a Christiansen grammar. For the considered problem, the experimental results show that the best performance of the system, measured as cumulative success rate, is achieved using Christiansen grammars.

4 - Context-free grammar

On the degrees of non-regularity and non-context-freeness

The paper titled Hampi: A Solver for String Constraints was published in the proceedings of the International Symposium on Software Testing and Analysis (ISSTA) 2009, and has been selected to receive the ISSTA 2019 Impact Paper Award. The paper describes HAMPI, one of the first practical solver aimed at solving the satisfiability problem for a theory of string (word) equations, operations over strings, predicates over regular expressions and context-free grammars. HAMPI has been used widely to solve many software engineering and security problems, and has inspired considerable research on string solving algorithms and their applications. In this talk, we review the state of research on the theory and practice of string solving algorithms, specifically highlighting key historical developments that have led to their widespread use. On the practical front, we discuss different kinds of algorithmic paradigms, such as word- and automata-based, that have been developed to solve string and regular expression constraints. We then focus on the many hardness results that theorists have proved for fragments of theories over strings. Finally, we conclude with open theoretical problems, practical algorithmic challenges, and future applications of string solvers.

The grammar formalism Labelled BNF (LBNF) and the compiler construction tool BNF Converter are introduced. Given a grammar written in LBNF, the BNF Converter produces a complete compiler front end (up to, but excluding, type checking), i.e. a lexer, a parser, and an abstract syntax definition. Moreover, it produces a pretty-printer and a language specification in L A T E X, as well as a template file for the compiler back end. A language specification in LBNF is completely declarative and therefore portable. It reduces dramatically the effort of implementing a language. The price to pay is that the language must be well-behaved, i.e. that its lexical structure must be describable by a regular expression and its syntax by a context-free grammar.

One of the most important and difficult classes in the computer science major is the course on computational theory. The authors describe an approach to incorporate some hands-on laboratory activities, while sacrificing as little classic content as possible. The authors decided to focus on creating labs in which students could see a practical application to using regular expressions (using Lex) and context-free grammars (using Yacc). Along the way, the students gain insight on how finite automata work, and how a compiler parses source code and checks for syntax errors. The only significant problem we came across was that some students were unfamiliar with the C language or the UNIX operating system, but the students in general found the labs worthwhile. Next year, the authors plan on continuing with these laboratory experiences for the class, and plan to introduce one or two more exercises.

We define classical language equations and relate them to nondeterministic finite automata. Then, we give two quite different solution approaches, both constructive. One is based on syntactic properties of the equations and essentially yields regular expressions as solutions. The other is by construction of nondeterministic finite automata. Advantages and disadvantages of both approaches are discussed; the second method (construction of a nondeterministic finite automaton) prepares us for the construction of a different kind of finite automaton (namely boolean) in the next chapter. The first method (construction of a regular expression) motivates a discussion of the substitution property, a property that holds for our solutions if OP does not contain the complementation or the intersection operator. We conclude this chapter by commenting on context-free grammars and the equations they engender.

From the Publisher: This text makes in-depth explorations of a broad range of theoretical topics in computer science. It plunges into the applications of the abstract concepts in order to confront and address the skepticism of readers, and instill in them an appreciation for the usefulness of theory. A two-part presentation integrates logic and formal language—both with applications. Chapter topics cover mathematical preliminaries, propositional logic, proving things: why and how, predicate logic, proving with predicates, program verification, logic programming, language models for computer science, language models, finite automata and their languages, regular expressions, Lex: a tool for building lexical scanners, context-free grammars, pushdown automata and parsing, and turing machines. For future computer scientists.

Kleene Algebra (KA) is a useful tool for proving that two programs are equivalent. Because KA's equational theory is decidable, it integrates well with interactive theorem provers. This raises the question: which equations can we (not) prove using the laws of KA? Moreover, which models of KA are complete, in the sense that they satisfy exactly the provable equations? Kozen (1994) answered these questions by characterizing KA in terms of its language model. Concretely, equivalences provable in KA are exactly those that hold for regular expressions. Pratt (1980) observed that KA is complete w.r.t. relational models, i.e., that its provable equations are those that hold for any relational interpretation. A less known result due to Palka (2005) says that finite models are complete for KA, i.e., that provable equivalences coincide with equations satisfied by all finite KAs. Phrased contrapositively, the latter is a finite model property (FMP): any unprovable equation is falsified by a finite KA. Both results can be argued using Kozen's theorem, but the implication is mutual: given that KA is complete w.r.t. finite (resp. relational) models, Palka's (resp. Pratt's) arguments show that it is complete w.r.t. the language model. We embark on a study of the different complete models of KA, and the connections between them. This yields a novel result subsuming those of Palka and Pratt, namely that KA is complete w.r.t. finite relational models. Next, we put an algebraic spin on Palka's techniques, which yield a new elementary proof of the finite model property, and by extension, of Kozen's and Pratt's theorems. In contrast with earlier approaches, this proof relies not on minimality or bisimilarity of automata, but rather on representing the regular expressions involved in terms of transformation automata.