Parameterized matching on non-linear structures

BackgroundApproximate string matching is the problem of finding all factors of a given text that are at a distance at most k from a given pattern. Fixed-length approximate string matching is the problem of finding all factors of a text of length n that are at a distance at most k from any factor of length ℓ of a pattern of length m. There exist bit-vector techniques to solve the fixed-length approximate string matching problem in time mathcal {O}(mlceil ell /w rceil n) and space mathcal {O}(mlceil ell /wrceil) under the edit and Hamming distance models, where w is the size of the computer word; as such these techniques are independent of the distance threshold k or the alphabet size. Fixed-length approximate string matching is a generalisation of approximate string matching and, hence, has numerous direct applications in computational molecular biology and elsewhere.ResultsWe present and make available libFLASM, a free open-source C++ software library for solving fixed-length approximate string matching under both the edit and the Hamming distance models. Moreover we describe how fixed-length approximate string matching is applied to solve real problems by incorporating libFLASM into established applications for multiple circular sequence alignment as well as single and structured motif extraction. Specifically, we describe how it can be used to improve the accuracy of multiple circular sequence alignment in terms of the inferred likelihood-based phylogenies; and we also describe how it is used to efficiently find motifs in molecular sequences representing regulatory or functional regions. The comparison of the performance of the library to other algorithms show how it is competitive, especially with increasing distance thresholds.ConclusionsFixed-length approximate string matching is a generalisation of the classic approximate string matching problem. We present libFLASM, a free open-source C++ software library for solving fixed-length approximate string matching. The extensive experimental results presented here suggest that other applications could benefit from using libFLASM, and thus further maintenance and development of libFLASM is desirable.

For genomic databases, approximate string matching with k errors is often considered for genomic sequences, where k errors could be caused by substitution, insertion, or deletion operations. In this paper, we propose a new approximate string matching method, the hash trie filter, for efficiently searching in genomic databases. Our method not only reduces the number of candidates by pruning some unreasonable matched positions, but also dynamically decides the number of ordered matched grams of one candidate, which results in the increase of precision. The experiment results show that the hash trie filter outperforms the well-known (k + s) q-samples filter in terms of the response time and the precision, under different lengths of the query patterns and different error levels.KeywordsApproximate string matchingfilterhash trie

From the empirical studies, it is quite difficult for the license plate recognition to perform 100% accuracy in a real world environment. Nevertheless, it is common that only a few characters are misread from a license plate recognition system. In this paper, license plate matching is used for vehicle re-identification. We evaluate several approximate string matching techniques to determine an applicable method for the domain of license plate matching. We also propose a weighted approximate string matching to further improve the performance. Our experimental results show that the generalized edit distance (GED) yields highest precision while the 2-Grams yields the highest recall. By weighting the distance cost from those two methods, our proposed weighted method gives the best performance.

Exact string matching in labeled graphs is the problem of searching paths of a graph G=(V, E) such that the concatenation of their node labels is equal to a given pattern string P [1. m ]. This basic problem can be found at the heart of more complex operations on variation graphs in computational biology, of query operations in graph databases, and of analysis operations in heterogeneous networks. We prove a conditional lower bound stating that, for any constant ε > 0, an O (| E | 1 - ε m ) time, or an O (| E | m 1 - ε )time algorithm for exact string matching in graphs, with node labels and pattern drawn from a binary alphabet, cannot be achieved unless the Strong Exponential Time Hypothesis ( SETH ) is false. This holds even if restricted to undirected graphs with maximum node degree 2—that is, to zig-zag matching in bidirectional strings , or to deterministic directed acyclic graphs whose nodes have maximum sum of indegree and outdegree 3. These restricted cases make the lower bound stricter than what can be directly derived from related bounds on regular expression matching (Backurs and Indyk, FOCS’16). In fact, our bounds are tight in the sense that lowering the degree or the alphabet size yields linear time solvable problems. An interesting corollary is that exact and approximate matching are equally hard (i.e., quadratic time) in graphs under SETH . In comparison, the same problems restricted to strings have linear time vs quadratic time solutions, respectively (approximate pattern matching having also a matching SETH lower bound (Backurs and Indyk, STOC’15)).

Exploiting word-level parallelism for fast convolutions and their applications in approximate string matching

Approximate string matching has been widely used in many applications, including deoxyribonucleic acid sequence searching, spell checking, text mining, and spam filters. The method is designed to find all locations of strings that approximately match a pattern in accordance with the number of insertion, deletion, and substitution operations. Among the proposed algorithms, the bit-parallel algorithms are considered to be the best and highly efficient algorithms. However, the traditional bit-parallel algorithms lacks the ability of identifying the start and end positions of a matched pattern. Furthermore, acceleration of the bit-parallel algorithms has become a crucial issue for processing big data nowadays. In this paper, we propose two kinds of parallel location-aware algorithms called data-segmented parallelism and high-degree parallelism as means to accelerate approximate string matching using graphic processing units. Experimental results show that the high-degree parallelism on GPUs achieves significant improvement in system and kernel throughputs compared to CPU counterparts. Compared to state-of-the-art approaches, the proposed high-degree parallelism achieves 11 to 105 times improvement.

Approximate string matching over suffix tree with depth-first search (ASM_ST_DFS), a classical algorithm in the field of approximate string matching, was originally proposed by Ricardo A. Baeza-Yates and Gaston H. Gonnet in 1990. The algorithm is one of the most excellent algorithms for approximate string matching if combined with other indexing techniques. However, its time complexity is sensitive to the length of pattern string because it searches $$m+k$$m+k characters on each path from the root before backtracking. In this paper, we propose an efficient pruning strategy to solve this problem. We prove its correctness and efficiency in theory. Particularly, we proved that if the pruning strategy is adopted, it averagely searches O(k) characters on each path before backtracking instead of O(m). Considering each internal node of suffix tree has multiple branches, the pruning strategy should work very well. We also experimentally show that when k is much smaller than m, the efficiency improves hundreds of times, and when k is not much smaller than m, it is still several times faster. This is the first paper that tries to solve the backtracking problem of ASM_ST_DFS in both theory and practice.

Reliability problems in software engineering — a review: Downs, TComput. Syst. Sci. Eng. Vol 2 No 3 (July 1987) pp 131–147

In a 2-dimensional (2D) pattern matching problem, the text is arranged as a matrix 𝖬[1..n, 1..n] and consists of N = n × n symbols drawn from alphabet set Σ of size σ. The query consists of a m × m square matrix 𝖯[1..m, 1..m] drawn from the same alphabet set Σ and the task is to find all the locations in 𝖬 where 𝖯 appears as a (contiguous) submatrix. The patterns can be of any size, but as long as they are square in shape data structures like suffix trees and suffix array exist [Raffaele Giancarlo, 1995; Dong Kyue Kim et al., 1998] for the task of efficient pattern matching. These are essentially 2D counterparts of classic suffix trees and arrays known for traditional 1-dimensional (1D) pattern matching. They work based on linearization of 2D suffixes which would preserve the prefix match property (i.e., every pattern match is a prefix of some suffix). The main limitation of the suffix trees and the suffix arrays (in 1D) was their space utilization of O(N log N) bits, where N is the size of the text. This was suboptimal compared to Nlog σ bits of space, which is information theoretic optimal for the text. With the advent of the field of succinct/compressed data structures, it was possible to develop compressed variants of suffix trees and array based on Burrows-Wheeler Tansform and LF-mapping (or Φ function) [Roberto Grossi and Jeffrey Scott Vitter, 2005; Paolo Ferragina and Giovanni Manzini, 2005; Kunihiko Sadakane, 2007]. These data structures indeed achieve O(N log σ) bits of space or better. This gives rise to the question: analogous to 1D case, can we design a succinct or compressed index for 2D pattern matching? Can there be a 2D compressed suffix tree? Are there analogues of Burrows-Wheeler Transform or LF-mapping? The problem has been acknowledged for over a decade now and there have been a few attempts at applying Φ function [Ankur Gupta, 2004] and achieving entropy based compression [Veli Mäkinen and Gonzalo Navarro, 2008]. However, achieving the complexity breakthrough akin to 1D case has yet to be found. In this paper, we still do not know how to answer suffix array queries in O(N log σ) bits of space - which would have led to efficient pattern matching. However, for the first time, we show an interesting result that it is indeed possible to compute inverse suffix array (ISA) queries in near compact space in O(polylog n) time. Our 2D succinct text index design is based on two 1D compressed suffix trees and it takes O(N log log N + N logσ) bits of space which is much smaller than its naive design that takes O(N log N) bits. Although the main problem is still evasive, this index gives a hope on the existence of a full 2D succinct index with all functionalities similar to that of 1D case.

BackgroundSemantic Category Disambiguation (SCD) is the task of assigning the appropriate semantic category to given spans of text from a fixed set of candidate categories, for example Protein to “Fibrin”. SCD is relevant to Natural Language Processing tasks such as Named Entity Recognition, coreference resolution and coordination resolution. In this work, we study machine learning-based SCD methods using large lexical resources and approximate string matching, aiming to generalise these methods with regard to domains, lexical resources and the composition of data sets. We specifically consider the applicability of SCD for the purposes of supporting human annotators and acting as a pipeline component for other Natural Language Processing systems.ResultsWhile previous research has mostly cast SCD purely as a classification task, we consider a task setting that allows for multiple semantic categories to be suggested, aiming to minimise the number of suggestions while maintaining high recall. We argue that this setting reflects aspects which are essential for both a pipeline component and when supporting human annotators. We introduce an SCD method based on a recently introduced machine learning-based system and evaluate it on 15 corpora covering biomedical, clinical and newswire texts and ranging in the number of semantic categories from 2 to 91.With appropriate settings, our system maintains an average recall of 99% while reducing the number of candidate semantic categories on average by 65% over all data sets.ConclusionsMachine learning-based SCD using large lexical resources and approximate string matching is sensitive to the selection and granularity of lexical resources, but generalises well to a wide range of text domains and data sets given appropriate resources and parameter settings. By substantially reducing the number of candidate categories while only very rarely excluding the correct one, our method is shown to be applicable to manual annotation support tasks and use as a high-recall component in text processing pipelines. The introduced system and all related resources are freely available for research purposes at: https://github.com/ninjin/simsem.

A practical suboptimal (variable source coding) algorithm for lossy data compression is presented. This scheme is based on approximate string matching, and it naturally extends the lossless Lempel-Ziv (1977) data compression scheme. Among others we consider the typical length of an approximately repeated pattern within the first n positions of a stationary mixing sequence where D percent of mismatches is allowed. We prove that there exists a constant r/sub 0/(D) such that the length of such an approximately repeated pattern converges in probability to 1/r/sub 0/(D) log n (pr.) but it almost surely oscillates between 1/r/sub -/spl infin//(D) log n and 2/r/sub 1/(D) log n, where r/sub -/spl infin//(D)>r/sub 0/(D)>r/sub 1/(D)/2 are some constants. These constants are natural generalizations of Renyi entropies to the lossy environment. More importantly, we show that the compression ratio of a lossy data compression scheme based on such an approximate pattern matching is asymptotically equal to r/sub 0/(D). We also establish the asymptotic behavior of the so-called approximate waiting time N/sub l/ which is defined as the time until a pattern of length C repeats approximately for the first time. We prove that log N/sub l//l/spl rarr/r/sub 0/(D) (pr.) as l/spl rarr//spl infin/. In general, r/sub 0/(D)>R(D) where R(D) is the rate distortion function. Thus for stationary mixing sequences we settle in the negative the problem investigated by Steinberg and Gutman by showing that a lossy extension of the Wyner-Ziv (1989) scheme cannot be optimal.

Video is the most powerful and, at the same time, the most complex of all media used for conveying information. Hence, representing video information to enable effective and efficient retrieval is an interesting problem. In this paper, we first discuss the architecture of our video retrieval system.Then we discuss a set of metrics for validating this system through measurements of it's ‘effectiveness’ and ‘efficiency’. Effectiveness is characterized through the metrics of Recall and Precision for ‘exact match’ queries, and through F-Effectiveness for ‘approximate match’ queries. Efficiency is characterised through a metric called EM. We also show how to fine-tune a system using these metrics. Considering a video database of visuals of a cricket match, the details of the work are discussed.

Many software engineering tasks can potentially be automated using search heuristics. However, much work is needed in designing and evaluating search heuristics before this approach can be routinely applied to a software engineering problem. Experimental methodology should be complemented with theoretical analysis to achieve this goal. Recently, there have been significant theoretical advances in the runtime analysis of evolutionary algorithms (EAs) and other search heuristics in other problem domains. We suggest that these methods could be transferred and adapted to gain insight into the behaviour of search heuristics on software engineering problems while automating software engineering.

<strong>Background: </strong>Statistical concepts and techniques are often applied incorrectly, even in mature disciplines such as medicine or psychology. Surprisingly, there are very few works that study statistical problems in software engineering (SE).<strong>Aim: </strong>Assess the existence of statistical errors in SE experiments.<strong>Method: </strong>Compile the most common statistical errors in experimental disciplines. Survey experiments published in ICSE to assess whether errors occur in high quality SE publications.<strong>Results:</strong> The same errors as identified in others disciplines were found in ICSE experiments, where 30% of the reviewed papers included several error types such as: a) missing statistical hypotheses, b) missing sample size calculation, c) failure to assess statistical test assumptions, and d) uncorrected multiple testing. This rather large error rate is greater for research papers where experiments are confined to the validation section. The origin of the errors can be traced back to: a) researchers not having sufficient statistical training, and, b) a profusion of exploratory research.<strong>Conclusions:</strong> This paper provides preliminary evidence that SE research suffers from the same statistical problems as other experimental disciplines. However, the SE community appears to be unaware of any shortcomings in its experiments, whereas other disciplines work hard to avoid these threats. Further research is necessary to find the underlying causes and set up corrective measures, but there are some potentially effective actions and are a priori easy to implement: a) improve the statistical training of SE researchers, and b) enforce quality assessment and reporting guidelines in SE publications.

Search-based software engineering (SBSE) consists of the application of metaheuristic search techniques—such as genetic algorithms, simulated annealing, hill climbing, and tabu search—to software engineering problems. This idea is based on the observation that some software engineering activities can be thought of as optimization problems, and on the fact that techniques providing exact solutions are, very often, unable to adequately scale to large, real-world software engineering problems. Define fitness functions to guide an automated search for solutions, rather than attempting to construct solutions from scratch by hand. SBSE moves the engineer up the abstraction chain, from the tedious and expensive role of detailed solution construction, to the potentially more productive and more cost-effective area of fitness function design. This approach has been repeatedly demonstrated to have successful applications across the software engineering life cycle from requirements and project planning to maintenance and testing. This special issue focuses on applications of search SBSE to the maintenance phase of the life cycle. After software testing 2, software maintenance is the most widely studied application area for SBSE research. Work on SBSE applications to problems in software maintenance can be traced back to the pioneering work of the Mancoridis et al. 3, which led to the development of the Bunch tool for software re-modularization 4. In the past five years there has been a significant increase in SBSE applications to software maintenance and evolution. Search-based techniques have been applied to post-development code improvement problems, such as object-oriented refactoring 5, 6, and imperative language slicing and transformation 7. Other applications are related to software engineering economics and project management problems, such as release planning 8-12, effort, and cost estimation 13-16, and project staffing and planning 17-20. Last, but not the least, SBSE has been applied to deal with service-oriented architectures, for example for services QoS-aware composition 21, service-level agreement negotiation 22, and testing 23. A complete overview of SBSE with a discussion of future research trends can be found in the recent International Conference on Software Engineering (ICSE) ‘Future of Software Engineering’ paper 24. This issue of the Journal of Software Maintenance and Evolution: Research and Practice features three outstanding papers, dealing with different aspects of software maintenance and evolution: release planning and staffing, re-factoring, and component substitution. The paper ‘Optimized Staffing for Product Releases—Focused Search and its Application at Chartwell Technologies’ by Puneet Kapur, An Ngo-The, Günther Ruhe, and Andrew Smith presents an approach for optimizing the staffing for product releases. The problem they solve can be viewed as part of the more general release planning problem, which assigns features to releases such that technical, resource, risk, and budget constraints are met. In particular, the genetic search-based approach the authors propose aims at assigning human resources to software development tasks, taking them from a pool of developers having different levels of skills. The applicability of the proposed approach is demonstrated with an industrial case study. The paper ‘Search-Based Refactoring: an Empirical Study’ by Mark O'Keffe and Mel Ó Cinnéide presents an approach to automatically re-factor, using search-based techniques, object-oriented systems that undergo repeated addition of functionality. Such addition can, very often, deteriorate the quality of the system's underlying design. Their re-factoring is guided by the QMOOD quality model 25, and the paper compares different optimization techniques, such as Genetic Algorithms, Simulated Annealing, and Multiple Ascent Hill-Climbing. The results of experiments carried out on five software systems suggest that the latter technique outperforms the others. The paper ‘Search-based Many-to-one Component Substitution’ by Nicolas Desnos, Marianne Huchard, Guy Tremblay, Christelle Urtado, and Sylvain Vauttier presents a search-based approach that allows the replacement of an obsolete component with a composition of multiple components. When a component becomes obsolete, the approach searches for a minimal assembly of other components, exposing their functionality by means of ports that preserves the behavior of the original component. The proposed approach has been evaluated on a set of randomly generated components. We hope that readers will enjoy this special issue and find the selected papers insightful and inspiring. We would like to thank all the authors who presented papers for this special issue: there were many good quality papers, but unfortunately we could accept only three of them. Last, but not the least, we would like to thank the reviewers who provided timely, detailed and constructive feedbacks to the authors. Without their precious help this special issue would not have been possible.