A practical method for implementing string pattern matching machines

This paper describes a simple, efficient algorithm to locate all occurrences of any of a finite number of keywords in a string of text. The algorithm consists of constructing a finite state pattern matching machine from the keywords and then using the pattern matching machine to process the text string in a single pass. Construction of the pattern matching machine takes time proportional to the sum of the lengths of the keywords. The number of state transitions made by the pattern matching machine in processing the text string is independent of the number of keywords. The algorithm has been used to improve the speed of a library bibliographic search program by a factor of 5 to 10.

Scalable and queryable compressed storage structure for raster data

The objective is to develop a data structure that is capable of handling large data volumes and offers support for querying, analysis and validation. Based on earlier results (i.e. the full decomposition of space, the use of a TEN structure and applying Poincare simplicial homology as mathematical foundation) a simplicial complex-based TEN structure is developed. Applying simplicial homology offers full control over orientation of simplexes and enables one to derive substantial parts of the TEN structure, instead of explicitly store the entire network. The described data structure is developed as a DBMS data structure and the usage of views, function based indexes and 3D R-trees result in a compact topological 3D data structure. Theoretical aspects of this approach are described earlier [12, 17,18]. This paper describes both theory and implementation of the approach.

Forwarding packets based on networking names is essential for network protocols on different layers, where the `names' could be addresses, packet/flow IDs, and content IDs. For long there have been efforts using dynamic and compact data structures for fast and memory-efficient forwarding. In this work, we identify that the recently developed programmable network paradigm has the potential to further reduce the time/memory complexity of forwarding structures by separating the data plane and control plane. This work presents the new designs of network forwarding structures under the programmable network paradigm, applying three typical dynamic and compact data structures: Bloom filters, Cuckoo hashing, and Othello hashing. We conduct careful analyses and experiments in real networks of these forwarding methods for multiple performance metrics, including lookup throughput, memory footprint, construction time, dynamic updates, and lookup errors. The results give rich insights on designing forwarding algorithms with dynamic and compact data structures. In particular, the new designs based on Cuckoo hashing and Othello hashing show significant advantages over the extensively studied Bloom filter based methods, in all situations discussed in this paper.

Using Compressed Suffix-Arrays for a compact representation of temporal-graphs

The classic string indexing problem is to preprocess a string S into a compact data structure that supports efficient subsequent pattern matching queries, that is, given a pattern string P, report all occurrences of P within S. In this paper, we study a basic and natural extension of string indexing called the string indexing for top-k close consecutive occurrences problem (Sitcco). Here, a consecutive occurrence is a pair (i,j), i<j, such that P occurs at positions i and j in S and there is no occurrence of P between i and j, and their distance is defined as j−i. Given a pattern P and a parameter k, the goal is to report the top-k consecutive occurrences of P in S of minimal distance. The challenge is to compactly represent S while supporting queries in time close to the length of P and k. We give three time-space trade-offs for the problem. Let n be the length of S, m the length of P, and ϵ∈(0,1]. Our first result achieves O(nlog⁡n) space and optimal query time of O(m+k). Our second and third results achieve linear space and query times either O(m+k1+ϵ) or O(m+klog1+ϵ⁡n). Along the way, we develop several techniques of independent interest, including a new translation of the problem into a line segment intersection problem and a new recursive clustering technique for trees.

In this paper, we discuss an indexing method for solving the multiple string pattern matching problem, by which we are given a set of short strings R = {r1, ..., rl} and required to locate all substrings of a target string s such that each of them matches an rj in R. The main idea is to construct a pattern matching machine A and transform the reverse of s to a BWT-array as an index, denoted as BWT(s), and search A against it. During the process, the failure function of A is used to decrease the subranges of BWT(s) to be searched at each step. In addition, we change a single-character checking against BWT(s) to a multiple-character checking, by which multiple searches of BWT(s) are reduced to a single scanning. In this way, high efficiency can be achieved. Extensive experiments have been conducted, which shows that our method works better than almost all the existing methods for this problem.

The problem of sequence identification or matching — determining the subset of references from a given collection that are likely to contain a query nucleotide sequence — is relevant for many important tasks in Computational Biology, such as metagenomics and pan-genome analysis. Due to the complex nature of such analyses and the large scale of the reference collections a resourceefficient solution to this problem is of utmost importance. The reference collection should therefore be pre-processed into an index for fast queries. This poses the threefold challenge of designing an index that is efficient to query, has light memory usage, and scales well to large collections.To solve this problem, we describe how recent advancements in associative, order-preserving, k-mer dictionaries can be combined with a compressed inverted index to implement a fast and compact colored de Bruijn graph data structure. This index takes full advantage of the fact that unitigs in the colored de Bruijn graph are monochromatic (all k-mers in a unitig have the same set of references of origin, or “color”), leveraging the order-preserving property of its dictionary. In fact, k-mers are kept in unitig order by the dictionary, thereby allowing for the encoding of the map from k-mers to their inverted lists in as little as 1+o(1) bits per unitig. Hence, one inverted list per unitig is stored in the index with almost no space/time overhead. By combining this property with simple but effective compression methods for inverted lists, the index achieves very small space.We implement these methods in a tool called Fulgor. Compared to Themisto, the prior state of the art, Fulgor indexes a heterogeneous collection of 30,691 bacterial genomes in 3.8× less space, a collection of 150,000 Salmonella enterica genomes in approximately 2× less space, is at least twice as fast for color queries, and is 2 − 6× faster to construct.

Efficient counting of square substrings in a tree

This paper proposes an efficient method for large-vocabulary continuous-speech recognition, using a compact data structure and an efficient algorithm. We introduce a very compact data structure DAWG as a lexicon to reduce the space. We also propose a algorithm to obtain the N-best hypotheses using the DAWG structure. This algorithm is composed of two phases: forward search and traceback. Forward search, which basically uses the time-synchronous Viterbi algorithm, merges candidates and stores the information about them in DAWG structures to create phoneme graphs. Traceback traces the phoneme graphs to obtain the N-best hypotheses. An evaluation of this method's performance using a speech-recognition-based telephone-directory-assistance system having a 4000-word vocabulary confirmed that our strategy improves speech recognition in terms of time and recognition rate.

Streaming triangle counting is a critical issue in graph stream mining, with applications in dense subgraph discovery, web mining, anomaly detection, and more. Recent efforts have focused on estimating triangle counts in graph streams, primarily through sampling methods. However, because of limited memory resources for handling high speed streams, traditional sampling methods suffer from reduced sampling rate and thereby performance loss. In this paper, we propose a new compact data structure called uHLL to process edge streams by considering the tradeoff between estimation accuracy and memory efficiency. Furthermore, different from conventional triangle counting algorithms, we solve the estimation of union set cardinality for edge-local triangle count under both centralized and distributed framework, so as to efficiently estimate the global triangle count by a one-pass streaming algorithm. To the best of our knowledge, this is the first implementation of a distributed framework using a compact data structure for streaming triangle counting. We provide theoretical proof of unbiasedness and derive the variance of the union set and global triangle count. We compare our scheme with 11 algorithms, showing that under the same experimental setting, uHLL and distributed uHLL are at least <inline-formula><tex-math notation="LaTeX">$ 2.3$</tex-math></inline-formula> and <inline-formula><tex-math notation="LaTeX">$ 1.7$</tex-math></inline-formula> times more accurate than the state-of-the-art, respectively.

Existing scalable data cleaning approaches have focused on batch data cleaning. However, batch data cleaning is not suitable for streaming big data systems, in which dynamic data is generated continuously. Despite the increasing popularity of stream-processing systems, few stream data cleaning techniques have been proposed so far. In this paper, we bridge this gap by addressing the problem of rule-based stream data cleaning, which sets stringent requirements on latency, rule dynamics and ability to cope with the continuous nature of data streams. We design a system, called Bleach, which achieves real-time violation detection and data repair on a dirty data stream. Bleach relies on efficient, compact and distributed data structures to maintain the necessary state to repair data. Additionally, it supports rule dynamics and uses a cumulative sliding window operation to improve cleaning accuracy. We evaluate a prototype of Bleach using both synthetic and real data streams and experimentally validate its high throughput, low latency and high cleaning accuracy, which are preserved even with rule dynamics.

Bloom filter is a compact and randomized data structure for representation of a set, in order to answer membership queries; Queries such as Is element x in set Y? There is no need to have set Y, for answering these kinds of questions; these questions can be answered by constructed Bloom filter from the set. Therefore, space saving is the main advantage of using Bloom filter. In this paper, first general structure of Bloom filter is explained and then by using the Bloom filter, a new method for error management and detection in computer networks is presented. In this approach, the sender makes a Bloom filter for the flow data that is to be sent and attaches this Bloom filter to data flow and sends them to receiver. The receiver by receiving the data flow and Bloom filter, makes the Bloom filter corresponding to data flow, then by comparing this Bloom filter with the received Bloom filter detects the possible errors. The results show that Bloom filter with the length of x input string is able to detects all errors except for errors with the length of exactly 2x, 4x and... etc.

This paper introduces a novel edge-based Automatic Passenger Counting System (APCS) that leverages passive Wi-Fi probe-request capture to estimate boarding and alighting events. Implemented on an ESP32 microcontroller, the system employs a multi-threaded firmware architecture to coordinate channel hopping, packet acquisition, GPS synchronization, and route state classification without relying on external infrastructure. Over a three-month deployment on 84 diesel and electric bus trips at UNICAMP, the APCS logged more than 2.1 million probe requests and achieved up to $\mathbf{7 3}$ % agreement with a commercial video-based counter. A compact on-board data structure records initial and final GPS fixes per device, facilitating accurate passenger attribution and seamless integration with downstream analytics. These findings validate the feasibility of an edgeonly Wi-Fi APCS as a cost-effective alternative to camera-based solutions in real-world transit settings.

This paper proposes a new CAD model for the design of lattice material components. The CAD model better captures the user’s design intent and provides a dual-scale framework to represent the geometry and material distribution. Conventional CAD model formats based on B-Rep generate millions of data files, which also makes design intent and material information missing. In the present work, a new shape-material model for lattice material components is proposed. At the macroscopic scale, a compact face-based non-manifold topological data structure is proposed to express the lattice shape-material information without ambiguity. At the microscopic scale, implicit function is adopted for the representation of lattice material components. Numerical experiments verify that the proposed CAD model provides a powerful support for design intent with minor space costs. Meanwhile, the representation method supports solid modeling queries of geometric and material information on each scale.